Webhooks - The Introduction

This is the first in a series of technical articles on webhooks. They begin from an understanding of the technical fundamentals like HTTP to arrive at an understanding of how to build a solid and functional webhook reception system.

The Basics

Webhooks are HTTP callbacks.

Normally when you work with a web service like Stripe or Github, you call their API by sending them an HTTP request. Webhooks reverse that, and when something happens their service sends you an HTTP request. This is a lot simpler and more to the point than a complex polling architecture, especially for the service provider's engineering team. Of course, you have to have a service running to receive those requests 100% of the time. But because it's an HTTP query they make to you, in principle any variety of application can be built out of webhooks. Twilio's earliest iterations were completely webhook driven, where the way you built a Twilio app was to expose a webhook service Twilio could call.

APIRequestsWebhookEvents

a conceptual view of webhooks and api calls

The webhooks technical spec is basically this: "You register a URL with your service provider. Usually they will send POSTs to this URL with data in the payload body. In some cases it will be a GET. There may be authentication.".

An experienced system designer's blood runs cold when they read this because they know in the absence of specification happenstance rules. Request and retry semantics are undefined: some retry multiple times if they don't get a 2xx; Github's approach is "lol. lmao even". Payload format? Undefined. Maximum payload sizes? Undefined. Is meaningful data communicated in HTTP request headers? It's up to the provider. Authentication? Webhooks as a convention are 20 years old and if you tell me the authentication mechanism I can make a very good guess when the sender was written. What about timeouts? Also undefined, but "faster than you can do complex multi-step processing". Velocity? The same system that sends you one event per second normally will send you five hundred events per second during or after an incident.

These days, webhooks are mostly used so that vendors can notify you of lifecycle events: payments, git commits, document comments, subscription renewals, and so on. The ecosystem has settled into a sort of worse is better situation. When you use event data, it's best when you have all the historical records available, but also can have multiple processors reading events as they come in real-time. This is what an event bus provides, but vendors almost never offer them. Instead they provide webhooks and you're left to put the pieces together.

So the problem comes down to this: the established patterns of webhooks are far more specific than the technological conventions used to implement them. That means that developing a robust webhooks system either requires penetrating insight or the benefit of experience. Webhooks are a case where the "You Aren't Going To Need It" and "Build The Simplest Thing That Could Possibly Work" rules of thumb break down and lead you down the wrong path.

The Naïve Architecture

Teams or programmers who overapply YAGNI typically write a service that executes some business logic within the HTTP request handler. A Stripe handler may directly modify payment or account information, while a Github handler may have administrative rights to AWS, GCP, or Azure. They treat HTTP ingress from the Internet for webhooks as if it's the same as HTTP ingress for a JSON or gRPC API. They may omit (or misimplement) authentication because it's harder to imagine an attacker discovering and attempting to exploit a webhook receiver than an API endpoint. They may treat logging as an afterthought, where logs either exist and are too verbose about the wrong things, or don't exist at all and things just magically change in the backend for no apparent reason.

We who are steeped in The Ancient Ways (i.e., we remember the 1990s and know what a DMZ is) consider it unwise for a service directly reachable from the Internet to have any more than the minimum access it needs to do its job, and it's best if that job is made as simple as possible.

This architecture usually lasts until one of the following things happens:

  1. There is a security incident.
  2. The webhook sender has an outage and lights the webhook handler on fire when it clears its backlog.
  3. Adding more and more actions to the handler causes it to take too long and the sender starts timing the handler out. Chaos ensues once retries come in.
  4. Someone or something needs access to historical event data.

Once one or more of these limitations are reached, the team will reconsider the design. All the original problems with webhooks obtain, but now with an added urgency that can lead to an overdesigned system that suffers from the second system effect. System design is an art where less is usally more, but this comes into conflict with the feeling of needing to add things in order to fix each of these issues.

The Good Architecture

There is a good architecture for this that one can implement many different ways. It solves each of the problems with the naïve architecture by making the problem irrelevant instead of trying to address it directly. In this architecture we split webhook handling into three independent layers:

  1. Receiver: Accepts HTTP requests, authenticates them, and persists the authenticated requests into the durable store the fabric provides.
  2. Fabric: Routes requests to consumers, holds the request records for them, and tells them when event records are available to process.
  3. Consumer: Reads request records from the fabric, processes them, and marks them as processed when done.

The Receiver is limited in what it does, so you can make it fast and give it very few permissions. It needs only to receive requests from the Internet and write to the fabric. A well-chosen fabric can handle event surges without issue, meaning you don't have to worry about timeouts.

The Fabric can be active or passive. An active fabric is a store-and-forward event router, it pushes the events persisted to it into consumers and manages retries, rate limiting, and so on. A passive fabric is an event bus, where consumers pull and process events as fast as they can and retries and rate limiting are non-issues.

The Consumer layer actually does the event processing. The fabric buffers the event stream and handles multiple consumers, so the worker programs that process webhook events can be kept small and single-purpose. Individual consumer performance is a question of "Is this fast enough to keep up with average daily load?" rather than "Is this fast enough to keep up with peak load?".

There are many possible implementations of this architecture. Mister Webhooks represents one of them. The rest of our articles will investigate each layer in turn with some discussion about how we've implemented them in Mister Webhooks.