Guided Study: Continuous Deployment with Mr. Webhooks;
or, Github Continuous Disintegration
As I write this in early August 2026, I think pretty much everyone can see that Microsoft has been neglecting Github and it's gotten bad. A 10x increase in PRs thanks to LLM slop is the immediate cause, but the rot started years ago. Somehow in 2026, a full fifteen years after IPv4 address exhaustion, Github still doesn't support IPv6. The UI is buggier and slower than it was 15 years ago. Actions has grafted a CI/CD system onto something never meant for it. Other divisions of Microsoft (say, XBOX) have also suffered from a decline in quality over the same timeframe. I was a Halo fan in my teens and twenties and if you know what's happened since Microsoft took over from Bungie, you know. As it went with Bungie and Halo, so it's gone with Github and...Github.
(In fact, as I write this Github is in hour eight of a serious outage that broke Github Actions and along with it their webhooks emitters.)
I myself (Jesse, I'm the backend and infrastructure principal here at The Equitable Society of Bit Plumbers) have been considering lately how to jump ship from the SS Github. I've hardly ever been shy about running my own stuff, but I also don't have a habit of going out of my way to do something "just because". Github's reliability troubles speak to an organizational rot that can't not affect security, too. So far Github Actions-related security breaches have had to do with npm supply-chain attacks ('No Way To Prevent This,' Says Only Package Manager Where This Regularly Happens) and VSCode's insecurity-by-design.
What's holding me here is that Github Actions are good-enough for most purposes, and they're the default. However, I'm a Xennial. It's quite clear to me things are never so bad they can't get worse.
It's 2026. Do you know what AWS permissions your Github role has?
I'm not saying it is or it isn't going to happen, but I think we can reasonably anticipate that an attacker will be able to compromise Github Actions itself for a short time. An ambitious attack would use Github Actions not for a broad supply-chain attack (git's construction as a Merkle tree of commits makes this exceptionally difficult at scale), but rather as the staging point for an attack outwards to the systems it has access to.
I have found it's a fairly common pattern ("best practices" are wild these days) to give Github Actions an IAM role that can edit AWS Elastic Compute Service task definitions. Just off the top of my head, if I had this kind of access I could use it to slip in a container that connects to the AWS Instance Metadata Service and exfiltrates the instance's credentials.
For those more familiar with Kubernetes, a task is like a pod, it's composed of multiple container definitions. Rather than restricting down the permissions to just provide a URL to a docker image, people are giving Github Actions blanket rights to define entire pods.
I'm tired of being what you want me to be
We who remember the Buffer Overflow Golden Age of the early 2000s have put on our cargo pants and are screaming while Linkin Park blasts in the background.
Anything you let the internet connect to, you had to assume it could be compromised. You designed your network around that. You had a special untrusted network segment called the DMZ (if you don't know what this is, email us and I'll be your security consultant) and you didn't let things in the DMZ connect back into your private network unless they had a good reason and you had no alternative.
We separated our privileges, obeyed the principle of least privilege, and we worried about our deputies being confused. We did this not because we were geniuses, but because we were idiots who lived in a world without law or reason.
I remember how, on January 24th, 2003, an exploit in Microsoft SQL Server ground the Internet to a halt. Suck it, OpenClaw cringeboys. One man and some machine code did in an evening what you and your terawatts of GPU cannot.
Network security in the mid-to-late 2010s became YOLO, and ten years later we live in such a state of intellectual collapse that those seem like the salad days of yore.
All I want to do is be more like me and be less like you
Ok, so our problem is that Github Actions has too many permissions. How on Earth could we ever solve this problem!? Oh, right, we could take those permissions away because they were granted by AWS IAM policy and not by God.
How it (generally) works now:
- Someone pushes changes to
main. - Github Actions builds a Docker container with those changes and pushes the container to some repository.
- Github Actions then uses its permissions to reconfigure ECS or the Kubernetes deployment or whatever.
I'm going to use one of the two or three intellectual techniques I've determined over a 20 year career to be overpowered (plz nerf): reverse and reduce the problem, then stubbornly insist on doing the small amount of work to follow through on it. This is what makes me a Cargo Pants Unix Sorcerer.
Or if you like, the art of system engineering is deciding on what not to do. Amateurs debates whether or not we could; professionals debate whether or not we should.
This is service automation, so I am not going to overbuild it. I'm targeting ECS services. The technique transfers to your runner of choice -- or if you're not using docker containers (I love you and you are after my own heart) even the replacement of EC2 instances or updating services running on machines.
Could we skip using Mister Webhooks here and "just" restrict Github Actions to restarting ECS services? As it turns out, the IAM policy section of the AWS manual for ECS just says "lol. lmao, even". We cannot separate restarting a service from changing its definition. Just. Wonderful.
For our purposes terraform will manage our ECS cluster and task definitions. We are not going to let something outside terraform modify resources that terraform controls. If you're thinking "hey, does that mean we're going to make the continuous deployer run terraform and inject tfvars into it through the shell environment?" email us and can work out an advisory relationship where I help you to analyze and simplify problems.
What it means is that we're going to configure every ECS task to use the latest tag for our image. Our Docker image build process will set the commit SHA as an environment variable so we can handwave away the question of "but how do I know what version of my code is running?" and focus on deploying each new build as it arrives. The image upload step will update the latest tag so that all the deployer has to do is trigger a restart.
Every second I waste is more than I can take
What I'm going to do:
- Someone pushes changes to
main. - Github Actions builds a Docker container with those changes and pushes the changes to a repository.
- Mister Webhooks has been receiving all Github webhooks and storing them.
- A piece of code we control reads the Github lifecycle notifications and restarts the ECS service when new code is available.
I already know how to deploy software to ECS, so that's going to be left as an exercise to the reader (or if you like this and want us to set it up for you, as always, email us). I'm going to run it locally using my own AWS credentials because what's interesting is processing Github lifecycle events. IAM policies are boring.
I'm going to do this project in TypeScript since to get yourself into the kind of trouble that Mister Webhooks bails you out of you have got to have been using JavaScript.
Breaking the Habit
First off, I've got to assume you've followed the quick start instructions from the Mister Webhooks UI and have used mwtail.
We keep all the Mister Webhooks client implementations together in a single repo. We'll start with creating a project scaffold:
$ mkdir deployd && cd deployd
$ pnpm init
$ pnpm add -D typescript @types/node tsx
$ pnpm install @mister-webhooks/client
$ mkdir src && touch src/index.ts
$ echo 'console.log("Hello World!");' > src/index.ts
$ pnpm set-script dev "tsx src/index.ts" # needs pnpm > 11.3
Setting up an appropriate tsconfig.json I'll leave up to you, or you can use the one I used.
I pronounce it "deploy-dee".
As programming is the art of debugging an empty file, I always start my programs, no matter how complex, as a(n un-)glorified Hello World. Then I build up the central logic of the program (and for a real program, write tests once the organizational structure of the logic is clear and it's a matter of filling in cases rather than shaping up the inputs and outputs). When that looks good, I'll connect the central logic to the effectors, which are what actually do the work.
Mister Webhooks exists to support this kind of programming. It keeps a permament record of all webhooks events to allow us to develop against real data.
We'll develop the code, then, in the following steps:
- Write a Hello World-equivalent, which reads every message in the log and writes it to the terminal.
- Adapt the message reader so it looks only at the event types we want, extracts the relevant data from them, and prints that.
- Use the relevant data to produce a description of what should be done.
- Hook the what-should-be-done code up to actually-do-the-thing code.
I don't want to be the one the battles always choose
Our Hello World equivalent looks like this:
import { ConnectionProfileConfig, Logger } from "@mister-webhooks/client";
import { MisterWebhooksConsumer } from "@mister-webhooks/client";
import { logLevel } from "kafkajs";
var fs = require("fs");
async function main() {
//
// Validate environment parameters
//
if (!process.env.CONNECTION_PROFILE) {
process.stdout.write(`CONNECTION_PROFILE=<PATH> env parameter required>\n`);
process.exitCode = 1;
return;
}
if (!process.env.TOPIC) {
process.stdout.write(`TOPIC=<NAME> env parameter required\n`);
process.exitCode = 1;
return;
}
// past here the env parameters are all valid
// Read Connection Profile from disk
const profile: ConnectionProfileConfig = JSON.parse(
fs.readFileSync(process.env.CONNECTION_PROFILE),
);
// Set up webhook log consumer
const consumer = new MisterWebhooksConsumer({
config: profile,
topic: process.env.TOPIC,
handler: async (_logger: Logger, payload: any) => console.info(payload),
startPoint: "EARLIEST",
logLevel: logLevel.DEBUG,
});
// Trap ^C and cleanly shut down
process.on("SIGINT", async () => {
await consumer.shutdown();
process.exit(0);
});
// Run it
await consumer.start();
}
main();
Now for some remarks. We take parameters as environment variables, to be all 12 Factor App about it. This is the defacto standard for anything packaged in a docker container, so I'm going to assume you're familiar with it. I'm also going to assume you've set up a Github endpoint and pointed Github's webhooks at it.
The Connection Profile contains what a client needs in order to connect to the Mister Webhooks data plane, which is where we keep the immutable records of all the webhook events you've been sent. Our data plane consists of a Kafka cluster we host. Either reuse the one you just used for mwtail, or create one specifically for deployd.
new MisterWebhooksConsumer({
config: profile,
topic: process.env.TOPIC,
handler: async (_logger: Logger, payload: any) => console.info(payload),
startPoint: "EARLIEST",
logLevel: logLevel.DEBUG,
});
The topic is the Kafka output stream for an endpoint. The handler prints each event to the terminal at INFO level, while we set the underlying consumer to DEBUG so we can see everything that's happening (omitting this and leaving it at the default of INFO is also ok).
Finally, startPoint. There are three values for this: EARLIEST, LAST_PROCESSED, and a Date. If you give it a date you'll pick up reading at the first event we received after that timestamp. But EARLIEST and LAST_PROCESSED are where the magic happens. When developing a program we'll set the startPoint to EARLIEST and each time we run it it'll read the whole historical record from the beginning. If we set it to LAST_PROCESSED, the data plane keeps track of the last message your consumer consumed, and if it crashes it'll pick up where it left off.
We also want to trap when the user hits Ctrl-C and exit the program cleanly. It's important to do a clean shutdown, otherwise it'll take the Kafka cluster about 30 seconds to detect a client's disconnected. During that time new clients that connect won't receive any data.
To run this, you'd do something like:
$ CONNECTION_PROFILE=~/path/to/connection-profile-deployd.json\
TOPIC=incoming.randomletters.github pnpm dev
where you've set the CONNECTION_PROFILE and TOPIC env parameters to the path you downloaded your connection profile to, and the topic associated with your Github endpoint. You can find it in the Mister Webhooks UI, in the detail screen for your endpoint.
When you run it you'll see the client connect, wait a few seconds, and then start printing every webhook Github's sent your endpoint. Good. Good.
Congratulations. You've written your first Mister Webhooks consumer.
'Cause inside, I realize that I'm the one confused
Good God. There is a lot of data. Let's try paring things down a little.
Mister Webhooks endpoints capture relevant HTTP headers for you, and Github includes in the X-Github-Event header the event type name. Perhaps what we could do is write out the partition ID (it's a Kafka thing), the offset (it's a Kafka thing, but it's also the message sequence number), the event type name, and any lifecycle state associated with it.
Let's give this a shot:
new MisterWebhooksConsumer({
config: profile,
topic: process.env.TOPIC,
handler: async (_logger: Logger, payload: any) =>
console.info(
payload.partition,
payload.offset.value,
payload.headers.get("X-Github-Event"),
payload.message.action,
),
startPoint: "EARLIEST",
logLevel: logLevel.INFO,
});
and when we run it, what do we get?
{"level":"ERROR", "message":"[Consumer]
Crash: KafkaJSNonRetriableError:
payload.headers.get is not a function", ...}
Oh dear, that's not a good prize. Even worse, the program hangs. Fortunately the MisterWebhooksConsumer is an event emitter and we can listen to its lifecycle.
It turns out what we should have done is:
// Trap ^C and cleanly shut down
process.on("SIGINT", async () => {
consumer.shutdown();
});
consumer.on(MISTER_WEBHOOKS_EVENT.DISCONNECTED, () => {
process.exit();
});
A handler exception, just like shutdown(), causes the consumer to stop and then to disconnect. So all we have to do is listen to the DISCONNECTED event and exit the process then.
This is far better. It's also clear we should use the [] operator for pulling out a header:
handler: async (_logger: logger, payload: any) =>
console.info(
payload.partition,
payload.offset.value,
payload.headers["X-Github-Event"],
payload.message.action,
payload.repository.full_name,
)
And here we have output that looks like:
0 2415n [ 'push' ] undefined mister-webhooks/website
0 2416n [ 'workflow_run' ] requested mister-webhooks/website
0 2417n [ 'workflow_job' ] queued mister-webhooks/website
0 2418n [ 'check_run' ] created mister-webhooks/website
0 2419n [ 'workflow_run' ] in_progress mister-webhooks/website
0 2420n [ 'workflow_job' ] in_progress mister-webhooks/website
0 2421n [ 'check_suite' ] completed mister-webhooks/website
0 2422n [ 'check_run' ] completed mister-webhooks/website
You could say we're getting somewhere. I spent some time in a café poking around at the Github webhooks payload, as well as getting the @octokit/webhook-types type definitions working. I defined an intermediate type called Deployable, which contains information about something that could be deployed. It distills down the content of a WorkflowRunEvent, specifically for when a workflow run (a workflow run is the logical grouping of a bunch of workflow jobs) completes successfully.
Its type looks like this:
type Deployable = {
repo: string;
branch: string;
commit: SimpleCommit;
workflow: {
path: string;
timestamp: string;
run: {
number: number;
attempt: number;
};
};
};
and an exemplar instance looks like this:
{
repo: 'mister-webhooks/webhooksd',
branch: 'initial-build',
commit: {
author: { name: 'Jesse', email: 'jesse@bitplumbers.example' },
committer: { name: 'Jesse', email: 'jesse@bitplumbers.example' },
id: 'c5c0c24eb1a89ab331816361e60143ccee6b249b',
tree_id: '3285408640ff746db5eddc53060c41df3d58b19e',
message: 'Make statisticsd fetch parameters configurable',
timestamp: '2026-06-28T08:33:30Z'
},
workflow: {
path: '.github/workflows/build-docker-image.yaml',
timestamp: '2026-06-28T08:37:10Z',
run: { number: 10, attempt: 1 }
}
}
while the distillation function looks like:
(event: WorkflowRunEvent) => {
if (!(
event.action == "completed" && event.workflow_run.conclusion == "success"
)) {
return;
}
return {
repo: event.workflow_run.head_repository.full_name,
branch: event.workflow_run.head_branch,
commit: event.workflow_run.head_commit,
workflow: {
path: event.workflow_run.path,
timestamp: event.workflow_run.updated_at,
run: {
number: event.workflow_run.run_number,
attempt: event.workflow_run.run_attempt,
},
},
} as Deployable;
};
Note that we're using the head repo, and not the base repo (which is just .repository). Why? We want to avoid the possibility that an attacker might try to cause our continuous deployer to deploy code that's only been built from a branch they control.
Secure systems get secure by enumerating what can be done. If we were instead to try to point-by-point rule out what cannot be done, there would be no end to the matter. Secure systems are careful in how they handle untrusted input. This tutorial is only going to restart an ECS service, but I believe in archaic concepts like "foreseeable hazards" and "social responsibility" and "modeling correct behavior".
Once we've got a Deployable, we only have a thing that could be deployed. Just because we can does not automatically mean we should.
So I'm going to be fancy and throw this in:
type Rule = {
workflow_path: string;
branches: string[];
};
const filter = (rules: Map<string, Rule>) => (event: Deployable) => {
const rule = rules.get(event.repo);
if (rule) {
return (
(rule.workflow_path == "*" ||
rule.workflow_path == event.workflow.path) &&
rule.branches.includes(event.branch)
);
}
return false;
};
and some filtering rules:
const select = filter(
new Map([
[
"mister-webhooks/frontend",
{
workflow_path: ".github/workflows/build-docker-image.yaml",
branches: ["main"],
},
],
[
"mister-webhooks/webhooksd",
{
workflow_path: ".github/workflows/build-docker-image.yaml",
branches: ["main", "initial-build"],
},
],
]),
);
For each repo, we trigger only when a particular workflow finishes on one or more branches. The next thing we need is to know what ECS services to restart when we get a Deployable to take action on:
const serviceTable = new Map([
[
"mister-webhooks/frontend",
["arn:aws:ecs:us-west-2:123456789012:service/ecs-dev/api"],
],
[
"mister-webhooks/webhooksd",
[
"arn:aws:ecs:us-west-2:123456789012:service/ecs-dev/configuratord",
"arn:aws:ecs:us-west-2:123456789012:service/ecs-dev/statisticsd",
"arn:aws:ecs:us-west-2:123456789012:service/ecs-dev/webhooksd",
],
],
]);
The multiple services restarted for one repo is to account for a mono-container style of deployment where you run a single docker container but give it different entry point commands. Our handler now looks like:
handler: async (logger: Logger, payload: MessagePayload<WebhookEvent>) => {
if (!(payload.headers["X-Github-Event"][0] === "workflow_run")) {
return;
}
const deployable = extract(payload.message as WorkflowRunEvent);
if (deployable && select(deployable)) {
const serviceARNs = serviceTable.get(deployable.repo);
if (!serviceARNs) {
throw new Error(
`${deployable.repo} does not have a service table entry`,
);
}
serviceARNs.forEach((serviceARN) => {
logger.info(`would restart ${serviceARN} to pick up ${deployable.repo} build \
${deployable.workflow.run.number} (${deployable.workflow.timestamp}): \
${deployable.commit.author.name}'s '${deployable.commit.message}' \
(sha: ${deployable.commit.id}; branch: ${deployable.branch})`);
});
}
}
and we've got output like:
would restart arn:aws:ecs:us-west-2:123456789012:service/ecs-dev/configuratord to pick up mister-webhooks/webhooksd build 10 (2026-06-28T08:37:10Z): Jesse's 'Make statisticsd fetch parameters configurable' (sha: c5c0c24eb1a89ab331816361e60143ccee6b249b; branch: initial-build)
But now I have some clarity to show you what I mean
The Mister Webhooks, and consequently novel, part is done. Printing out what we would do is about 80% of doing it. Describing what we're going to do, in detail, is another 5 - 10% of it.
For each service in our restart list, we will:
- Create an ECS client to connect to the service's region.
- Send an empty ECS
UpdateServiceexcept for theforceNewDeploymentflag. This effectively requests a service restart. - Wait for the service to return to a stable state.
I'm going to keep the code as simple as possible for demonstration purposes. An improved version of this would use serviceARNs.map to build up all the restart jobs for a service at once, and Promise.all to run them concurrently. Services take about two to three minutes to return to a stable state so we won't concern ourselves about reusing ECS client sessions for efficiency.
$ pnpm install @aws-sdk/client-ecs @aws-sdk/util-arn-parser
for await (const serviceARN of serviceARNs) {
const arn = parse(serviceARN);
const cluster = arn.resource.split("/")[1];
const client = new ECS({ region: arn.region });
await client.updateService({
service: serviceARN,
cluster: cluster,
forceNewDeployment: true,
});
logger.info(
`${payload.topic}[${payload.partition}]@${payload.offset}: sent restart to ${serviceARN}`,
);
const startTime = Date.now();
const waitResult = await client.waitUntilServicesStable(
{ services: [serviceARN], cluster: cluster },
Number.MAX_SAFE_INTEGER,
);
logger.info(
`${payload.topic}[${payload.partition}]@${payload.offset}: ` +
`${serviceARN} restarted after ${waitResult.state} ${(Date.now() - startTime) / 1000} seconds`,
);
client.destroy();
}
The code does exactly what I listed out before. We extract the ECS cluster name and region from the service, we use that to make an ECS connection to kick the service, we kick it, and we wait for it to return to steady state.
In The End
If you've been folowing along, congratulations. You've built a working Mister Webhooks-driven continuous deployer for Github. You should make sure to go back and set startPoint to "LAST_PROCESSED" so that you don't keep starting from the beginning.
If you're just interested in running the code itself, or seeing the fully-worked example, I've got the code live on Github.