How Wake-On-Message Lets AI Agents Coordinate Without a Human in the Loop

Give a team of AI agents a shared inbox and you've solved half the problem. The other half is harder: getting an idle agent to notice that mail has landed. No human should have to refresh a dashboard to relay the message. That second half is what Wake-On-Message solves. This is a technical follow-up to How We Built Our AI Agent Operating Infrastructure. It goes one level deeper into a single mechanism from that stack — one we now consider important enough to put under its own change-control process.

A durable inbox is not a timely one

Our Agent Bridge gives every AI agent an identity and a place to receive work. A session runs cw connect <seat>. From then on it can send and receive handoffs through cw handoff and cw reply. Those handoffs land in a table, agent_org_handoff, that survives the session that wrote to it. That solves durability. A handoff never disappears, even if the recipient is offline when it arrives. It does not solve timeliness. A table is a pull construct by nature: a row sits there until something queries it. Left alone, "coordination" just meant a seat happening to reconnect and noticing it had mail. That's closer to luck on a timer than to coordination.

What polling actually cost us

Our first fix was the obvious one: check more often. We ran twenty-two separate cw-watch polling daemons, one per seat. Each looped against the database on a fixed interval, asking "anything for me yet?" It worked, in the sense that mail eventually got seen. But it also meant twenty-two live connections hammering the database around the clock, whether or not anything had happened. It meant a latency floor pinned to the poll interval, no matter how urgent the message. And it degraded unpredictably the moment the connection pool got busy. Polling doesn't scale, because you pay its cost on every tick regardless of outcome. You're still capped at "as fast as you're willing to check" — never as fast as a real interrupt.

One listener, a genuine push

Wake-On-Message replaced the entire poller fleet with a single long-running process: tools/host-ops/bus-wake-listener.mjs. It holds a live Postgres LISTEN on one channel, agent_org_bus. A database trigger fires pg_notify() on that channel the instant a handoff row is inserted. The listener sits blocked and idle, using close to no resources, until it wakes — in well under a second. Our own operations documentation calls it "true push, sub-second." That phrase is the whole design: the database tells the listener something happened. The listener doesn't have to keep asking.

One detail cost us real debugging time and is worth stating plainly. LISTEN has to run on a direct, non-pooled database connection. Route it through a connection pooler — the sensible default choice for almost everything else that talks to the database — and the pooler recycles the connection before the notification ever arrives. A pattern that is correct everywhere else in the stack is silently wrong for exactly this one thing.

Not every message earns an interrupt

When the listener wakes, it doesn't boot a session for every message. It classifies the mail first. A directed request or decision — something that needs a reply or a call made — boots the recipient seat into a real session so it can act. A directed report, or a broadcast announcement to the whole org, lands in the recipient's inbox but boots nobody. That distinction prevents a storm. A message sent to everyone shouldn't spin up dozens of sessions just because everyone technically received it. Most bus traffic is informational. Only the messages that actually need a response are worth the cost of a full agent session.

Getting woken and getting answered are different claims

Treating "the agent was notified" and "the agent is acting on it" as the same event is exactly how a sender ends up unsure whether anyone is working on their request. Two small behaviours are enforced mechanically, at the listener level, not left to whichever seat handles the mail.

First: acknowledge before you act. The moment the listener boots a seat for a directed request, it posts a lightweight "received, acting" back to the sender. That happens before any real work starts. It fires from the listener itself, not the agent it just booted — so it goes out even if that session is slow, or crashes outright.

Second: one reminder, never silence. If a handoff is still unanswered a few minutes later, the listener re-pings the recipient exactly once. It also tells the sender, in its own voice, that a reminder went out. Not zero reminders, which lets things quietly drop. Not unlimited reminders either, which the org would learn to ignore. Together, the two rules turn "I sent it, so it should be handled" — an assumption — into something a sender can actually observe: acknowledged or not, reminded once or handled.

The failure we found, and the layered watch that followed

A single listener process is also a single thing that can die. It can be killed by a host restart. It can wedge on a bug. It can silently stop consuming its own channel while every surface-level health check still reads "alive." We hit a version of exactly that. A hidden error in one of the acknowledgement queries broke wake-acks and reminders for days, while liveness checks kept passing — because liveness and correctness turned out to be different questions, and the listener was answering the second one wrong.

Patching that one bug wasn't the real fix. The real fix was building an outside supervisor, tools/host-ops/bus-monitor.mjs, that injects a synthetic message on a schedule and checks that the acknowledgement genuinely appears. Not that the process exists — that the behaviour happens. That catches the next bug of this shape too, not only the one we already found.

Today that supervisor runs three distinct canaries, on three different clocks. A basic acknowledgement check fires every fifteen minutes, with an eight-second grace window before it counts as a miss. A second canary runs every thirty minutes and verifies the listener would actually boot a session, not merely acknowledge one. A third, also every thirty minutes, exercises the separate claim path used by our other automated workers. Underneath those sits a slower rhythm of presence signals: each agent's own heartbeat lands roughly every forty-five seconds, a watcher-level heartbeat every five minutes, a general staleness sweep every ten minutes, and a cloud-side check every thirty minutes that opens a GitHub issue automatically if a beacon has gone stale for more than thirty-five minutes. No single check is load-bearing on its own. Several independent clocks watch the same nervous system from different angles.

Protecting the thing that protects everyone else

A canary catches the listener dying. It does not catch a well-intentioned change that quietly breaks how the listener behaves. That happened to us. On one day in early August, Wake-On-Message regressed silently four separate times. A bug in how it acquired its own lease meant it stopped booting a worker at all. A revert accidentally dropped the logic that reclaimed a dead session's lease. A deploy ran the listener on code forty-one commits stale. A separate change introduced a silent non-zero exit that nothing was watching for. Each of the four was an authorised-looking change — reviewed, merged, shipped. None was caught by a design review before it landed, because none of our gates at the time asked "is this design still correct." They only asked "did the process crash" or "was this commit unauthorised."

The response was a change-control doctrine built specifically for this listener. It was ratified after a direct instruction from our CEO: a change to this system now has to go through the same discipline as a production change request. An RFC has to be written, and its design approved by an Architecture Review Board, before any implementation is committed. Approval is dual-key — the Lead Architect and the co-founder both have to sign off. A pre-commit gate enforces it mechanically, blocking any commit that touches the listener's core files unless it cites an approved RFC.

There is a deliberate exception for a live outage. If the bus is actually down and no agent can be woken, a hotfix can ship first, under a logged override. But a retro-RFC recording what changed and why is due within twenty-four hours. That keeps incident response fast without losing the paper trail.

The distinction matters. The canary answers "is it alive and behaving correctly right now." The RFC gate answers a different question: "should this design be allowed to change at all." Both are needed. The four silent regressions in August were not liveness failures. Every process involved was still technically running.

Why the economics hold as the org grows

This is infrastructure, not a convenience script, because it changes what scales with headcount and what doesn't. A poll-based system gets slower and more expensive as you add agents, because every new agent is another loop asking "anything for me?" on its own schedule. A push-based listener doesn't care how many agents are on the bus. It wakes exactly the one that got mail, exactly once, and everyone else stays silent and free. Coordination cost stops being a function of headcount. That's the same structural bet behind The Self-Coordinating AI Company. It's also the same instinct that led us to build a mechanical guard, rather than trust a memorised rule, after a separate audit found agents making claims on a human's behalf with no real instruction behind them — a story told in full in How We Stopped AI Agents Inventing the CEO's Decisions.

Model quality was never the bottleneck here. A capable agent that never finds out it has work to do is not a capable agent in practice. It's an idle one with good credentials. Wake-On-Message is unglamorous, low-level plumbing: a database trigger, a listener, two small enforced habits. Now it also has a governance process deciding whether the plumbing itself is even allowed to change. That's the newer lesson. It's not enough to watch that critical infrastructure is alive. You also have to control who can redesign it, and under what review, before the next well-intentioned change breaks something a health check will never catch.

Frequently asked questions

What is Wake-On-Message? It's the mechanism that lets our internal message bus push new mail to an AI agent in real time, instead of waiting for the agent to check. A single listener process holds a live Postgres LISTEN on the bus's notification channel, agent_org_bus. It boots the right agent in well under a second of a directed message arriving, instead of the agent finding out whenever it next happens to look.

Why not just have every agent poll for new messages? We started there — twenty-two separate polling daemons, one per seat, each checking on a fixed interval. It worked, but it meant constant database load whether or not anything had happened, plus a latency floor equal to the poll interval. A single push-based listener replaced all twenty-two, cutting both the load and the delay to close to zero.

Does every message wake an agent? No. Only a directed request or decision — something that needs a reply or a decision made — boots the recipient into a session. A directed status report or a broadcast announcement reaches the inbox but wakes nobody. A message sent to the whole org doesn't spin up dozens of sessions at once.

How do you know a woken agent is actually acting, not just marked as delivered? The listener posts an acknowledgement to the sender the moment it boots the recipient, before any real work starts. It re-pings once if the message is still unanswered a few minutes later. A separate supervisor also runs layered canaries: one every fifteen minutes checking that acknowledgements genuinely appear, another every thirty minutes confirming the listener would actually boot a session rather than just acknowledge. The check verifies behaviour, not merely that a process is running.

Can the wake-listener's own code be changed casually? No, not anymore. After a single day in which the listener regressed silently four separate times, through ordinary, authorised-looking changes, we put its core files under a formal change-control gate. Any edit now needs a written RFC, approved by both the Lead Architect and the co-founder, before it can be implemented. A pre-commit gate blocks an unapproved commit outright. A live outage can still be hotfixed immediately, but it requires a retro-RFC within twenty-four hours.

More in this series: How We Built Our AI Agent Operating Infrastructure.