On-call is a design review you are having too late
Every page is feedback about a decision made months earlier by someone who was not on the rota.
On-call is generally treated as an operational function: a rota, an escalation path, a set of runbooks. It is more usefully understood as a feedback mechanism — the point at which architectural decisions made in comfortable circumstances are evaluated under the least comfortable ones available.
Every page carries information about a decision. A service that pages weekly is telling you something specific about its design, its dependencies, or the assumptions it was built on. Most organisations receive this feedback, silence it, and never route it back to anyone able to act on it.
The separation that causes the damage
Where the people who build systems are not the people who carry the pager, the feedback loop is broken by design. Design choices that are convenient to implement and unpleasant to operate carry no cost for the person making them, and the results are predictable: chatty dependencies, retries without backoff, error messages that describe the symptom rather than the cause, no graceful degradation because degradation was never a requirement.
None of these are mistakes exactly. Each is a reasonable local decision by someone who was never going to be woken up by it.
Closing the loop
- Put the people who build a system on its rota. This single change alters design behaviour more than any standards document.
- Review pages as design feedback rather than incident volume. The question is not how many alerts fired but which decisions they are complaining about.
- Treat an alert that requires no action as a defect. Alerts that are routinely ignored train the team to ignore alerts, which is a serious and cumulative failure.
- Make failure a first-class requirement. What happens when this dependency is slow, rather than down, should be answered at design time.
- Track the cost. Hours on call, pages per week and interrupted nights are a real expense; when they are invisible, nobody funds the fix.
The compounding case
The argument for this is not fairness, though the fairness argument is sound. It is that operational pain is the most accurate information you will ever get about system quality, and it arrives free, continuously, and with precise timestamps.
Organisations that route that information back into design find their systems quietly improve. Those that absorb it into a rota and treat it as the cost of doing business find that the same three services page every month for years, and that the engineers who understand them best are the ones most likely to leave.
Let's build the system your business will run on next.
Tell us where it hurts. We'll bring the architects, engineers, and delivery model to fix it — and scale it.