Most monitoring designs answer one question and skip the other. The question everyone answers is what makes this thing healthy. The question almost nobody writes down is who is allowed to say it has stopped being healthy, and, harder, who is allowed to say it is healthy again.
The second question decides whether the first is ever enforced, and it is on my mind this year because the systems we are putting next to real work are watchers before they are anything else. A copilot reading a ticket queue, a retrieval-augmented assistant checking a draft against policy, an early and carefully scoped autonomous experiment sitting beside a delivery and raising its hand when something looks wrong. Each of them has to decide what happens to a raised hand.
The best answer I have seen to that was not written for software at all. It was written for a project.
A recommendation that reads like housekeeping
The better part of a decade ago I led the technical workstream of an independent post-mortem review of a failed offshore application build, commissioned by a European banking group's IT services subsidiary. The build had lost its quality and then its schedule, and the work had been taken back in house before the review opened.
The recommendations section carried the usual heavy material. It also carried one small item that read, on the day, like housekeeping: a template of status definitions for both parties to agree up front on the next assignment, so that there could be no confusion or disagreement about how good or bad things were. The note attached to it was careful about what it was not saying. It was not suggesting the commissioning side's own ratings had been wrong. The template was there to remove the argument in advance, not to settle one.
That is the kind of recommendation that gets nodded at and filed, and I should be straight about its standing. The review does not record two parties reporting the same month in different colours. It records a template written so that they could not. I have come back to it more than once since, because the definitions themselves are not what makes it good. The structure is.
The two rules were not the same kind of rule
Read the healthy definition and the warning definition side by side and you notice they are built out of opposite logic.
Healthy is conjunctive: a list of conditions that must all hold at once, and it stays healthy only while every one of them is true. No partial credit, no weighted average, no scoring, nothing on the list tradeable against anything else on it.
The warning rule is disjunctive: any single condition being true is sufficient. You do not need a pattern, a trend or a second signal.
That pairing is not an accident of drafting. The two rules are written from opposite ends so the boundary between them cannot drift. Had healthy been written as a score, a project could stay healthy by being excellent on most of the list and quietly broken on one item of it. Had the warning rule needed several conditions at once, the first real problem would have had nowhere to go, and the person watching it would have had to wait for company before saying anything.
Anyone who has built an alerting layer knows both failure modes. The composite health score that stays green while one component is on fire. The alert rule with so many conditions on it that by the time it fires you already knew. The template refuses the middle. Health is unanimous. Trouble is unilateral.
The right to raise and the right to silence
The part I now consider the actual finding is not in the definitions at all. It is in who holds which right.
Raising the flag was written to be the easy act, available to people close to the work without permission from above. Standing it down was written to be the hard one, reserved to a single named owner. Escalation was the wide right. De-escalation was the narrow one, and it had a name attached to it.
Notice what that buys. It removes the main incentive problem in status reporting, which is that the person who most wants a project to look healthy is usually the person closest to it and under the most pressure. If they can both set and clear the flag, the flag measures their nerve. If they can only set it, the flag measures the project, and the cost of raising one falls to nearly zero, which is what you want: better to absorb a pile of flags that turn out to be nothing than miss the one that was not.
It is not free. A wide right to raise and a narrow right to clear produces a backlog of open flags, and that backlog is real load on the named owner. The design is only honest if the load is resourced. A clearing right nobody has time to exercise becomes an unread list, which is auto-clearing with worse paperwork.
Worth stating plainly: this was a template inside a set of recommendations at the close of a review, proposed for assignments that had not started. The build had already been taken back in house. The review produced findings and recommendations. I have no results from the template, and would not trust a piece that offered you any.
Why this matters more when software does the watching
Everything above is a human governance pattern. It gets more useful, not less, the moment the watcher is software.
Put an assistant next to a delivery with a view of the plan, the ticket queue, the test results and the change log, and it will find things. Some will be real. A lot will not be, because the model is reading a partial picture from a document set nobody curated for it. The instinct, once the false positives arrive, is to make it more conservative: raise the threshold, require corroboration, teach it to speak only when sure.
That instinct optimises the wrong side of the design. A monitor with a high bar for raising is a monitor that will be quiet through the failure you built it for. The correct move is the one the template made: keep the raising cheap and wide, and put the discipline on the other side of the ledger. Let the model flag freely, and make clearing a flag a privileged action with a named owner and a written reason.
Which makes auto-resolution the anti-pattern to watch for in the monitoring layers now being built around copilots and early autonomous experiments. A flag that clears itself when the next poll comes back clean has silently converted an incident into a blip. A model permitted to raise a concern and then withdraw it holds both rights, which is the arrangement the template was written to prevent, and the one most of these dashboards ship with by default.
A monitor that can clear its own alarms is not a monitor. It is a second opinion with the first opinion's password.
There is a second reason to write this down early. Every open flag and every clearing decision, with its owner and its reason, is a record of judgement about a system's behaviour: the evaluation set you will wish you had when somebody asks you to prove the thing worked, and the lineage you will need when they ask why it did what it did. European regulators have reached political agreement on rules for higher-risk uses of these systems, and whatever the final obligations turn out to be, none of them will be satisfied by a dashboard that resolved its own history.
What to write down before you deploy anything
A short block of definitions, and none of it requires a data scientist.
The unanimous list: what must be true, all at once, for this to be reported as fine, written as conditions rather than a score so no item can be traded against another. The single-trigger list: what one thing being wrong is enough to raise a flag, with no requirement for a second signal and no threshold for how often it has happened.
Then the two rights, separately. Who may raise, which should be nearly everyone and every monitor you deploy. Who may clear, which should be one named person or function with the time to do it. Never the same party on both sides, and never the model on the clearing side at all. And a reason field on every clearing decision, because without it you have permissions and no memory, and nobody will later be able to reconstruct which flags were real. When RealAI Platform work starts on a monitoring layer, that field is the first column we insist on: everything else on the dashboard can be rebuilt from logs afterwards and that column cannot.
That is an hour of work and it belongs in the design, not in the retrospective. The review I keep returning to found the documentation and the governance around the failed build in decent order on paper. What was missing was not documents. It was an agreement, written down before anyone needed it, about what the words meant and who was allowed to use them.
We are about to hand the same vocabulary problem to software that reports continuously rather than at the end of a reporting period. The fix has not changed.
Drawn from the recommendations section of an independent post-mortem review of a failed offshore application build, commissioned by a European banking group's IT services subsidiary, on which I led the technical workstream. The status definition template described here was proposed for future engagements: the review produced findings and recommendations, not delivered results. Reading its structure as a pattern for automated monitoring is ours.
“A monitor that can clear its own alarms is not a monitor. It is a second opinion with the first opinion's password.”
Get in touch
Put RealAI’s applied-AI team on your hardest data problem.
We help enterprises move from pilots to production: sovereign models, governed data, and agents you can audit. Start with a value-first assessment.