Coverage is a number about the code, not about the tests. It says what fraction of the lines ran while the suite was running, and nothing at all about what the suite would have refused to let through. Those are two different quantities, and when they come apart far enough the number stops understating assurance and starts misstating it, because a line that executes without anything contesting its behaviour is still reported as covered, and a build server reads that report as green.
A European retail and private banking group had an outside supplier build a front-office advisory application, and commissioned an independent technical assessment of what came back. The assessment was point in time and said so in a footnote on every page: it read one build, on one date. Eighteen quality areas sit on a fixed grid, most of them broken into named sub-headings, each sub-heading traced to evidence. Testing was one of the eighteen, and it is the only aspect in the whole pack where the evidence is a measurement rather than a defect ticket. Thirty-five tracker items were logged across twelve of the eighteen pages. Testing carries none of them. The measurement is the finding.
The reviewer wrote three sentences under it. Coverage on the server-side stack, measured with a named coverage tool, was approximately 14 percent. Coverage on the front-end stack could only be estimated, and was most probably below 10 percent. And most of the tests were not very useful, for lack of content checks, being mostly null and length tests.
The challenge
Those three sentences do three different jobs, and the third one undoes the first two.
The first is a measurement with a named instrument behind it, which makes it the hardest number in the pack. The second is explicitly an estimate, and the reviewer says so rather than rounding the honesty away. The asymmetry is worth sitting with: one stack had the instrumentation to produce a figure and the other did not, so the half of the application an adviser actually touches was assured by inference. That is a measurement gap before it is a coverage gap, and it is the cheaper of the two to close. You cannot set a floor under a number nobody can produce.
The third sentence is the one that changes the reading of both. A test that asserts a call returned something, and that the something has a length, will pass for any change that leaves the shape of the answer intact. Rename a field and the shape holds. Invert a comparison on a suitability check and the shape holds. Return a stale cached position instead of a current one, drop a currency conversion, round a fee the wrong way: the shape holds in every case, the assertion is satisfied, and the line is counted as covered. In a securities advisory workflow those are not cosmetic failures. They are the answer the adviser reads off the screen and repeats to a client.
So the 14 percent is a ceiling on the fraction of lines that executed under test, and the assured fraction is the subset of those lines whose behaviour something would have contested. Nobody measured that subset. The reviewer's own sentence says it is small. Coverage that asserts nothing is worse than no coverage, because no coverage reports nothing and this reports green.
One qualification belongs on the record, and it comes from the file rather than the words. All three testing sentences are set in the colour the author used, on two other pages, to mark items he had not yet verified. The hardest number in the pack is flagged unverified by the person who produced it. It is still the best number available and it names its instrument, but it belongs in print with that caveat rather than laundered into a certified result.
The approach
The grid ran on three chains of evidence, and the shape is worth copying. Most areas were bound to a tracker item, which anyone can open and read. Testing was bound to a direct measurement, because a tracker cannot evidence an absence and thin coverage is an absence. One area, mobile responsiveness, was bound to the statement of work rather than to a defect, because neither a tracker nor a measurement can evidence a promise, and the section number was left blank. A pack that uses only the first chain will under-report everything that failed by not existing.
The rest of the pack tells you what the testing finding costs. Exception handling carried four items across three named failure modes: swallowing an error, handling errors only sporadically, and not handling them at all. The logging page was left entirely blank, and it is not alone: only four of the eighteen pages carry a written finding at all, the rest holding the template's own placeholder word where the text should be. The pack never distinguishes an area assessed and found clean from one never reached, so a blank supports no conclusion in either direction. Put the parts together: errors discarded at runtime, logging whose state is unknown, and a build-time check that asserts presence only. Three ways of learning that something went wrong, none of them reliably firing.
The remediation record is the sharpest thing in the pack, and it is an absence too. One class of defect is recorded as fixed: front-end race conditions and shared static state on the server, which the same sentence describes as easily identifiable, corrected some months after the assessed build and well before the pack was written. Every expensive structural finding, including coverage, carries no remediation note at all. That is not laziness. Remediation follows detectability, and the assurance gap that would have caught everything else stayed where it was.
The correct instrument here is not a coverage target. A coverage target applied to a suite of null and length tests produces more null and length tests, faster. Two measurements do better. First, the assertion mix: what share of tests compare a value against an expected value rather than checking that a value exists. Second, a deliberate break. Invert a boundary, flip a comparison, change a rounding rule, and see whether anything goes red. A suite that stays green through a planted defect has just measured its own assurance. Our Platform work wires both into the pipeline, because either one run once is a report and either one run on every merge is a control.
The outcome
What this engagement produced was a findings pack, and a draft one at that: an aspect grid, a column of tracker references, and on a single page a measurement. No suite was rewritten, no coverage floor was enforced, and the pack was written some seven months after the build it describes, long enough for the codebase under it to have moved. The numbers here are readings, not results.
What has changed since is the cost of being in this position. Coding assistants and agent loops now offer to do exactly the work this codebase needs, and the offer is real: hundreds of oversized classes, controllers running to around two thousand lines, and one large cyclic dependency holding the layers together are the sort of mechanical restructuring a well-harnessed agent handles competently. The constraint is not the agent's ability to write the change. It is that graded autonomy is graded against a check, and the check here asserts that something came back and had a length. An agent given that as its regression signal will refactor confidently, report success, and be believed.
So the sequencing is fixed, and it inverts what most banks are currently attempting. The assertions come first: characterisation tests that pin what the system actually does today, an evaluation set the change has to survive, a planted-defect run to prove the set has teeth, and only then an agent allowed to touch the code. With supervisors and the AI Act both asking institutions to evidence how automated components are tested rather than assert that they are, that ordering has stopped being an engineering preference. Our Consult engagements start by measuring the assertion mix, because it is the cheapest way to tell whether a modernisation programme has a floor under it.