Test coverage usually gets treated as a hygiene number. It goes into the status pack next to the burndown, it gets a colour, and nobody argues about it, because everyone privately agrees that chasing a percentage is not the same as writing good software. That is true, and it has been used to excuse the wrong conclusion for a long time. Coverage is not a measure of quality. It is the limit on what any measure of quality is allowed to tell you, because a line no test executes cannot be observed by a test, however well that test would have been written.
A European banking group's IT services subsidiary commissioned an independent review after an offshore build of a regulator-driven investment advice application failed to reach user acceptance testing. The engagement had commenced at 2,576 contracted person-days. By the time the review ran, the statement of work had been terminated and the code had been taken back in-house. Part of the review was a technical read of the codebase in the client's own environment, and one of the numbers it produced was measured coverage across both the server components and the web client: approximately 10 to 15 percent.
Everything else in the technical findings follows from that number, though neither the report nor anyone in the room framed it that way at the time.
The challenge
Three defect classes came out of the technical findings. It is worth being precise about what they were and why the existing tests were never going to reach them.
The first was concurrency. Caching had been implemented on the client case by case rather than transparently, and the local cache was being used less for efficiency than as a shared memory area that different parts of the application wrote into and read out of. Data arrived from the server asynchronously, and the code consuming it had no way of knowing whether it had arrived yet. The consequence recorded in the findings is the textbook one: the order of reads and writes became effectively random, so the application sometimes showed the old value and sometimes the new one.
The second was session state. Static variables were being used on the server to hold state belonging to a session. With several sessions active at once, one session could reach data belonging to another. In an application that carries a client's investment position, that is a confidentiality problem before it is a correctness problem.
The third was authorization. Not every service call tested for the existence of an appropriate authorization for the caller. That gap was demonstrated through a penetration test, which is the load-bearing detail here: it was found by an exercise designed to look for it, not by the test suite, and not by anyone running the application the way it was meant to be run.
Now put those three next to a coverage figure between 10 and 15 percent.
A race condition is visible only to a test that deliberately interleaves two operations and asserts on the result. A session leak is visible only to a test that runs two sessions at once and asserts that neither can see the other. An authorization gap is visible only to a test that makes a call the caller should not be permitted to make and asserts that it fails. Each needs a second actor: a second thread, a second session, a second identity. A single happy-path run will pass in the presence of all three, forever.
That is the structural part. The arithmetic is worse. At 10 to 15 percent, most lines in that codebase were executed by no test at all, so the ordinary single-actor assertions were absent too. The suite was not merely thorough about normal behaviour and blind to concurrency. It was blind to most behaviour, and the three defect classes above are simply the ones that would have stayed invisible even at a far higher number.
The codebase was also actively suppressing its own symptoms. The review recorded a large number of guard statements that tested whether a following statement could execute and skipped it silently when it could not. That removes the visible exception and leaves the missing effect to surface somewhere else, later, as behaviour nobody can trace back. A suite that cannot see a defect is one problem. A codebase that swallows the signal the suite would have used is the same problem applied twice.
The approach
The review had no suite to lean on, so it read the code: an expert technical read of one build, the defect and code-finding logs, interviews on both sides, then a difference test against a later build to establish whether quality had moved. No significant improvement in quality or reliability had appeared, certainly not enough to justify entering user acceptance testing.
The recommendations that followed were unglamorous and almost entirely about evidence rather than technology. Agree the quality model between both parties in writing before delivery starts, rather than referring to a standard in a contract clause. Configure a commonly agreed tool against that model, run it frequently, and report the result into the project rather than into the supplier's own file. Read a sample of at least 10 percent of the code against the agreed standards, extrapolate the non-conformance rate, and track it as a regular indicator instead of discovering it at a gate review. Hold peer review to a discipline that works: a few hundred lines per sitting, and an inspection rate below five to six hundred lines an hour, because reading faster stops finding anything.
The review also named a contractual point plainly. Compliance with the contracted design principles was to be validated through the supplier's own internal quality process. A quality clause verified only by the party being measured is not a test of anything. It is a statement of intent with a process name attached.
The outcome
What this engagement produced was a finding, not a recovery. The application never reached user acceptance testing under the original arrangement, the statement of work was terminated, and the client's own architects took the code in-house. Some repairs had started before the review closed: the circular references between modules had been fixed by the client-side team, and the build was set to flag new ones as errors. That detail matters because it shows the codebase was fixable. What was not fixable inside the remaining schedule was the absence of evidence about it.
This is where our Consult work starts on an inherited codebase now, and not out of nostalgia for a delivery post-mortem. The arithmetic has moved in one direction only.
Copilots are landing in exactly this kind of team, and they raise the volume of code a small group can produce without raising the hours available to read it. If the assistant writes the implementation and a human writes the tests, coverage falls as a percentage even when the absolute number of tests rises. If the assistant writes the tests too, and nobody reads them with the attention they would give the implementation, the result is a suite that asserts whatever the code happens to do, encoding the defect rather than catching it. Both paths raise output and lower visibility, and the second does it while making the dashboard look better.
The discipline that answers this already exists in how careful teams handle a language-model feature. Nobody ships one on a demo; they build an evaluation set first, argue about what belongs in it, and treat that set as the artefact making the feature reviewable at all. A test suite is the same instrument pointed at ordinary code, and generated code needs it more than hand-written code did, not less, because the volume arrives faster than the reading.
Coverage was never the quality metric. It is the visibility floor, and the three things this review found by hand are what it was hiding.