A supplier can pass every test it writes and still hand over software nobody can accept. That is usually neither a lie nor negligence. It is what happens when two parties agree in writing that testing will occur and never agree on what a test tests.
A European banking group's IT services subsidiary commissioned a post-mortem review of an offshore software development project it had already terminated. The review looked at the contract and the statement of work, the requirement specifications, the design documents, the plans, the risk and change and defect logs, the test plan and its acceptance criteria, the weekly reports from both sides, and the interview record from both organisations. It also ran a high-level expert review of the code over two builds, with a difference test between them.
The finding on the contract is the one most readers will not expect. The master agreement was judged comprehensive, covering both application development and testing sufficiently. The commercial instrument was not the hole. The hole sat one level down, in what neither party had written about the shape of a test.
The challenge
Across the same six sprints, two testing approaches ran side by side. The supplier tested in silos, component by component. The subsidiary expected end to end. The review states the consequence plainly: the supplier's testing approach did not meet the client's expectations because it did not take account of the dependencies, being tested in silos.
Both parties were measuring something real. A component passes against its own declared inputs, and if you built the component and wrote the inputs, it will pass. A journey passes only when every upstream contract it leans on holds under the data it will meet. One of those questions can be answered inside a sprint by the team that wrote the code. The other cannot be answered by anyone until the pieces are wired together, which is exactly why it slides to the end of a plan and lands in acceptance testing.
It would flatter the supplier to leave it there. The review also put test coverage across the two stacks at roughly ten to fifteen percent, so the component testing that did run covered a fraction of what existed. The divergence still matters, and it matters more rather than less: at full component coverage the journey would have been just as untested, because neither party had written down that the journey was the thing under test.
The same divergence appeared in reporting. The two sides held different definitions of a red status. For the subsidiary, red meant the delivery date would be missed. For the supplier, red meant something was blocking forward progress. Both are defensible. Neither is wrong. They cannot be added together, and the review recommended fixing definitions wherever the project documentation leaves room for different readings, including where language and working culture make ambiguity likelier.
A further gap sat underneath all of it. The review found no evidence of a system design document setting out coding standards, design principles and the supplier's rationale for how it would build. That activity had been estimated at 140 person-days in the statement of work. High and low level design documents did exist, and the review rated them adequate. What was missing sat above those: the artefact describing how the supplier intended to build and what each component owed the next. A journey test needs that kind of stated contract before it can assert much. The review does not claim the omission caused the silo approach, and neither do we.
The misalignment was known. The review records that planning sat with the supplier and control with the client, that the two diverged on testing methodology, and that this was escalated regularly and never resolved at the execution layer. Escalation moved the issue upward, where everyone agreed it mattered, and it came back down unchanged.
The approach
The recommendation on this point is unglamorous and it is the whole lesson. Agree the shape of the test scope, component or journey, in the statement of work, alongside the acceptance criteria, before the first sprint. Not in sprint three, when the divergence surfaces as a status argument, and not at the point where one party is defending a delivery date and the other a code base.
That is cheap to write and hard to hold, because it forces a second decision at the same moment. If the agreed test is a journey test, somebody has to own the artefact that says what each component promises the next. Here that artefact was priced, then not produced, and nobody stopped the sprints over it.
The rest of the review is a reminder of what a component-only view leaves uncovered. Caching was done case by case on the client side and used as shared memory rather than for efficiency, so data written asynchronously could be read before it arrived and the order of reads and writes was not deterministic. The review's phrase for that is the most useful sentence in the technical section: sometimes you see the old data and sometimes the new. Static variables held session state on the server, which in a product handling customer financial information means one session could reach another session's data. Security testing found unacceptable authorisation violations, and the supplier had run neither a vulnerability assessment nor penetration testing.
Every one of those is invisible to a component test and fatal to a journey. Race conditions do not appear until two components run against each other under real timing, session bleed does not appear until sessions are concurrent, and authorisation holes do not appear until a request travels the whole path a customer would.
The outcome
What this engagement produced was findings and recommendations, plus a proposed follow-up workshop on lessons learned for both parties. The project was already over. The code review across the two builds found no significant improvement in quality or robustness sufficient to allow acceptance testing to proceed. When acceptance testing was due, the application was described inside the review as roughly seventy percent complete, error-prone and unfit to enter it. That percentage is a reported figure carried in quotation marks in the source, not a measurement the reviewers made, and it should be read that way.
Of the ten questions the subsidiary commissioned the review to answer, eight came back as an outright no, one was hedged, and one was a clear yes. The yes was that the client's own requirement specifications met common international practice, which removes the most common defence available to a delivery organisation and puts the failure in the build rather than the brief.
The honest question is what changes if the same programme runs now. Not the contract clause, which stands as written. What changes is that the clause can be enforced by tooling rather than by a monthly meeting.
Test generation assisted by a copilot is already good at the component level, and that is the level that was never the problem here. The useful move is to point the generation at the dependency graph instead of at the file. A build system that knows which upstream contracts a component depends on can refuse to report green on that component while any of them is unverified, and can carry the list of assumptions into the result, so a project manager reads coverage and exposure in the same line. Wiring that refusal into the pipeline rather than into a policy document is what our Platform work is for. That is ordinary engineering, not research: the interface definitions and the lineage between services are already in the repository. The work is deciding that an unverified upstream turns a pass into a hold, and then not overriding it when a date is close.
The same discipline makes early and carefully scoped autonomous experiments in delivery safe to run at all. A system permitted to propose or repair tests on its own needs an evaluation set it did not write and a lineage trail showing what it changed and why. The political agreement reached on the EU AI Act late last year points the same way for the systems it will eventually cover, and a bank already keeping that trail for its build pipeline will find those documentation obligations easier when they land. Our Consult engagements start by asking a delivery organisation to show which of its green results can name their own dependencies. Most cannot, and that answer arrives in an hour rather than at acceptance testing.
Six sprints of component testing and one acceptance test the software could not enter is not a story about a careless supplier. It is a story about two organisations measuring different things, in good faith, for the whole build.