A review of a failed software delivery is bounded by what it can get its hands on, and that bound is set early, quietly, and usually by whoever is most willing to hand things over. Reviews that start from interviews finish with a narrative the most articulate participants recognise as their own, and the documents arrive late and mostly to confirm it.
A European banking group IT services subsidiary commissioned an independent review of a securities advisory application it had outsourced to a software vendor and then taken back in house after the delivery and the relationship both came apart. The review's evidence appendix is one register of artefacts. Fifteen classes of artefact, each one recorded as obtained and, separately, recorded as read. All fifteen carry both confirmations.
That register is the finding. Not because fifteen is a large number, but because of what sits on the list and what it made possible.
The challenge
The fifteen: the statement of work and its addendums; the master contract with the vendor; the business, technical and non-functional requirements; the high level and low level design documents; the architecture diagram; the plan with its milestones and its two variants; the weekly progress reports; the risk log; the issue log, as a spreadsheet export; the test plan and acceptance criteria; the change log; the defect log; the code fix log; the monthly bills for vendor headcount; and the communications plan, including the committee definitions and the register of approved decision makers.
Thirteen of those are what any competent reviewer asks for. The last two are the ones that decided what the review was able to conclude, and they are the two most post-mortems never request.
Start with the invoices. The note beside that row reads as administration and is not: it records an absence of billing for the architect. The staffing reconstruction shows what that means in days. The technical architect billed four days in the opening month and eighteen in the second, then shows nothing for the eight months that followed. In the busiest month on the same sheet, eight labelled developers and team leads each billed between eighteen and twenty-two days, which is effectively full time, and four further rows billing that month carry no role label at all: two of them at twenty and twenty-two days, two at three days and two days. Twenty-two days of architect time in all, on a build originally allocated more than two and a half thousand person-days and replanned partway through to exceed three thousand.
The review's own statement of the problem named poor quality of code, error handling and application crashes. Calling the absent architect the cause of those would overstate the evidence. It is fair to say a candidate explanation was sitting in monthly invoices that both organisations had already filed and nobody had read across.
Now the communications plan. It was used, the note says, to understand the governance structure and the key people to interview. The roster it produced ran to fifteen people, seven on the client side and eight on the vendor side, and it included both the current sourcing director and the one who had held the role before, so the decision to outsource was interviewed as well as its consequences. Pulling a roster from a governance document rather than from referrals removes the sponsor's ability to curate who you speak to.
It also exposes a gap by arithmetic. Only two of the client-side roles had a counterpart on the vendor side, at architect and project manager level, and nobody on the client roster held a testing or quality role at all, on an engagement whose stated complication was the quality of the code.
The approach
Three things about the method are worth carrying, and all three sit in the register's shape rather than its contents.
The first is that obtaining a document and reading it are recorded as two separate confirmations. Having a document and having read it are different claims, and keeping them apart lets a reader see which artefacts were merely available. Most evidence lists we meet make one of those claims and let the reader assume the other.
The second is that the note kept against each artefact carries findings rather than administration. By our reading, five of the fifteen notes say something substantive. The non-functional requirements state that mobile capability did not need to be worried about, and the reviewer's note is careful and hard: it could have been misunderstood if read in isolation. The plan existed in two variants, an optimistic one and a contingency, and the team stayed on the optimistic version throughout: somebody had done the work of writing the fallback, and nobody had written the condition that would trigger a switch to it. The invoices carry the architect note. The weekly report row records that reports were obtained from both parties, which is the only reason a two-sided comparison of status ratings was possible at all. And the design documents carry the most instructive note of the five: they were shared to confirm a drawing of the system design the reviewer had already made independently. Reading the official design first anchors you to the story the project tells about itself. Drawing it yourself first turns the difference between the two into a finding.
The third is consent. Every interviewee had a slot reserved for an attributed summary, and every slot was held empty until the named individual approved their own words. With fifteen people on opposite sides of a broken contract, and both employers reading, that gate is slow and it is what makes attributed evidence usable rather than dangerous.
The outcome
What this engagement produced is a draft review: an evidence base, findings, an interview programme, and a proposed shared definition of project status colours for future assignments, since client and vendor had plainly been running different scales. The month by month status comparison was designed and its cells left unfilled. The attributed summaries were empty, awaiting approval. Several of the numbers above are our own arithmetic over hand-kept tables. None of it is a delivered outcome, and the recommendations were offered to be validated rather than implemented.
The honest question is what would be done differently now. The cost of ingestion has collapsed. Fifteen artefact classes go into a vector store in an afternoon, and retrieval-augmented generation over that set answers questions about the contract, the requirements and the defect history with citations back to the source. A copilot pointed at a well-bounded evidence base is genuinely good at this, and it is the least contentious use we see inside banks.
Two cautions follow from the same register. Retrieval is only as good as the provenance underneath it: which version of a document you hold, who supplied it, whether anyone has actually read it. Those two confirmations are a lineage record drawn by hand, and most document stores we meet cannot produce their equivalent. That is a data readiness problem before it is a model problem. A review copilot also needs an evaluation set, a list of questions whose answers you established by hand, so you can tell whether a retrieved answer is right rather than merely fluent. The agreed European rules on artificial intelligence point the same way, since what a decision rested on has to remain answerable afterwards.
The Platform work we do starts at that lineage layer rather than at the model, because retrieval over an unversioned document store produces confident answers with no way to check them. Reconciling billed roles against contracted roles, monthly, for every supplier, is the kind of narrow and checkable task that makes a good early and carefully scoped autonomous experiment: structured inputs, a verifiable answer, and a false alarm as the worst failure.
None of that changes the human part. Nobody would have asked for the invoices. The list is the finding, and the list is still a choice somebody makes before the fast machinery starts.