An Honest Scope, and an Obsolete One

Every assurance report contains one line that decides what the rest of it is worth, and it is never a finding. It is the scope statement, and most readers walk straight past it to the red boxes.

In a post-mortem review of a troubled outsourced build for a European retail and private banking group, that line said the technical workstream would cover the software development process including a sample technical review of code. A slide further in said the same thing without the softening: this was a high level review of key aspects and did not deep dive into the code.

I wrote that section. I still think it was the right sentence to put on the page. It is also, at this point, a sentence about a world that no longer exists.

What a five-week clock actually buys

Look at what the review covered completely and what it sampled, and the asymmetry tells you everything about the economics rather than about the reviewers.

The documentary side was handled artefact by artefact. Every entry on the register carries a comment against it, most of them the single word reviewed, and several carry a comment saying what reading it changed: the non-functional requirements were noted as saying mobile capability did not need to be worried about, which could have been misread in isolation; the plan carried an optimistic and a pessimistic variant and the project stayed on the optimistic one; the risk log was updated throughout. That is what artefact-by-artefact coverage looks like. The register can account for each document one line at a time, because the set of documents was finite and somebody worked down it.

The code got a sample, because a codebase is not a finite pile of documents. Fifteen artefacts can be read in a week. A codebase cannot be read in a week, or in five, and no honest scope statement in that period pretended otherwise. The choice was between sampling and saying nothing, and sampling is the better of those two.

So the review said sample, and then did the most useful thing available inside the constraint: it read what it could, tied what it found to the tracker the engineers already used, and left the rest alone.

The rubric asked corpus questions

The checklist the technical workstream scored against is the part that has aged into something uncomfortable, because it was already asking questions that no sample can answer.

Its dimensions were the ordinary engineering ones, and not one of them was exotic. What matters is not which they were. It is that almost every question on the sheet is a question about a repository rather than about a file. Whether a design principle holds is a claim about all the places it could have been broken. Whether the intended architecture was honoured is a claim about where every piece of logic ended up. Whether failures are handled is a claim about the paths nobody opened. Whether data is sitting somewhere it should not be is a claim about everywhere.

Read the sheet as an engineer rather than as a reviewer and the shape of the problem is plain. Almost none of what it asks is a property of a file. Almost all of it is a property of a corpus.

Duplication is the cleanest case, and duplication had a line of its own on the findings pages. A clone is a relationship between two places in a repository. Sample one of those places and you have learned precisely nothing about the other, and you cannot know from inside the sample whether the sample was unlucky. Caching sat on the same page, and the summary of findings recorded caching of customer data and absent error handling as things that should not have existed at the point of handover. A data-handling defect has no preferred location. It sits wherever somebody wrote it.

The sample was honest and it was structurally blind to exactly the class of defect the engagement most needed to find. Both of those things are true, and holding them together is the whole point of this piece.

The constraint moved

The expensive input in that review was senior engineering attention measured in reading hours. That input has become the cheap one, and it happened quickly enough that most methodologies have not noticed.

A repository can be chunked, embedded and held in a vector store. Retrieval-augmented reading lets a model answer a rubric question against every file that bears on it rather than against the files a person had time to open. The rubric itself stops being a checklist a reviewer carries in their head and becomes an evaluation set: one question per dimension, a stated pass condition, a recorded verdict per file with the evidence attached. Run it once and you have what the five-week plan used to buy. Wire it to the build and it runs on every commit, on code that has not been written yet.

Copilots got all the attention over the last year because writing code is the visible half. Reading code at scale is the half that changes assurance, and it is the more tractable of the two, because reading has a ground truth to check against and drafting does not.

Sampling was never a methodology. It was a budget. The budget changed and the methodology did not notice.

The consequence is not better reviews

If the story ended at reviews getting more thorough it would be a small story. It does not end there.

Sampling is load-bearing in places nobody has been back to inspect. Software assurance methods are built on it: choose a representative slice, weight the choice by risk, extrapolate what you found to what you did not look at. Audit practice is built on it more formally still, with sample selection, sample size, projection from the sample to the population, and a materiality argument explaining why that projection is defensible. Whole professional standards exist to make an unavoidable gap respectable.

Every one of those constructs was engineered around a constraint that has now moved. When reading everything costs less than arguing about which slice was representative, a sample stops being an honest disclosure and starts being a decision that has to be justified on its own terms. Ask the question directly and it sounds almost rude: given what a full read costs today, why did you read a slice of it?

5 weeks

The review window the plan set out, mobilisation to final report

15

Classes of project artefact on the review's register

Sample

The coverage the code review was scoped and sold at

Every commit

What the same rubric can be run against now

What did not get cheaper

The draft I worked from had an evaluation column sitting on the right-hand side of that checklist, and in the version on my screen the column was empty. Somebody still had to fill it.

That column is the job, and coverage does not touch it. A model can tell you that customer data is held client-side in every place it happens. Whether that is a handover blocker, a release-note footnote or an accepted risk is a question about a bank's appetite, its regulator and the date the product goes live. Full coverage makes that harder before it makes it easier, because the volume of findings goes up and the ranking work goes up with it. The reviewer's scarce skill was never reading. It was deciding what a violation costs.

Which is why the discipline around a per-commit rubric matters more than the model behind it. Version the rubric and record which version produced which verdict. Record which commit was assessed and by what. Keep the lineage so a judgement can be re-derived a year later rather than recalled. This is ordinary MLOps hygiene applied to assurance, and without it you have traded an honest sample for an unauditable full read, which is a worse position than the one you started in. The early autonomous experiments worth running here are narrow ones: let the system read the corpus and propose findings against the rubric, keep a named person on the verdict, and keep the scope of what it may conclude on its own tightly drawn. That sequencing is what the RealAI Platform team builds first with clients who want this in their pipeline.

There is a regulatory edge to it too. Political agreement on the European AI Act was reached at the end of last year, and whatever obligations eventually follow from it, the evidence they will ask for is evidence of coverage rather than evidence of effort. A sample, however carefully drawn, is evidence of effort.

The line I would write now

I would not change the scope statement in that deck. It was accurate, it was declared, and it did not claim an inch more than the engagement paid for. Assurance work goes wrong when the scope quietly overstates what was read, and this one did the opposite.

What I would change is what happens next time somebody writes it. A limit that was physical is now commercial, and a commercial limit is a choice. Ask your assurance provider what fraction of the codebase they read. If the answer is a sample, the follow-up question is not whether the sample was well chosen. It is what a full read against the same rubric would cost this year, and why nobody quoted for one.

Drawn from a post-mortem delivery review of an outsourced software build commissioned by a European retail and private banking group: its scope statement, its artefact register, its five-week plan and the checklist of engineering dimensions its technical workstream scored against. That review produced findings and recommendations, not delivered results, and the scope line quoted here is a statement of what the work covered rather than a claim about what it achieved. The reading of that scope line as an obsolete constraint is ours.