Traceable On Paper

The most useful column in a technical review is the narrow one on the right. It holds a reference: an identifier in a defect tracker somebody can open, a clause in the statement of work, a measurement and the tool that produced it. Everything to its left is argument. That column is what makes the argument checkable by someone who was not in the room.

I went back through a software development process review I led for a European retail and private banking group, an independent assessment of a new advisory application built for them by an outside supplier. The instrument was a grid, a page per topic grouped into quality areas: design, performance, security, architecture, testing, deployment and support, and a general section at the end. Each page takes its area apart into the particular things examined under it, writes findings against them, and binds each one to a reference in the group's defect tracker.

Eighteen pages, thirty-five distinct tracker references spread across twelve of them. Read as a delivery document it is conventional, competent assurance work. Read as a claim about evidence, there is a hole in it, and the hole sits on the page that carries the most finished work.

What the grid promises

The design standard was not a matter of taste. It was contractual: a named set of object-oriented design principles agreed with the supplier, with an internal quality process behind it meant to check that anything built came out compliant. That makes the review a measurement against a term somebody signed rather than one engineer's preference against another's.

The grid is the mechanism that keeps it that way. A quality area, the particular thing examined under it, the finding written against it, the reference that backs the finding. Read the reference, open the item, see the code, disagree if you like. The point of the layout is that a reader never has to take the reviewer's word for anything.

Auditors assume that, and so does the supplier being assessed, because the reference is the only thing standing between a finding and an argument about whether it is real.

The drift runs both ways

Take the untracked one first. The narrative reports a service that goes out through a REST interface for data it should have taken from the layer beneath it. Structurally that is the most quietly expensive defect on the page, because it is how layer separation stops being true, and everything else follows from layers that will not separate. There is one large cyclic dependency, so nothing can be pulled apart, so the result is monolithic.

That finding has no reference. If somebody had contested it months later, there is nothing to open: the reviewer would have had to find the code again, from memory, in a build that had moved.

Now the other direction. The table has a row for the observer pattern with an identifier against it, and nowhere in eighteen pages does anyone say what is wrong with the observer pattern here. The evidence exists and the argument does not. A reader who wants the finding has to leave the document and read a tracker item written by somebody else for another purpose.

Neither of these is a slip. They are the arithmetic of keeping two representations of one body of evidence by hand. Prose gets written in one pass, in the order the reviewer thought of things; the table gets filled from the tracker, in the order the tracker returned them. Both are correct on their own terms, and reconciling them is a separate job nobody had scheduled.

One detail sharpens it. On this page, and no other, the references are live links; everywhere else the identifiers are inert text. Traceability was clickable on one page in eighteen, and that page is the one where it does not close.

Nobody was careless

I want to be precise about the accusation, because there is not one. The review was thorough. The cyclic dependency finding was right, the class size finding was right, and both are what comes of someone reading real code rather than running a report. The instrument is better than most: it insists on a reference, and it footnotes every page as a point in time reading of a build already months old when the findings circulated.

The failure is structural. Anyone keeping a narrative and a tracker as two parallel shapes of one truth will drift both ways, and neither shape holds what is needed to detect it. There is no field for a claim with no artefact behind it, and none for an artefact with no claim in front of it. The document cannot report on its own integrity, so the reader assumes a completeness the page never asserted.

That is the normal state of most review findings. I have read a lot of these and written a fair number, and the surprise is not that this one drifted. It is where it drifted: on the page somebody had taken furthest.

Why this matters more now than it did

Every assurance function I have spoken to this year wants an assistant over its own back catalogue. Years of reviews, audits, remediation plans and steering packs behind retrieval, so somebody can ask what we found last time and get an answer in a minute rather than a fortnight. It is a genuinely good use case, and it is the one this page breaks.

Retrieval-augmented generation over a vector store of documents like this one finds the narrative every time, because prose is what embeds well. A tracker identifier is a short opaque string with almost no meaning to an embedding, in a table cell whose column header the extraction probably dropped. So retrieval surfaces the sentence and loses the reference, and a copilot restates the untracked violation with exactly the confidence it gives the three tracked ones. One of the four was never checkable, and the assistant cannot tell which.

It gets worse than losing the binding. A flat text extraction of this page appends the narrative into the coupling row, because the narrative sits in a floating text box and the extractor reads by position in the file. Fed to a retrieval pipeline, the document does not merely drop the link between claim and evidence. It asserts a wrong one, tidily.

So the evaluation set for a review assistant is not questions about content. It is assertions, each with one question attached: does this resolve to an artefact, and which one. That is a lineage problem, the same one that stalls MLOps work everywhere else. Data readiness here does not mean having the documents. Everybody has the documents. It means them carrying, in a form a machine can follow, the binding they always claimed to carry. That is the work RealAI's Platform team does before a retrieval pilot rather than after it, because an assistant built over an estate that cannot substantiate its own citations inherits the problem at speed.

4 of 18

Pages in the review carrying written findings

1 of 4

Violations on the design page with no tracker reference

6

Tracker references on that page, three never mentioned in the prose

1 of 18

Pages where the references are live links

There is a regulatory tailwind, and I would rather state it carefully than oversell it. The European AI Act is in force, and its obligations arrive in stages on a published schedule rather than all at once. Almost none of them bind anyone yet, and the direction of travel on documentation and record keeping is not ambiguous. The habit is worth acquiring early anyway, for the ordinary reason that a finding you cannot substantiate is one you will eventually lose an argument about.

A statement with a reference behind it is a finding. The same statement without one is an opinion held by a good engineer on a Tuesday. They look identical on the page, and only one of them survives being questioned six months later.

What we would fix, and in what order

None of this needs a model, which is the awkward part, because all of it has to land before a model is worth pointing at the estate.

One record per finding, with the sentence and the row as two views of it rather than two copies. Write once, render twice. Nothing publishes without a resolvable reference, and a claim that has none gets rendered as what it is, a hypothesis with an owner and a date, rather than as a peer of the ones that do. A status per assertion instead of per page: tracked, measured, checked against a contract clause, or unverified. References as links rather than strings, on every page. And a check over the finished document in both directions, listing every claim without a reference and every reference without a claim, before anyone circulates it.

Answer those five and you have found nothing new. You have a document that can be interrogated, which is the precondition for the assistant everyone wants next and for the early, carefully scoped autonomous experiments that follow it. The tempting one is something that reads a draft review and raises the missing tracker items itself. That works only where the drift is detectable at all. The bottleneck was never the writing. It was that the document had no idea what it could not prove.

Drawn from an independent software development process review of a new advisory application at a European retail and private banking group: its assessment grid, its written findings and its traceability column, read as recorded. That review produced findings and recommendations against a build already several months old when it was written up, not delivered results. Reading its internal drift as a data readiness problem is ours.