Why a Bank AI Scorecard Reports Test Coverage and Not Time to Restore

Most AI programmes inside a large bank report themselves into four different rooms. Value goes to a finance business partner. Adoption goes to a change lead. Compliance health goes to a control forum. Delivery performance goes to an engineering review nobody outside technology attends. Four owners, four cadences, and no single page on which a slip in one is visible beside a slip in another.

A board deliverable I worked on for a consumer and SME banking group operating across several European markets carries, inside its execution section, a page that refuses that split. It is an executive scorecard, headed as a month-end view, fed from the centre of excellence report, and reviewed by the AI oversight committee on the third Thursday of every month. On it sit net value realised, cycle time from idea to live, critical incidents, active users, a year-to-date panel of value against cost, a risk and compliance health score derived from EU AI Act readiness audits, and a delivery panel labelled SDLC health carrying deployment frequency, change failure rate and test coverage.

That last panel is why I stopped on the slide. Deployment frequency and change failure rate are numbers a delivery lead reports upward to a CTO. Here they are handed to a committee whose remit is oversight, in an institution where a change to a live system is a supervised event. The instinct is right. One substitution inside it is not.

One caution about the figures, because it decides what this piece may claim. The scorecard is dated January 2026 and carries a value in every cell: 4.2 million euro of net value realised at 12% above target, 5.2 weeks from idea to live, zero critical incidents in ninety days, 1,240 active users growing 8% month on month, a change failure rate of 1.2% marked excellent, and test coverage at 78%. None of them is a measurement. Two pages later the same deck asks the board to greenlight the pilots, authorise the funding envelope and formalise the oversight committee itself, all pending signature. Whether the numbers are projections or plausible values dropped in to show the layout, the slide does not say. The same ambiguity in the sister dashboard from this deck is worked through in What Changes When a Training Badge Becomes a Condition of Promotion, and I will not repeat that argument.

What survives as evidence is the panel design: which measures a bank chose to put in front of its oversight committee every month, and which it left off.

4

Reporting worlds fused onto one monthly page: value, adoption, compliance health, delivery

3

Delivery measures present in some form; time to restore service is not one

90

Days in the incident window, reported as a count with no duration attached

0

Months of this scorecard run at the point of writing; the deck is a design

The delivery panel is three quarters of a standard set

The software delivery research of the last decade settled on four measures, reported together because each covers a failure the others hide. How often you ship. How long a change takes to reach live. How often a change breaks something. How long it takes to get back.

Read the scorecard as a whole page rather than panel by panel and three of the four are present. Deployment frequency is there, rated High at weekly. Change failure rate is there at 1.2%. Lead time is there in coarser form as the 5.2 week idea-to-live cycle time, which measures a longer arc than lead time for changes but answers a version of the same question.

The fourth is absent. Time to restore service does not appear on the page in any form. The slot where it would sit is occupied by test coverage.

That swap is not neutral. Of the four, time to restore is the only one describing what happens after something has gone wrong, which makes it the only one speaking the language of the committee reading it. An AI oversight committee in an EU bank is not primarily a productivity forum. It sits downstream of an operational resilience regime built around recovery objectives, in which the supervisory question about any important service is how quickly it comes back, evidenced, and within what tolerance. Three speed and quality measures with no recovery measure is an instrument designed for an engineering audience and then placed in front of a risk audience unchanged.

A count of critical incidents is not a recovery time

The obvious objection is that recovery is already covered by the tile reporting zero critical incidents in the last ninety days with a status of stable. It is not.

An incident count answers how often. A recovery metric answers how long. Those properties move independently: a quarter with two incidents lasting eleven minutes each and a quarter with one incident lasting two days rank in the wrong order on a count. The count also has a floor problem, because while the number is zero nothing has exercised the recovery capability. A zero that has never been tested and a zero backed by a rehearsed restore path render identically on the tile.

Weekly deployment into regulated systems sharpens this. Shipping often is defensible, because small frequent changes are easier to reason about and easier to reverse than quarterly batches. The property that makes them safe is reversibility, and reversibility is measured in restore time. A page reporting weekly deployment and no restore time reports the half of that argument that sounds good.

A count of incidents tells a committee how often the system broke. It says nothing about how long the bank was without it, and the second question is the one an operational resilience examiner asks first.

Test coverage is a build metric wearing an assurance badge

Coverage at 78% is not a bad number to track. It is a bad number to put on a governance page unqualified, for two reasons.

Coverage does not say what is covered. The figure most tooling emits is line or branch coverage of application code. For an AI portfolio the tests a committee cares about are a different set: model validation results, bias and drift checks, the evidence pack required at each lifecycle gate. Those live in another pipeline and produce another number. A committee reading 78% beside a compliance health score will reasonably assume the two are related, and nothing on the slide says they are.

Coverage is also the most gameable number on the panel. It can be raised without improving assurance by anyone who understands how it is calculated, which is everyone whose performance it now touches. Change failure rate resists that, because production decides it.

None of this argues for removing the delivery panel. It argues that a build hygiene number is the wrong thing to have traded away a resilience measure for.

The acronym collides with the regulation

A practical warning for anyone reproducing this panel in a financial services pack. The four delivery measures are widely known by a four-letter acronym, and in an EU banking room that acronym already means the Digital Operational Resilience Act, binding since January 2025.

Put the abbreviation on a slide in that room and half the audience hears a regulation while the other half hears an engineering framework, and the two readings never reconcile. The bank's slide sidesteps this by labelling the panel SDLC health. Write the measures out in full, and keep the acronym out of anything reaching a governance forum.

What I would change before this page reaches a committee

Carry time to restore service, beside the incident count rather than instead of it. If the programme is too young to have a measured restore time, report the tested one from the last recovery exercise and label it as such, because an untested recovery objective is a plan rather than a metric.

Put the threshold on the page next to the verdict. Three tiles ship a judgement as well as a value: deployment frequency rated High, change failure rate rated excellent, incidents marked stable. A judgement implies a boundary somebody set, and no boundary appears on the slide. When the number and the word beside it are compiled by the same team each month, the word is the part that drifts.

Say which pipeline the coverage figure comes from, or replace it with the model validation pass rate at the lifecycle gates, which is the assurance number the committee believes it is already reading.

Label the period on every tile. The header says month-end while one panel is explicitly year-to-date, and mixed periods on one page produce arguments that look like disagreements about performance and are actually disagreements about dates.

The board in this deck was asked to define the metrics this page reports, and the placement of that ask is the subject of Show the Board Its Three Decisions Before You Show the Roadmap. I am starting one step later, at the page that ask produces. A metric definition stopping at the name of the measure leaves the harder half undone: the panel, the threshold, the period and the pipeline all stay open, and each changes what a committee believes when it reads the tile.

Source: a single executive scorecard slide from a board deliverable and workshop kit written in January 2026 for a consumer and SME banking group operating across several European markets. The slide fuses value realised, cycle time, incidents, adoption, a year-to-date financial panel, a compliance health score and a software delivery health panel onto one page reviewed monthly by the AI oversight committee. It is a proposal and a design; the deck still asks the board to greenlight the pilots and formalise the oversight committee, so nothing on the scorecard had been run and its figures are not measurements. What the source evidences is the choice of panels. The reading that the missing measure is time to restore service, that an incident count does not substitute for it, that test coverage is the weaker occupant of that slot, and the four changes above, are mine.