The Data Audit That Came Before the Model

Every service desk conversation I am in this year ends in the same place. Somebody asks whether a model could triage the tickets, predict which ones will breach, or deflect the calls that never needed a human. The question is fair. Deployment tooling has never been this available, feature stores are something you buy rather than build, and the chat assistants that landed over the winter have made a demo cheap enough that nobody has to justify asking for one.

What almost nobody asks first is whether the ticket data would survive contact with a training pipeline.

I keep going back to a service management review of an outsourced IT service desk running across several sites at a large European public-sector organisation. It was not an AI engagement: contracted service levels, processes, governance, a maturity assessment, two dozen stakeholder interviews. Read it as an operations person and it is a competent account of a desk under strain. Read it as someone who has to build a machine learning pipeline on top of that desk and it is the clearest document I know about why these projects stall before anyone deploys anything.

The headline numbers did not match their own charts

Six service levels were reported across two slides, each with an industry figure, a contract target, a three-month average and a trend chart beside it. Three of the six averages reconcile against those charts. Two do not: speed to answer is stated at 17.42 seconds while the charted months work out to 17.71, and service level is stated at 94.9 percent while the same months average 94.09. The sixth has no trend data behind it at all, which is its own section below.

Nobody was cheating. The gaps are far too small for that, and one moves in the direction that makes performance look worse. What they record is more ordinary and more damaging: two reporting paths, neither authoritative. The supplier produced its own set of reports, partly from the telephony system and partly self-prepared, while the client pulled automated reports out of the ticketing tool. Both were real, neither reconciled to the other, and nothing said which one governed.

That is a lineage failure, and lineage is the part of data readiness teams skip because it produces no visible artifact. A feature store is, in the end, a promise about where a number came from. If your source of truth for handle time is whichever of two systems the person building the pack reached for that month, every feature derived from it inherits a coin flip. The model trains happily. It is learning the reporting process rather than the desk.

The market study bound into the same document repeated the pattern. Its comparison ranges across six metrics carried a note saying plainly that these were not performance measurements: they were vendor answers to a questionnaire, best value taken where several were given. A benchmark of claims.

Three volumes for the same month

Within three consecutive exhibits, the same nominal month appears as 4,620 contacts, 4,837 tickets and 5,784 incidents, with nothing defining the difference between them. A separate exhibit states 844 escalations for that month while the chart beneath it sums to 703, and that chart already carries an "others" bar of 141 rolling up every group with fewer than ten escalations. So the filter is not the explanation: the roll-up is inside the 703, and 141 is still missing. The stated total and the drawn total were produced by two different passes over the same table, and neither pass was written down where either number was read.

Now compute an escalation rate. Is it 844 over 4,620, or 703 over 5,784, or one of the six other combinations? Those answers differ enough to change the conclusion, and the ambiguity propagates into first call resolution, cost per incident and any breach probability a model would estimate. A model trained on escalation rate here learns a definition someone chose without writing it down.

This is the part of a data audit that pays back fastest and needs no data science at all. One definition per counted thing, the query that produces it stored beside it, filters declared where the chart is read.

The label the tool could not carry

The contract defined five priority levels with acknowledgement and resolution targets attached to each. The ticketing tool supported three, set by impact. Framework and tooling were never reconciled, so contracted resolution compliance was not merely unmeasured, it was unmeasurable. Both trend panels carried "no information available", and acknowledgement had neither a target nor an actual.

Then the second problem. Close to 95 percent of incidents were recorded as low priority, and three of the four ticket types recorded no high-priority items at all in the sampled period. Priority is the most attractive label on a service desk: it is what a triage model predicts and what a breach model conditions on. Here it is very nearly a constant. A classifier reaches 95 percent accuracy by learning to say low, and is worth nothing.

One call resolution says the same from another angle. It came in at 47 percent against a contract target above 65 and an industry figure of 75, and the review recorded, in as many words, confusion about how it was being measured. The figure was labelled a three-month average and carried a different single month in brackets. Before you model a rate, somebody has to say what the rate counts.

Filler that looks like data

Three chart panels in that chapter share an identical 90-point series running between 3.0 and 6.9 against sequential day numbers. It is template filler, pasted three times, sitting behind the panels the deck itself stamped as having no information available. On a projector it reads as a trend. The process tables in the same section carry the matching artifact in prose: an author's placeholder, "need how this is managed currently", left standing in the incident, request and problem management rows, with the problem management row holding nothing else.

None of that is scandalous in a document meant for people, who see the stamp and move on. It matters because the ingestion job does not see the stamp. Point a scraper at a reporting pack and it lifts the dummy series as confidently as the real one, and the resulting feature is smooth, plausible and fictional.

When money attaches to a number

The same organisation ran a different desk under a credit-point regime: fifty points a month, ten to twenty deducted per recorded error, plus a service level bonus worth five percent of contract value. A sensible commercial arrangement, and a warning label on the data. Any field with money attached becomes a managed one rather than a neutral observation. The two KPIs named in that regime were accuracy of incident categorisation and recording the incident within twenty minutes: exactly the fields a triage model wants, and exactly the fields under pressure. Never treat a bonus-bearing field as ground truth without a second, unincentivised source to check it against.

The blind spots were operationally real too. Agent utilisation was not a contracted metric at all, and staffing sat at exactly 7.0 agents a day for seventeen consecutive months while monthly demand swung from 3,592 to 7,554 incidents. Then unit cost jumped from about 20 euros an incident to about 48 across two consecutive months, when a second supplier was added for proactive floor support. The review proposed utilisation as the metric that explains unit cost, and on its own numbers the largest move in unit cost came from staff who were never in the pool utilisation counts. A forecast built on that pairing inherits the blind spot.

The signal was there the whole time

It would be the wrong lesson to conclude that this desk had no learnable structure. Nearly four in five contacts arrived by telephone while self-service accounted for three in a thousand, which the review put down to portal usability rather than user preference. About one incident in five escalated beyond the first line. One monitoring category recorded 202 incidents and 202 escalations, a 100 percent rate, which is a routing rule somebody should have written years earlier, not a modelling problem. Below it sat forty more categories running from 190 incidents down to six, escalating between five and forty times each: a spread wide enough to show exactly where first line capability stops, once somebody agrees what a category is.

Every one of those is actionable this quarter without a model, and several are the seed of a useful one. What stood in the way was not a shortage of data.

2 of 6

Headline service levels that did not match their own charts

3

Different volumes reported for the same month

~95%

Of incidents recorded at a single priority

202/202

Incidents to escalations in one category

What the audit should have asked

Seven questions, none needing a data scientist. One definition per counted thing, with the query that produces it. One authoritative path per number, traced to the system that emitted it. Label integrity: does the field you intend to predict actually vary. Coverage: which contracted metrics have no measured actuals, and can the tooling express them. Provenance for every benchmark: measured, or self-reported by whoever benefits. Filters declared where the chart is read. A pass for template series and drafting artifacts before ingestion.

Answering those does not build a model. It tells you which parts of your estate can carry one, which is why we run the seven as the opening block of every RealAI Platform engagement and why our Consult team asks for a metric rather than a use case. On this desk the honest answer would have been that call handling metrics were ready, priority and resolution were not, and cost per incident needed a definition before it needed a forecast.

Nobody could tell you which numbers on the page were measurements. That is the finding, and it sits upstream of every model anyone wanted to build.

The temptation now is to skip all of it, because the modelling end has never been easier. Pipelines are assembly work, deployment is close to solved in most stacks, and the first language model demos crossing my desk stand up in days rather than quarters. That is why the audit matters more than it did five years ago, not less. When the model was the expensive part, the data got scrutinised on the way in. The model got cheap and the scrutiny went with it.

Figures are as recorded in an outsourced IM service desk review at a large European public-sector organisation, covering the reporting months to March 2015. That document is a sourcing diagnosis, not a delivered programme: nothing here is a result we produced. The reconciliations against the charts, and the escalation arithmetic, are ours.