The Bugs That Were Easy to Find

There is a sentence in a software assessment I led for a European banking group IT services subsidiary that reads like housekeeping. Race conditions in the front end and static variables on the server side, the finding says, which were easily identifiable, were fixed in August.

The parenthesis is the whole piece. Easily identifiable. Somebody wrote that to signal the problem was under control, and it does: the defects were named, ticketed and repaired by the subsidiary's own engineers within months of the build being assessed.

The route those defects took to get found is the problem. A defect a reviewer identifies easily is, by construction, cheap to detect, and cheap detection is the one thing we have machines for. If such a defect survives every stage of a development process and is finally caught by an outsider reading code, the failure is not in the attention of the people who wrote it. It is that nothing sat between them and the release that was capable of noticing.

What a null check cannot see

The suite was structurally incapable of catching either defect, even at three times the coverage. A race condition is a defect of ordering. Two asynchronous calls return, two handlers write to the same piece of client state, and the result depends on which lands second. The correct interleaving and the wrong one produce the same shape. Both are objects, both non-null, both with the same number of fields. A test asserting that a result exists and has a length passes on both, every time, and reports green.

Server-side shared state has the same property from the other direction. A static field in a request-handling class only misbehaves when two requests are in flight together, and a unit test runs one. The field is correct in isolation, correct under the test, and wrong in production the first afternoon two advisers open the same screen.

So the coverage numbers and the concurrency findings are one observation stated twice. About 14 percent measured on the server, probably under 10 percent estimated on the front end, and the tests that did exist asserting null and length rather than content. That is a suite that can confirm code ran. It cannot confirm code was right, and under concurrency it cannot confirm anything at all.

One caveat, because the source shows its own working: the whole testing finding, both coverage numbers and the null-and-length remark with them, is written in the red the author reserved elsewhere for open action items addressed to named colleagues. The 14 percent came from a named measurement tool and is the hardest number in the document, but it sits in the ink of something not yet closed out. Read it as an order of magnitude.

Nobody wrote the concurrency

The concurrency finding carries a second thing that matters more than the defects. It bounds the risk before listing it: the application was not massively parallel, so there were no starvation issues and no threads ending up starved. That is a discipline I wish more assessments had, naming which failure classes do not apply rather than reciting a textbook.

Then comes a sentence the source garbles. It says there is explicit thread programming and that it was the implicit logic that was the challenge. In a web application with no thread management of its own, that is almost certainly missing a negation. I quote the ambiguity rather than repair it, because the reading that survives either way is the useful one: the difficulty was implicit concurrency.

Nobody on that team wrote a thread or made a decision about locking. The concurrency arrived as a by-product of asynchronous calls in a browser and of a field marked static in a class serving many requests. There is no moment in that story where a developer looks at their own change and sees a concurrency decision to get right, which is why the defect class needs a machine rather than more care.

Why the suite was never going to exist

The design section of the same review explains why nobody wrote better tests, and it is not laziness. The application had hundreds of classes described as very large and responsible for many actions. Many controllers ran to around 2,000 lines. One large cyclic dependency made layer separation impossible and left the codebase monolithic, and dependencies ran between service and controller in the wrong direction.

Code with that shape has no cheap seams. To unit-test a 2,000 line controller inside a dependency cycle you have to stand up most of the application, so the test is slow, so it does not run on every change, so it is written once and quietly skipped. Coverage of 14 percent is not a moral fact about a team. It is the arithmetic consequence of an architecture that made the twentieth test cost more than the first.

That is the loop worth naming, because it closes. The structure prevents a cheap test suite, the missing suite lets easily identifiable defects through, and those defects are found by a reviewer months later at consultant rates.

Built against mocks

A separate technical document from the same programme shows where the front end's timing assumptions came from. It inventories the services the application depends on with a status against each. Of twenty-four listed, two ran against a real service bus, two were mocked and two more were marked as mocks coming soon, eight were specified with no implementation behind the specification, nine were not defined at all, and one was under research.

A front end assembled against that inventory learns its timing from mocks. Mocks return in a millisecond, in the order they were called, and never fail halfway. Every ordering assumption formed there stays untested until real services with real latency arrive, and by then it is spread across many handlers written by many people. Front-end race conditions are the exact defect this manufactures, and they were easy to find later precisely because nothing had been able to find them earlier.

~14%

Measured server-side code coverage, in a finding its author had not closed out

Below 10%

Front-end coverage, an estimate rather than a measurement

Null and length

What most existing tests actually asserted

2 of 24

Dependency services on a real service bus rather than mocked, specified or undefined

The audit was doing the machine's job

Line the dates up. The build was assessed in May, the concurrency and shared-state defects were fixed in August, and the findings were written up in December. By the time the document reached the people paying for it, its most concrete performance finding described repairs that had already happened. That is no criticism of the reviewer, who marked every page as a point-in-time assessment and said so. It is a criticism of using a point-in-time human review as the primary detection mechanism for defect classes a pipeline can check continuously and for nothing.

Exception handling in the same review produced four logged findings across three failure modes: swallowed errors, inconsistent handling, and no handling at all. The logging aspect carried nothing, and the document never says whether that means clean or never reached, so I claim neither. Note the pairing anyway. Swallowed exceptions plus unexamined logging is a system that can fail silently in an advisory workflow, where the visible consequence is an adviser shown a screen that quietly did not work.

For a bank that is a risk and compliance question rather than an engineering one. Straight-through processing rests on the proposition that a case passing through untouched passed its checks. A swallowed exception breaks that proposition without breaking anything a user can see.

A defect a reviewer finds in an afternoon is a defect a machine could have found in a second. Its survival all the way to an independent review is not a story about the people who wrote the code. It is a story about what nobody built to check it.

The rule this leaves behind

Every defect class you can name, you can test for. Every one you can test for and do not will be found by a person, later, at the worst available price, often after somebody has already fixed it.

The practical version is short. Search for shared mutable state in request-scoped code and fail the build on it, because that is a lexical check rather than a debate. Assert on content, not on shape, so ordering defects have somewhere to fail. Run the front end against services that are slow and occasionally wrong before real users do. Refactor for testability when a controller has outgrown any test that could reach it. And put every one of those checks in the pipeline rather than in a review calendar.

None of that needed inventing when this review was written. What made it hard was that the code's structure and the dependency environment had already priced the checks out of reach, which is why the first repair is architectural and the second procedural. That sequencing is where RealAI Platform work usually starts on an estate like this one: make the checks cheap before mandating them, or the mandate produces theatre.

The same logic is about to matter more than it did for a web application. Machine learning pipelines are assembled from exactly the ingredients that produce these defects: shared mutable state in feature transforms, ordering dependencies between a training job and the store it reads, and long chains where a swallowed error propagates as a plausible number rather than a stack trace. As modelling gets cheaper and code gets written faster, the detection layer stops being hygiene and becomes the only thing standing between a cheap defect and a regulated decision.

Findings are as recorded in an independent software development process assessment of a front-office advisory application at a European banking group IT services subsidiary, and in a companion technical overview from the same programme. The assessment was an explicitly point-in-time review of one build: it produced findings and recommendations, not delivered results, and several of its pages carry logged defects with no written narrative. Reading its concurrency finding as a statement about test tooling rather than about developers is ours.