Breaking the game on purpose: thirteen checks we have proved can fail
For most of this project’s life our checks could tell us one thing: the code ran. Whether a check was capable of failing at all was a question nothing here ever asked. A check that cannot fail clears every gate in exactly the same green as a check that can, and ours had been doing it for months. In July we finished the machinery that asks.
The clearest case had turned up ten days earlier, in a check we wrote ourselves, about insects.
Green by construction
On the twelfth we added two species for the coldest country in the world: a bumblebee that pollinates where nothing else does, and a carrion beetle for the column where bodies rot slowest. The animals themselves are written up elsewhere. What matters here is the promise that comes attached to any new species: adding one must not shift where any other species is placed. Seed a world with the newcomers and seed it without them, and every older animal has to land on exactly the same ground.
The first version of that proof took the two newcomers out of the roster and compared the placements against a run with the full roster. It passed. It could not have done anything else. The insects were the last two entries in the roster, so removing them truncated the tail of the list and disturbed nothing ahead of it. Every older species drew in the same order in both runs, and the comparison came back identical because the arrangement guaranteed it would.
The check had not been passing because the placement was steady. It had been passing because we had arranged for it to.
So we rebuilt it the other way round. The newcomers go to the front of the roster now, where their presence pushes everything behind them if anything is going to move at all. Then we settled which version had teeth by writing a throwaway copy of the placement code with a deliberate defect inside it, and running both versions of the check against the damaged copy. Of the fifty-three species that copy disturbed, the old arrangement detected none. The new one detected fifteen.
Only after that did the real result carry any weight: fifty-three existing species landing on identical ground with the insects present and with them absent, across four world seeds, every pairing compared, not one difference between them. Five days later the plants gave us the same shape from the other end: the scenario the canopy check runs against had been cloned from the oak, sterile pairing and all, so for seven weeks it had been certifying the defect it existed to catch.
A twin wired to be wrong
A harder version of the same problem had come up at the end of June, on the boundary that decides which far-off creatures keep being simulated one animal at a time and which of them fold into a summary. The rule there is that the boundary must never depend on where the player’s camera happens to be. A world whose population turns on which way you looked is not reproducible, and we had already found and closed one hole of exactly that shape.
“The answer never changes” is a claim that is very easy to prove badly. A fixture reporting the same value twice cannot, by itself, tell a rule that ignores the camera apart from a fixture that never really moved the camera. So alongside the real rule we wrote a deliberately broken twin: the same partition, wired to the camera on purpose, run through the same fixture, and required to return two different answers from two camera positions. Once the fixture had been shown capable of catching camera coupling, the real rule’s steadiness became a result rather than a triviality.
By the middle of July the practice was going into plans before any code was written. The plan for making one world hold every climate at once required its new test of whether a species can live somewhere to be tried first against a fault we already knew was in the world, on the grounds that a fault inside an instrument manufactures evidence. That same week, every erosion check was broken by hand and made to go red before we trusted it again. And when a friendly default on the climate maths left a caller that had forgotten to say which world it was in with nothing able to object to it, the repair was not another check but a build made to name every caller.
One line, one check, one exact complaint
What closed on the twenty-second turns that habit into standing machinery. Every check in the quick set that runs after each change now has a place in a written register. For the ones we have done the work for, a person writes down the exact one-line edit to our own source that must make that check fail, and why that particular edit falsifies that particular claim. The reason is mandatory and it has to be written by hand — a register assembled from the tool’s own output would be an echo of the tool, and would prove nothing.
A run applies one edit at a time, runs only the check that edit is meant to break, and requires it to fail carrying its own distinctive words: a failure for some other reason counts as no proof at all. Then it puts the file back and checks the restoration against the original byte for byte, using two independent readers rather than one trusted tool. Before each edit it writes down what it is about to damage, so a run killed halfway through refuses to start again until the damage has been named and undone.
There are three verdicts, not two. A check can be proved: broken on purpose, red for the declared reason, restored. It can be falsified, which is the finding the whole exercise exists to produce, where the code is wrong and the check does not object. Or nothing was established in either direction, which gets reported as itself rather than dressed up as one of the other two.
The control, and the warning not to fix it
A machine that breaks things has one failure mode worse than anything it inspects: applying nothing at all, and then reporting every check in the game as sound. So each run opens with a permanent control before it touches real code. One edit, two checks, opposite required outcomes. The sensitive one has to go red. Its insensitive twin has to stay green, and if that twin reds as well, the edit broke the file outright and every proof in that run is a fraud. A third edit exists to show the insensitive twin is not simply dead.
The register admits the recursion in its own header. Nothing mechanical proves the control meaningful; a person proved it once, by hand, in three directions, and that hand-check is the floor everything above it stands on. The file also carries a warning against tidying it, because tightening the control would look like an improvement and would delete the only way we have of telling a real proof from a broken file.
Thirteen against two hundred and twenty-two
The number the register publishes is the uncomfortable one. Thirteen checks now carry a written, verified demonstration that they can fail. Two hundred and twenty-two do not.
That second list is bare, sorted, and free of excuses, because a line on it makes no claim whatsoever. It is an admission of absence, and it is the work list of record.
Read that number, not the green one.
Partial coverage is deliberately a passing state. The count ratchets: it can climb and never fall, so the hardening cannot regress without anyone noticing, and adding a brand-new check to the game costs one line on the unproven list and no argument at all. Nobody should ever be taxed for writing a test. What a clean sweep means, in return, is narrow enough to be worth saying out loud: it is a statement about the thirteen, and about nothing else.
What it still cannot see
The register prints its own limits where they cannot be missed. Coverage is recorded per file rather than per claim, so a file marked as covered may hold nineteen further assertions behind its one proved break, every one of them hollow. The break is authored, so it is only ever as good as the person who wrote it. And it finds false coverage, never missing coverage: it can tell you that a check you own is empty, and it will never once mention the check you never wrote.
Building it produced a live specimen of the thing it hunts. The tool was reading the checks’ output through a route that mangled a dash inside their own failure messages, so the distinctive phrase it matched on never appeared, and three results came back as nothing established while their edits had applied perfectly and the checks had failed exactly as declared. Same edits, same checks, one different reading route, and all three turned into proofs. The instrument was blind, and its blindness looked precisely like an honest negative — the same failure we had just audited a fleet of instruments for.
None of the game’s own code changed across this arc; the only edits it ever made to the world were the ones it undid a second later. What changed is what we are able to notice. Our catalogue of green checks that turned out to be guarding nothing, and of green boards measuring the wrong axis, used to grow one embarrassment at a time, found by reading closely and getting lucky. There is now a list of two hundred and twenty-two candidates, and a machine that can put the question to them one at a time.
