On code you can read for yourself

Three trials on 30 October 2026, on code outside our own project: two parts of Unreal Engine's own C# and the scoring rules of a shipped open-source game. Every number below was measured on these runs, and every defect was reproduced by running it. No test and no product code was written for any target: only the requirements model, and /// CHECKS: comments in a scratch copy of the tests. Findings were proven with throwaway calls that are not part of any result.

What was tested

Unreal's source is available to anyone with an Unreal licence, not to the general public, so this file gives counts, method names and calls we wrote, and no engine code.

Check it yourself: three faults, each against a rule anyone can read

  • Unreal's SemVer class (Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs): var v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v); throws ArgumentOutOfRangeException. The SemVer spec, rule 9, lists that version among its valid examples. Re-run on the installed 5.7 source on 3 November.
  • Unreal's JsonObject: x.Equals(x) on {"a":null} throws NullReferenceException. Microsoft's Object.Equals contract says x.Equals(x) returns true and that Equals must not throw. Re-run on the installed 5.7 source on 3 November.
  • osu!'s ScoreProcessor: before anything is judged, TotalScoreWithoutMods reads 1,000,000 while TotalScore reads 0. Its own doc comment says it is the points awarded. Re-run on a fresh clone at that commit on 3 November.

The numbers

A break here is a planted bug: one small change to the code on purpose, a mutant in mutation testing, made on its own copy and run against the tests. "Live" leaves out breaks that did not compile and breaks whose compiled code was proven identical to the original. "Caught" means a test failed; "got past" means every test still passed.

SemVer

The 30 planted bugs no test caught, by line

Every one is in Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs as it ships in Unreal Engine 5.7, and every line number below was matched against that file. Each was one change, on its own copy, and the class's 2 tests (PrecedenceTests and PrereleaseTests, in EpicGames.Core.Tests) still passed. To check one, make that change in your own copy and run those tests: none fails. No engine code is printed here, only where each bug was and what changed.

JsonObject

osu! scoring

Checking the tool's verdicts without the tool

We applied breaks the tool said no test notices, by hand, to clean copies, and ran the tests ourselves.

What it found

Each of these fails when run, on the code as shipped.

SemVer against the spec

JsonObject against Microsoft's contract

osu! against its doc comments

JsonObject: the owner's questions, not defects we can call

What it found nothing on

Who found what

Be accurate about this. The defects above were found by an agent reading the code against each rule while writing the model: the requirements step. The breaks did not find them. The breaks measured how much the tests hold, and showed where no test looks: 9 breaks of JsonObject.Equals got past every test, and that is where the crash was. No graph neural network (GNN) played a part in any finding here.

More faults, found by reading the code carefully, each proven by the harness on the code as shipped:

What went badly

The trial's biggest finding was in our own tool. Real code broke it five ways on the first attempts, and a sixth way on osu!, none of which our own examples had shown:

It also could not read MSTest or xUnit tests' /// CHECKS: lines, only NUnit's, and it took a parameterized test's arguments for part of its name, so it reported named tests as never run. Because that fix came after osu!'s pass, its per-requirement check results are not quoted here. All of these are fixed, each with a planted test. A field's value and an indexer are still not walked, and the report now names them.

Our earlier figures on a game feature of ours came through nearly unchanged, because that code has no overloads: no existing verdict changed, and its 7 new breaks found one real gap, which a test now closes.

The budget did not hold. We set 45 minutes and 150K tokens per trial and spent about 100 minutes and over 260K output tokens, most of it on the tool's own bugs. A full pass took 13 minutes on SemVer, 9 on JsonObject and 74 on osu! (each constructor break rebuilds the game), at low priority on a busy machine. The earlier-budgeted 60 minutes for osu! did not hold either.

Re-running it

The models are in t7/ beside this file. For the Unreal trials, copy EpicGames.Core, EpicGames.Core.Tests and their props files from your own engine to a scratch folder and run closed_loop.py on the model; numbers may differ on another engine version, and these are from 5.7. For osu!, clone ppy/osu at commit 845336e, keep the three test files under osu.Game.Tests/Rulesets/Scoring, and run it the same way.

Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.