On code you can read for yourself
Three trials on 30 October 2026, on code outside our own project: two parts of Unreal Engine's own C# and the scoring rules of a shipped open-source game. Every number below was measured on these runs, and every defect was reproduced by running it. No test and no product code was written for any target: only the requirements model, and /// CHECKS: comments in a scratch copy of the tests. Findings were proven with throwaway calls that are not part of any result.
What was tested
- Unreal Engine 5.7,
EpicGames.Core/SemVer.cs. Requirements: the public Semantic Versioning 2.0.0 spec, every numbered rule either mapped or excluded with a reason. Tests:EpicGames.Core.Tests, 2 of them on this class. - Unreal Engine 5.7,
EpicGames.Core/JsonObject.cs. Requirements: the class's own doc comments, Microsoft's publishedEqualsandGetHashCodecontract, and RFC 8259. Tests: 43 on this class. 37 C# files under the engine'sSource/Programsname this class. - osu!, a shipped rhythm game (ppy/osu, MIT, commit 845336e),
ScoreProcessor.csandHitResult.cs. Requirements: their own doc comments. Tests: the 151 test cases inosu.Game.Tests/Rulesets/Scoring. The project has many more, which this pass did not run per break: see below.
Unreal's source is available to anyone with an Unreal licence, not to the general public, so this file gives counts, method names and calls we wrote, and no engine code.
Check it yourself: three faults, each against a rule anyone can read
- Unreal's SemVer class (
Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs):var v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v);throwsArgumentOutOfRangeException. The SemVer spec, rule 9, lists that version among its valid examples. Re-run on the installed 5.7 source on 3 November. - Unreal's JsonObject:
x.Equals(x)on{"a":null}throwsNullReferenceException. Microsoft's Object.Equals contract saysx.Equals(x)returns true and that Equals must not throw. Re-run on the installed 5.7 source on 3 November. - osu!'s ScoreProcessor: before anything is judged,
TotalScoreWithoutModsreads 1,000,000 whileTotalScorereads 0. Its own doc comment says it is the points awarded. Re-run on a fresh clone at that commit on 3 November.
The numbers
A break here is a planted bug: one small change to the code on purpose, a mutant in mutation testing, made on its own copy and run against the tests. "Live" leaves out breaks that did not compile and breaks whose compiled code was proven identical to the original. "Caught" means a test failed; "got past" means every test still passed.
SemVer
- Coverage tool, for comparison: 90% of lines, 85% of branches.
- 246 breaks made; 236 live; 206 caught (87.3%); 30 got past every test.
- 14 of those 30 sit on lines the coverage tool marks as covered.
- 48 of 68 statements held by a test; 18 have a break that gets past.
- 7 of the 16 requirements taken from the spec have no test about them, including every rule about which strings are valid. Which test is about what was our reading of each test, written before any break ran.
The 30 planted bugs no test caught, by line
Every one is in Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs as it ships in Unreal Engine 5.7, and every line number below was matched against that file. Each was one change, on its own copy, and the class's 2 tests (PrecedenceTests and PrereleaseTests, in EpicGames.Core.Tests) still passed. To check one, make that change in your own copy and run those tests: none fails. No engine code is printed here, only where each bug was and what changed.
Parse: line 54: argument 1 of the call toArgumentExceptionreplaced by its default value; line 54: the statement deleted.TryParse: line 68: theifcondition forced to true; line 76:falsechanged totrue.GetHashCode: line 81: argument 2 of the call toAsSpanreplaced by its default value; line 81: argument 1 of the call toGetHashCodereplaced by its default value; line 81: argument 2 of the call toGetHashCodereplaced by its default value; line 81: the constant 0 changed to 1; line 81: the value it returns replaced by its type's default.Equals: line 84: argument 1 of the call toEqualsreplaced by its default value; line 84: the value it returns replaced by its type's default.CompareTo: line 90: argument 1 of the call toComparereplaced by its default value; line 90: argument 2 of the call toComparereplaced by its default value; line 90: the value it returns replaced by its type's default.ToString: line 93: the value it returns replaced by its type's default.Compare: line 106: argument 2 of the call toGetFieldreplaced by its default value; line 107: argument 2 of the call toGetFieldreplaced by its default value; line 142: the second<on the line changed to<=; line 155: the constant 1 changed to 2.CompareFields: line 175: argument 1 of the call toIsNumericreplaced by its default value; line 175: theifcondition forced to true; line 177: argument 1 of the call toIsNumericreplaced by its default value; line 177: the constant 1 changed to 2; line 181: the constant 1 changed to 2.IsNumeric: line 192: the constant 0 changed to 1; line 194: the first<on the line changed to<=; line 194: the first>on the line changed to>=.operator!=: line 210: argument 1 of the call toEqualsreplaced by its default value; line 210: the value it returns replaced by its type's default.operator<=: line 220: the constant 0 changed to 1.
JsonObject
- Coverage tool: 77% of lines, 69% of branches.
- 582 breaks made; 429 live; 299 caught (69.7%); 130 got past every test.
- 67 of those 130 sit on lines the coverage tool marks as covered.
- 176 of 289 statements held by a test; 82 have a break that gets past.
- 6 questions only the class's owner can answer.
osu! scoring
- Coverage tool, from those 151 tests: 72% of lines, 78% of branches.
- 563 breaks made; 536 live; 286 caught (53.4%); 250 got past those 151 tests.
- 145 of those 250 sit on lines the coverage tool marks as covered.
- 81 of 191 statements held by those tests; 110 have a break that gets past.
- Read 53.4% as what those three test files hold, and as low for the game as a whole. We applied 7 of the surviving breaks by hand and ran the rest of
osu.Game.Tests:TestFailScore,TestResetFromReplayFrameandAccuracyRankAndTotalScoreOfLazerScorePreservedcatch some of them. We could not finish that project on this machine: its test host crashed on the clean copy and the broken one, at different points, with and without the visual tests, so what the rest of it would catch is unknown. - This pass ran before property bodies were walked, so 5 statements in property bodies got no break.
Checking the tool's verdicts without the tool
We applied breaks the tool said no test notices, by hand, to clean copies, and ran the tests ourselves.
- SemVer: 6 breaks at once (including
CompareToalways answering "equal" andTryParseno longer testing the match): all 90 tests that run still pass, 1 is skipped. No other test project names the class. - JsonObject: 8 breaks, each alone, against both test projects that name the class. 7 pass everywhere:
RemoveFieldremoves nothing,Equalsanswers true for anything,GetHashCodeis always zero, the file reader stops skipping a byte order mark, the file reader stops skipping comments,Parserefuses trailing commas, a missing double field answers true. - The 8th is the tests' credit. JSON
falsereading astruepasses the class's own project, butUnrealBuildTool.Testscatches it. We had first read that as uncaught, and running the second project corrected us. - 8 of
UnrealBuildTool.Tests' 116 tests fail on the clean copy in our setup, so they could catch nothing here.
What it found
Each of these fails when run, on the code as shipped.
SemVer against the spec
ComparethrowsArgumentOutOfRangeExceptionon valid versions, including the spec's own example compared with itself:v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v);. Any pre-release ending in a hyphen does the same.- A hyphen is treated as a separator, but the spec separates identifiers only at dots.
1.0.0-alpha-10sorts above1.0.0-alpha-9(the spec says below).1.0.0-2sorts above1.0.0-1-2(below). 1.0.0-a.band1.0.0-a-bcompare as equal whileEqualssays they differ.- It accepts strings the spec rules out:
"1.0.0\n", and non-ASCII digits such as an Arabic-Indic zero in1٠.0.0. Both come from .NET's meaning of$and\din the spec's regular expression. - Reach: no other C# file under the install's
Source/ProgramsorPluginsnames this class, so nothing in that tree is hurt today.
JsonObject against Microsoft's contract
EqualsthrowsNullReferenceExceptionon any object with a null field, evenx.Equals(x). The contract says it must not throw, and thatx.Equals(x)is true.- Two objects that compare equal have different hash codes, which the contract forbids. A
HashSetof two equal objects holds 2. We found noHashSet<JsonObject>orDictionary<JsonObject, ...>in the engine's C#.
osu! against its doc comments
TotalScoreWithoutModsreads 1,000,000 right afterApplyBeatmap, whileTotalScorereads 0, andPopulateScorebefore the first hit copies that 1,000,000 into the score. The field's doc says "The total number of points awarded for the score without including mod multipliers", andReset's doc says "Resets this ScoreProcessor to a default state", butResetnever clears it. We did not trace what the game does with the value afterwards.- After a second
ApplyBeatmapon one processor,MinimumAccuracyreads 1 before any hit. We did not check whether the game ever applies a beatmap twice.
JsonObject: the owner's questions, not defects we can call
Equalsanswers false for two parses of the same text when it holds an array or an object.- A repeated name keeps the last value at the top level and throws one level down.
{"A":1,"a":2}silently keeps one field, and throws one level down. RFC 8259 only says duplicates are unpredictable between programs. Readaccepts comments andParserefuses them. A root that is not an object throwsInvalidOperationExceptionfromReadandJsonExceptionfromParse.{"a":1e999}parses, thenToJsonStringthrows.{"id":9007199254740993}writes back as...992, which RFC 8259 allows.TryGetIntegerFieldaccepts the string"12".TryGetEnumField<DayOfWeek>accepts"99". On a German machineTryGetDoubleFieldreads the string"1.5"as 15.
What it found nothing on
- SemVer: 13 valid strings taken from the spec, 15 of the 19 invalid strings we tried, the printed precedence chain, and build metadata were all correct. All 21 breaks of
GetFieldand all 6 ofCompareNumericFieldswere caught. - JsonObject:
ToSystemJsonObject(8 of 8 caught),ToJsonNode(10 of 10),Read(5 of 5),ContainsField(3 of 3). A 562 KB file with long escaped strings read correctly, and from that we judged the pooled buffer inReadlatent. That was wrong, as the faults below show: "all breaks ofReadcaught" did not meanReadwas right. - osu!: all 41 breaks of the nine
HitResultquestions the tests ask about (IsHit,IsBonus,AffectsComboand the like) were caught, and so were all 8 ofApplyBeatmap, all 11 ofupdateScoreMultiplierand all 5 ofGetComboScoreChange. By hand, a reverted hit put the score, combo and accuracy back, and each rank's cutoff gave that rank.
Who found what
Be accurate about this. The defects above were found by an agent reading the code against each rule while writing the model: the requirements step. The breaks did not find them. The breaks measured how much the tests hold, and showed where no test looks: 9 breaks of JsonObject.Equals got past every test, and that is where the crash was. No graph neural network (GNN) played a part in any finding here.
More faults, found by reading the code carefully, each proven by the harness on the code as shipped:
ReadandTryReadparse a file over 128 KiB from a pooled buffer they have already returned. A valid 173 KB file with a very long name came back with other fields silently replaced, whileParseof the same text read it correctly, and a second file madeReadthrowFormatException.- Name matching uses a culture comparer, so names that are different strings can count as one field.
RemoveFieldandAddOrSetFieldValueignore the empty name, which RFC 8259 allows.NaNand infinity can be set, and thenToJsonStringcannot write them. An object parsed 64 levels deep cannot be written either: it reports a cycle that is not there.TryGetEnumFieldthrows on a struct that is not an enum.CreateDeepCopyshares reference values it does not know how to copy, and throws on keys that differ by case and on a multi-dimensional array.
What went badly
The trial's biggest finding was in our own tool. Real code broke it five ways on the first attempts, and a sixth way on osu!, none of which our own examples had shown:
- It silently skipped code. Any method with overloads, any operator, any generic method with a
whereclause and any property body was left out, and the report still read as if the whole file was held. On SemVer it saw 35 of 68 statements, missing the privateComparethat does all the comparing. On JsonObject it saw 213 of 288. - It crashed on read-only files, which is how a Perforce workspace keeps them, on the first run.
- It mistook a nested type's constructor for a method, and stopped.
- One mutant with broken braces stopped the whole run.
- An endless loop in a mutant tested alone hung the run: 11 minutes on one test host before we killed it.
It also could not read MSTest or xUnit tests' /// CHECKS: lines, only NUnit's, and it took a parameterized test's arguments for part of its name, so it reported named tests as never run. Because that fix came after osu!'s pass, its per-requirement check results are not quoted here. All of these are fixed, each with a planted test. A field's value and an indexer are still not walked, and the report now names them.
Our earlier figures on a game feature of ours came through nearly unchanged, because that code has no overloads: no existing verdict changed, and its 7 new breaks found one real gap, which a test now closes.
The budget did not hold. We set 45 minutes and 150K tokens per trial and spent about 100 minutes and over 260K output tokens, most of it on the tool's own bugs. A full pass took 13 minutes on SemVer, 9 on JsonObject and 74 on osu! (each constructor break rebuilds the game), at low priority on a busy machine. The earlier-budgeted 60 minutes for osu! did not hold either.
Re-running it
The models are in t7/ beside this file. For the Unreal trials, copy EpicGames.Core, EpicGames.Core.Tests and their props files from your own engine to a scratch folder and run closed_loop.py on the model; numbers may differ on another engine version, and these are from 5.7. For osu!, clone ppy/osu at commit 845336e, keep the three test files under osu.Game.Tests/Rulesets/Scoring, and run it the same way.
Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.