What it is not, yet

The first trial on code anyone can read

Measured The C# adapter was skipping code without saying so. On Unreal's own SemVer.cs it saw 35 of 68 statements, including none of the private method that does all the comparing, and on JsonObject.cs 213 of 288. It dropped overloaded methods, operators and generic methods with a where clause, and does not walk property bodies. The report still read as if the whole file was held. An earlier target of ours has no overloads, which is why its numbers never showed it.

Open All fixed, each with a planted case: naming every overload and operator, a report that names what it did not walk, a read-only tree such as a Perforce workspace, a constructor of a nested type, one bad break stopping a whole run, and a break that hangs a test forever. A field's value and an indexer are still not walked, and the report says so. Re-running that target changed no verdict.

Used where it does not fit, it can be worse than not using it at all.

On the slides: slide 15.

The two targets

These are goals for the loop at its full potential, used where it fits. Neither has been measured yet.

95% accuracy or better

Target Agents using the loop on work it fits should get the task right at least 95 times in 100. The remaining 5% would be bugs in our own code.

How it will be measured: answers graded blind, counted per failure pile, with a hidden acceptance test deciding whether the task was done.

Up to 90% fewer tokens

Target On average, up to 90% fewer tokens per closed requirement, because a later step looks up what the graph already holds instead of re-reading and re-deriving it. This number is always being re-evaluated.

How it will be measured: from each run’s own usage records (the tokens the model read and wrote), per closed requirement.

The target is about larger systems held across many changes, where re-reading and re-deriving is what costs; that has not been measured yet.

What is next: from a rough epic to code that is right

The tool’s main intent is to take the requirements for a game from a rough epic use case and grow them into a larger set, so that the code written from them is as accurate as it can be. The question behind it is a question and not a result: if the requirements for something very difficult are complete and checked, how far does that move what an AI can build? The direction is better tools for making video games.

Measured How much of existing code its tests hold, on three public targets (on code you can read for yourself).

Open The epic-to-code test has run, and its results are held for a paper and are not on these pages, so nothing here rests on them. The test starts from one rough epic for a small game whose right answers come from an independent published source. Agents write the code from the grown requirements, and hidden tests written from that source, never from our model, grade it blind. Tokens are recorded, with how many requirements the epic grew into and how many the hidden tests needed that it missed.

On the slides: slide 15, slide 18.

Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.