Switched off on 3 November
On the tool’s own code the network lost to a plain rule, so it is off. It wakes when a project’s graph reaches 5,000 entities, and must then win by more than 0.02 on code it has not seen. Its jobs are done by two small checking heads and by reading the test runs; this section is the record of what it did while it ran.
How the GNN works, what it records, and how it tries to improve
How it works
- It reads. Each node starts as its own words, hashed to 512 numbers.
- It listens. GraphSAGE layers let each node hear its neighbours, relation by relation. After that a node's numbers say something about what surrounds it.
- It ranks. A scorer ranks every requirement for a decision that has no owner.
- It asks. Below 70% sure it abstains. Above that, its answer goes out as a question for a person or an exact check. It is never a verdict.
It never reads breaks, tests or chains. Those hold the answer, and new code would not have them. Two ways that answer leaked in were measured and closed before any score here was taken. Whole methods are held out when it is tested, so each score is on code it has not seen.
What it records
- Its score beside the plain rules, on each pass run with
--gnn(stats.jsonl): the GNN, the best plain rule, the words alone, random and the same method, in mean reciprocal rank, where 1.0 is always ranked first. - Each job against the best plain rule (
kg/gnn_tasks.jsonl): who owns new code, which tests catch a break, which steps carry a requirement, which breaks are the same program. The results. - The shape of what it reads: its nodes and links, how many are owner links, how many methods, and how those links pile up: how few requirements hold most of them.
- Breaks ranked (
kg/survivor_rank.jsonl): how many of the survivors a learned order finds in the first fifth of a run, on average over ten seeds, against the plain rule and random. One seed flatters, so only the mean over seeds is recorded. - Its curve (
gnn_curve): its score by how much of the links it can see. - How sure it was: at 70% sure or more, how often it was right and on how much of the new code, by seed.
- Every tune (
gnn_tune.jsonl): each variant's score, and what was kept.
How it tries to improve
Its lead over the plain rules is itself a stat. When that lead falls as more tests arrive, the telemetry reports it as worsening and names reshaping the graph as a next job, beside writing tests. A tune is due after 5 passes without one anyway. A standing tick or a person then runs it: graph_gnn.py tune <graph>. A tune redraws what the GNN reads, with each relation left out and each derived link added. A change is kept only if it wins by 0.02 on the tuning seeds and again on a seed it was not chosen on.
So far
Measured Tunes have been run and nothing was kept: no variant won by the bar of 0.02. It attempts to improve itself, and the attempt has not paid yet.
Open A project's run plan records whether each learned job wins there. 3 of its 7 lines reach what a pass runs, wired by the tool's own tick, and all 3 are off today because a plain rule wins. The other 4 are measured only.
The GNN: who owns new code?
When someone adds code, the loop needs to know which requirement it belongs to. The GNN (a graph neural network, a learning model that reads the graph) suggested an answer, until it was switched off on 3 November, because a plain rule beat it. It looked at what was already linked around that code and ranked the requirements from most to least likely owner.
How it was tested fairly
It was trained on part of a project’s graph, with whole methods hidden from it. Then it was asked to rank owners for the hidden methods, code it had never seen. It was compared against simple rankers that do not learn, such as “the requirements already linked to this method” or “the requirement whose words overlap the code’s names most”. It was only allowed to make suggestions if it beat them.
What the numbers mean
MRR (mean reciprocal rank) scores a ranking: the right owner ranked first scores 1, second scores ½, third ⅓, and so on, averaged over every test case. An MRR of 0.75 means the right owner is usually first, sometimes second.
Where these numbers come from
Measured On the tool’s own code, with whole methods held out and a mean over five seeds, the GNN scored MRR 0.65 against 0.72 for the best ranker that does not learn: it loses, and is switched off there. Runs differ by about a tenth, so a single number is never the whole story.
Measured It only wins on a large enough graph that is about half claimed: see Seed it first, then it flies.
Measured The design is to abstain: below 70% sure the GNN says “not sure” and asks. How often it is right at 70% and on how much of the new code was measured on one project of ours, and is not repeated here because that project is not public.
Its suggestions were always questions for a person, never verdicts. One lesson from building it: the first version did worse than the simple rankers, because the links it was asked to predict were also links it could read. Separating the two fixed it.
Shaping the graph for the GNN
Not run since the network was switched off on 3 November, because a plain rule beat it. This is the record of what the tunes found.
The GNN learns from the graph’s links, and not every kind of link helps it. A tuning pass tries leaving out one kind of link at a time (for example the link from every branch to the one shared “space” node, which ties everything to everything) and measures the GNN again on code it never saw. A change is kept only if it helps on the tuning runs and again on a separate run it was not chosen on, so that a lucky result is not mistaken for a real one. The kept choice is written beside the graph, and the GNN reads it from then on.
The stats decided when to tune: when the GNN’s lead over the simple rankers stopped growing, or every five passes otherwise (how the stats choose).
The first tune, on the graph the network was built on
Measured The full graph set the score to beat, and run-to-run noise alone was about 0.02. Six variants were tried. None beat the full graph by the required 0.02, so nothing was changed. The best, leaving out the use-case structure links, gained a little; leaving out the shared “space” node gained less.
Measured The links the GNN relied on most were “binds requirement” and “is at”, which ties a branch to its method.
The second tune: adding links, three runs each
Measured Adding links the graph lacks did not help beyond the noise: linking the methods of one class, and linking requirements one step carries together, each gained under 0.01. Leaving out the use-case structure links was the only change never worse on any run, and it stayed under the 0.02 bar. Nothing was changed.
What that says: on this graph the GNN already gets what the structure can give it. The next gains are more likely from more graphs to learn from and better descriptions of each node than from reshaping this one.
Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.