Self-improvement, on by default

The tool tests itself, and that is on unless you turn it off. It happens at two sizes.

Every pass. A pass writes one row of stats about itself to the morph log, checks those stats against the tool's own requirements for improving, and prints what to improve next. Nothing about your code changes.

The standing tick. A fresh agent with no memory takes one item, measures the same stat before and after the change, and records whether it was worth it. Both ship in this kit, with the self-improving qualities we built for our ARC3 harness. Nothing starts a tick for you: TICK.md is its runbook, and you schedule it.

Turning it off

Run with --no-self-improve, or put self_improve: false in the model's loop: block. The pass then records no row and prints no picks. There may be some token saving, but on a pass it is small; the larger spend is the tick, which only runs if you schedule it, and stops when you stop scheduling it or create .loop_tick.STOP beside the tools.

Where these numbers come from

Measured A pass: 44 stats in a row, about a third of a second to collect them, three printed lines for the agent to read. On the kit's Python example, a clean install, with it off a pass wrote no row and printed 295 fewer characters.

Measured A tick: the app counted 197,134 tokens for the first tick and 291,042 for the second, over 9 and 22 minutes.

One shared graph, not networked in this version

Ideally everyone working on a project shares one graph, because it also works as a retrieval store (retrieval-augmented generation, RAG): an agent looks up what is already proven instead of re-reading the files to work it out again. We share ours across the team that way, in the belief that it saves tokens. This version is not networked, and each checkout keeps its own graph; you could add sharing in whatever way suits your network.

Where this is not measured

Open How far a shared graph cuts what an agent reads has not been measured. To learn who owns a method, what it must do and which tests catch it, the graph holds rows for that method and the files hold the model, the source and the tests; no agent has been timed using either, so a saving is a belief and not a measurement.

What it has found so far

Where these numbers come from

Measured 19 ways a check could pass for the wrong reason, found by fresh reviewers; 17 fixed, each with a planted mistake a selftest must catch (see reviewers).

Measured 11 stats were recorded with nothing reading them. The audit first said 6, and that was itself a bug: it counted test fixtures and the recorder as readers. Now 0: each stat triggers something, or has a reader with a date.

Measured 33 requirements on its own stats, 2 records banked for a learner that does not exist yet, and 5 triggers firing now. Each trigger that fires is a work item. It was 12 until a tick found that 8 of them fired on every pass whatever happened.

Measured 9 faults its first measuring round found in itself: 5 in the harness, 4 in the tool, all fixed.

Measured Its optional parts: of the run plan's 7 lines, 3 reach what a pass runs, and on today's evidence all 3 are off on both projects, because a plain rule wins by more than the 0.02 a learned part would need. The other 4 are marked measured only: they record a verdict and change nothing, and three of the verdicts behind the plan flip with the seed alone. Its own tick found that none of the 7 reached a pass by code before it wired these.

Measured 58% of the breaks of its own code were caught, 885 of 1,532, on its first self pass; a second pass on 30 October caught 989 of 1,637 (60%). The tool is less tested than the feature it tests.

Open 0 of the 20 measured outcomes its improvement learner needs, so rules choose the work for now, not a model. These numbers show what it has found, not yet how fast it gets better.

The loop, in pseudo C++

The toolkit is written in Python. This is the same logic for a C++ reader; the lower-case names are the real ones in the code.

// One pass. Self-improvement is on by default.
Verdict run(Model& m, bool selfImprove = true) {
  Breaks  b = Mutate(m.code);       // one compile
  Results r = RunEachAlone(b, m.tests);
  Graph   g = KnowledgeGraph(m, r);
  Holes   h = Criteria(g);          // 18 checks

  if (self_improve_on(m, selfImprove)) {
    Row row = PassStats(m).collect();  // 44 stats
    MorphLog(m).add(row);
    auto fired = improvement::fires(rules, log);
    Print(NextImprovement(log, fired).picks());
  }

  return h.ours.empty() ? HOLDS : OPEN;
}

// One tick. A fresh agent, one item, no memory.
void Tick() {
  Start s = loop_tick::start();     // the lock
  if (s.stopped || s.busy || s.idle) return;
  if (s.epochStarts) Brainstorm();  // every 12
  RunSelftests();                   // red = item
  telemetry_audit();                // unread = item
  Item i = Choose(improvement::check(), board);
  Stat before = i.stat.Measure();
  i.Do();                           // one change
  Stat after = i.stat.Measure();
  if (s.epochEnds) {
    run_plan::decide();             // parts on or off
    improvement::graph();           // did it work?
  }
  loop_tick::finish(i, before, after, worth);
}

Three ticks in a row worth "no" and the tick sleeps until something it watches changes, so an idle loop costs almost nothing. The improvement graph is the same shape one level up: stats are its code, triggers its requirements, ticks its tests.

The loop on its own improvement

The loop records a great deal about itself: every pass records 44 stats about itself, and the learned parts leave evidence of when they help. Recording a number is only worth it if something reads it and acts. On 29 October an audit found numbers that nothing read, and many more that were only printed.

So the improvement of the loop now has requirements of its own, written the way the product's are. Each number has a requirement saying what pattern in it should trigger something, what that is, and why the number is kept: a test that catches nothing for two passes is a speed item; an excuse for a harmless break that lapses is a hole; the graph not growing for three passes means the model is not being seeded. A number kept for a learner that does not exist yet must name that learner and a date. Past the date, unread, it fails.

Three levels, one graph

The same learning applies one level up: once enough actions have measured results, a model can learn which kind of action moves which number, and it is switched on only where it beats the plain rule, like everything else.

Where these numbers come from

Measured After the audit of 29 October gave each number a reader or a trigger, none is read by nothing. 14 now drive a decision in code and 28 are covered by a trigger.

Measured The first run of the triggers fired 12: among them, 5 tests in the tool's own suite that catch no break at all, 8 of the who/what/why questions never asked on each model, and one requirement whose breaks were all caught last pass and are not now.

Open No action has a measured result yet; the improvement graph becomes learnable at 20.

Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.