A requirements ladder

How to get a machine to stop guessing

Ask a model something hard and it answers in the same confident voice whether it knew the answer or filled the gap in. This is a way of laying the problem out so that the difference is visible on the page.

Use the arrow keys, or the buttons at the bottom.

The problem

A confident answer and a guess look identical

You cannot tell, from the answer alone, which parts came from something real and which were invented to make the sentence finish. That is not a flaw you can fix by asking more politely, and it is not fixed by the model being bigger — a bigger model fills the gap more plausibly.

So instead of trying to make the guess better, the guess gets marked.

The idea in one picture

Every decision becomes a list of options, cheapest first

Think of a repair manual. A seized bolt has a known list of things to try, in order: penetrating oil, then heat, then a breaker bar, then drill it out. The manual does not guess which one will work. It lists them, cheapest and least destructive first, and you walk down.

A hard problem for a machine is the same shape. Each decision inside it gets its options written down, in cost order, and each option says what it costs, whether it has ever worked, and where it was read from.

The labels

Four honest states, and one of them is "we cannot tell"

The rule that makes it worth reading

Every option names the place it came from

Not a summary. A file and a line number, which a checker opens. If the file is missing, or the line is past the end of it, the whole thing refuses to publish.

That is the difference between a document about a system and a document the system has to keep true. One rots quietly; the other fails loudly.

Going down

A real descent, from the run's own output

A question with many possible answers. Each option narrows the field and says why. Nothing below is typed into this page.

The endings

Three ways to finish, and only one of them is an answer

Here it is

One candidate survives, with the thing that ruled out the others.

It is not there

The thing you asked about genuinely does not exist in the place that matters — stated as a fact, with the proof.

You asked the wrong thing

None of the candidates can be it, and the reason is shown. Picking the least-bad one would be the guess.

A system that can only ever return an answer will always return one. Two of these three endings are the reason this one does not have to.

Going up — the part that is different

A problem is one way of achieving something

Every scheme for breaking problems down goes downward: here is the problem, here are its pieces. That finds you a better sub-step. It does not find you the jump.

This one also reads what a problem is for — the goal it serves — and then asks what other ways there are to reach that same goal. Those siblings are where a jump comes from: not a better way to get the bolt off, but noticing you could cut the bracket instead.

Going up branches far faster than going down, so the tree becomes effectively endless. That is the point: generate wide, then grind it down with checks rather than with opinion.

How this compares

Thinking harder is a different move

Think longer

Give the model more room to reason before it answers. It is the standard advice and it works on plenty of tasks.

Anthropic's guidance on giving a model room to think, and extended thinking.

Branch and score

Let the model propose several lines of reasoning and rate them, keeping the best. Tree-of-thoughts and its relatives.

Yao et al., 2023; Besta et al., 2023; least-to-most, Zhou et al., 2022.

Two measured catches. First, in branch-and-score the model is both the generator and the judge — and a judge that calls every branch promising cannot prune anything. Second, more reasoning is not always better: a study found accuracy falling as reasoning was extended on counting, deduction and constraint tasks.

"Inverse Scaling in Test-Time Compute" — arxiv.org/abs/2507.14417, alignment.anthropic.com/2025/inverse-scaling.

Why this one behaves differently

The judge is not the model

The test plan

Start small on purpose

It is being built and measured against small models first, and only then moved up toward the frontier ones. That order is deliberate, because it is the only way to see where the lift comes from:

What it is aimed at

The questions nobody can hold in their head

The reason for building it is a class of problem where the answer is not one insight but a long chain of decisions, each with options nobody has enumerated — such as how to make a system train itself very much faster.

You cannot reason your way through that in one pass, and a confident paragraph about it is worth nothing. But you can map every decision it contains, and every alternative anyone has written down, and then look for the one place a jump is available — or for the single missing skill that blocks it.

That is a target, not a result. Nothing on this page claims it has been reached.

Check it yourself

What would make this refuse to publish

Every figure in this deck and on the ladder page is injected from the generated file at load. The last one on that list is what stops the page quietly drifting away from the thing it describes.

Open the ladder · Back to the index