Where an LLM's Reasoning Matters: A Negative Result Worth Reporting

Take a strong model, make a weaker copy of it, and watch where the two disagree. The tokens where the weak copy would have written something else are the ones that needed the capacity it lost. If those are also the tokens that decide whether the final answer is right, you get a dense per-token training signal out of a single pass-or-fail reward, with no human step labels and no learned reward model.

We spent the semester testing that premise. It does not hold. Across three interventions on MATH-500, positions where the two models disagreed had no more influence on the final answer than matched positions where they agreed: McNemar p = 1.00 on the paired test, Fisher p = 0.80 at the step level.

The tempting move is to bolt the signal onto a GRPO objective and watch the reward curve. But a training run tangles the signal up with the learning rate, the advantage normalization, the KL penalty and everything else, and it burns a lot of compute to teach you very little. We collapsed the framework into one claim we could break instead: disagreement positions have more leverage on correctness than depth-matched positions where the models agree. Then we attacked it three ways, each harder than the last. Hand generation to the weak model partway through a solution. Flip a single token and let the strong model carry on. Corrupt a whole reasoning step and see whether it recovers.

Most of the work went into the controls. Depth turned out to dominate everything: release the weak model in the last quarter of a solution and it finishes correctly about 70 percent of the time, release it in the first quarter and it essentially never does. Without matching on depth we would have measured that and called it a discovery. Entropy is the other trap. Disagreements cluster where the strong model is already unsure, 0.41 nats against 0.08 at controls, and entropy-based token selection is the obvious single-model baseline any two-model signal has to beat, so every comparison is adjusted for it. We spliced in the strong model's own second-choice token to keep the damage equivalent across arms, and we released only at the five most confident disagreements per solution. That last choice stacks the deck toward the hypothesis on purpose, so a null result is harder to wave away.

One effect did survive. Perturbing at a disagreement position disrupts the next few tokens far more than perturbing elsewhere, 2.55 against 1.60 nats per token, and roughly two thirds of that gap remains after adjusting for entropy. It just never reaches the answer: the same perturbation moves the final answer's probability by about 0.001 nats in every arm. The signal is real, and it is local. It finds where the missing capacity was doing its work, which is mostly fluency and confidence at awkward positions, rather than where the answer is won or lost. Hard for a weaker model and load-bearing for the outcome are simply different properties. In distillation the first one is the whole point, which is why related divergence signals work well there. It does not carry over.

Characterizing the weakened model taught me something I keep reusing. On MMLU it looked usable far below full capacity, still scoring 0.42 against a 0.25 chance baseline at the most aggressive compression. Ask it to actually generate at that setting and it loops on the same phrase until the budget runs out. Likelihood ranking and free generation are not the same test, and the wrong one will cheerfully report that a broken model is healthy. We mapped where that cliff sits and ran every intervention just above it, where disagreement covers 12.3 percent of tokens instead of the 44 percent of noise below.

What the null licenses is worth stating precisely. Binary outcome metrics are not powerful, and ours could only have caught a difference of around 11 percentage points, so a modest effect would have slipped past. The regression treats positions as independent when they are nested inside 23 rollouts, which makes t = 2.35 an upper bound on the evidence rather than a clean reading. And all of this concerns top-1 disagreement, greedy decoding, one model, on problems that model already solves. Graded measures like KL or Jensen-Shannon divergence are still open.

The result I did not go looking for is that corrupting an entire reasoning step left the final answer intact 84 percent of the time. These chains are far more redundant than the credit-assignment framing assumes. Most steps are not load-bearing at all, and finding the few that are will take more than a capacity difference. That is the thread I would pull next. The infrastructure survives the negative result either way: one decomposition run yields the same model at any capacity you like, which is a useful thing to own when the question is what a network's extra capacity is actually for.