InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning

🍞 Top Bread (Hook) Imagine you wrote a long math solution and got the final answer wrong. A teacher glances through your work and says, 'Most of this is fine, but this one step here sent you off track.' You fix that exact step, and suddenly the whole solution works.

🥬 Filling (The Actual Concept): Large Language Models (LLMs)

• What it is: LLMs are computer programs that read and write text by predicting the next token (like the next word or symbol) very well.
• How it works: They learn from lots of text, then, given a prompt, they continue writing; for math or logic, they produce step-by-step reasoning.
• Why it matters: Without good step-by-step habits, they may produce long, confident answers that end wrong. 🍞 Bottom Bread (Anchor): When you ask an LLM to solve a puzzle, it writes a chain of steps; a single bad step early can ruin the ending.

🥬 Filling (The Actual Concept): Reinforcement Learning (RL) with outcome rewards

• What it is: A way to improve models by giving a reward only for the final outcome (correct or incorrect).
• How it works: The model tries many solutions; if an answer is correct it gets a positive reward, if not it gets zero; then it updates its behavior toward steps seen in successful attempts.
• Why it matters: This pushes the model toward better answers over time, but it treats every step in a correct solution as equally good and every step in a wrong solution as equally bad. 🍞 Bottom Bread (Anchor): If a 100-step solution ends wrong, traditional RL punishes all 100 steps, even if 99 were fine.

🥬 Filling (The Actual Concept): The credit assignment problem

• What it is: Figuring out which specific steps helped and which hurt the final answer.
• How it works: Ideally, we would label good steps as ‘keep doing this’ and bad steps as ‘change this’ inside each long solution.
• Why it matters: Without it, models learn the wrong lesson: they may throw away good habits from failed tries or keep bad habits from lucky successes. 🍞 Bottom Bread (Anchor): If your answer is wrong because of step 57, you should not be told steps 1–56 were bad too.

The world before: LLMs improved at reasoning using outcome-reward RL: if the final answer is right, reinforce the entire trace; if it’s wrong, discourage it. This works when problems are short or when many tries succeed. But on hard tasks (like Olympiad-level math), most attempts fail. Worse, failed attempts are often very long and contain many correct parts. Treating everything in a failed attempt as bad wastes useful learning signals.

The problem: When training is dominated by wrong rollouts, advantages collapse toward zero (no clear positive examples), and the model sometimes becomes too verbose, too short, or randomly changes strategy. We need to reward or replace only the steps that truly matter.

Failed attempts to fix it: One fix is to train a process reward model (PRM) that scores each step. But PRMs are expensive to build and still need a smart way to turn step-scores into better next steps. Another fix is to run many branched rollouts to estimate value per step, which is very costly.

The gap: We want precise, cheap credit assignment without training a separate PRM or doing expensive branching. We need the model to point to the first mistake and suggest a better step right there.

Real stakes: In real life, models tutor students, plan multi-step tasks, write code, and do math. If the model can pinpoint and fix the precise step that derailed its reasoning, it can learn faster from failures, need fewer resources, and become more trustworthy. That means better homework help, safer code synthesis, and more reliable decision making in long, delicate tasks.

🍞 Top Bread (Hook) You know how a GPS says, ‘Recalculating… take the next right’ after you miss a turn? It doesn’t restart the whole trip. It fixes the exact point where you went wrong.

🥬 Filling (The Actual Concept): Intervention (a self-proposed corrective step)

• What it is: A short, targeted step that replaces the first wrong step in a solution to steer the rest of the reasoning back on track.
• How it works: The model compares its reasoning to a reference solution, finds the first mistake, and writes a new single step to insert there.
• Why it matters: Fixing just that step preserves all earlier good work and leads to a correct finish. 🍞 Bottom Bread (Anchor): If you misapplied a formula in step 23, the intervention rewrites step 23 correctly so the later steps make sense.

Aha moment in one sentence: Let the model self-verify against a reference solution, find its first critical error, propose a one-step fix (intervention), then train on those fixes so future rollouts avoid the same trap.

Three analogies:

1. GPS reroute: Don’t start the trip over—change the next turn where you got lost.
2. Teacher margin note: Instead of rewriting your whole essay, the teacher marks the exact sentence to fix.
3. Lego repair: Replace the one wrong brick so the rest of the tower stands.

Before vs. After:

• Before: Outcome-reward RL rewards or penalizes whole solutions. Good steps in failed tries get punished; bad steps in lucky successes get rewarded.
• After: InT pinpoints the first wrong step, swaps it for a better one, and trains the model to prefer that fix. Now RL starts from a model that already knows how to avoid its prior pitfall.

🥬 Filling (The Actual Concept): Reference solution

• What it is: A correct, human-written (or trusted) solution used only to help the model verify its attempt.
• How it works: The model does a ‘textual diff’: it lines up its steps with the reference, spots the first mismatch that breaks logic, and drafts a correction.
• Why it matters: Verifying is easier than generating; models can detect local errors even if they can’t solve the whole problem from scratch. 🍞 Bottom Bread (Anchor): The model can notice ‘I said all roots are on the unit circle, but the reference shows zero is a valid root; I need to fix that claim.’

🥬 Filling (The Actual Concept): Self-verification

• What it is: The model checks each of its steps for correctness using the reference.
• How it works: Step-by-step, it marks steps as correct or identifies the first critical error; then it proposes a replacement.
• Why it matters: It merges ‘find the problem’ and ‘fix the problem’ into one action. 🍞 Bottom Bread (Anchor): The model flags step 12 as the first logic break and writes a corrected step 12 that aligns with the reference.

🥬 Filling (The Actual Concept): Counterfactual continuation

• What it is: Continue the solution as if the corrected step had been there all along.
• How it works: Condition the model on the original prefix plus the intervention, then let it finish.
• Why it matters: This shows whether the one-step fix truly steers the rest of the reasoning to a correct answer. 🍞 Bottom Bread (Anchor): With the new step 12 inserted, the continuation now reaches the right final integer answer.

Why it works (intuition, not equations):

• Verification is easier than full generation: spotting the first wrong step is simpler than inventing the whole correct solution.
• Localized credit: Reinforce the precise fix, not the whole trace. That preserves good habits and targets bad ones.
• On-policy advantage: Most tokens remain from the base model; only a short intervention is new. This keeps the model’s token probabilities stable, avoiding high-entropy drift that makes RL unstable.
• Better starting point for RL: After SFT on prefix+intervention, the model already avoids previous traps, so more problems produce at least one correct sample (lower zero-advantage ratio).

Building blocks:

• Self-verify and find the first critical error.
• Propose a single-step intervention (no final-answer leaks).
• Condition rollouts on prefix + intervention to test if it works.
• Collect and filter interventions that yield at least one correct continuation.
• SFT only on prefix + intervention (exclude suffix) to internalize the fix without over-constraining future exploration.
• Run standard outcome-reward RL (like GRPO) from this patched model to amplify gains.

🥬 Filling (The Actual Concept): On-policy vs. off-policy data

• What it is: On-policy data look like what the base model would naturally produce; off-policy data look very different.
• How it works: Interventions are short, so most tokens are still the model’s own; that makes them on-policy.
• Why it matters: Training on on-policy data keeps next-token entropy low and stabilizes RL. 🍞 Bottom Bread (Anchor): SFT on full human reference solutions (off-policy) can distort the model; SFT on short interventions (on-policy) preserves its strengths.

High-level pipeline: Input problem → Base model generates a reasoning trace (often wrong) → Self-verify against a reference to find the first critical error → Propose a one-step intervention to replace that error → Test by conditioning continuation on prefix + intervention → Filter good interventions → SFT on prefix + intervention only → RL from the patched model.

Step 1: Generate an initial reasoning trace

• What happens: Use the instruction-tuned base model to produce a step-by-step solution. Many will be wrong on hard problems.
• Why this step exists: We need real, on-policy mistakes to fix; they are the best training signals for credit assignment.
• Example: On an Olympiad problem, the model writes 80 steps; the final answer is incorrect.

🍞/🥬/🍞: Supervised Fine-Tuning (SFT)

• What it is: Teaching the model by showing target text for given prompts and having it increase the likelihood of those targets.
• How it works: Collect (prefix + intervention) pairs and fine-tune so the model learns to produce the intervention when it reaches that prefix.
• Why it matters: This localizes learning to the exact fix, not the whole trace, so the model keeps its previous good habits.
• Anchor: When the model again reaches ‘Step 23’ on a similar problem, it now prefers the corrected step it learned.

Step 2: Self-verify with a reference solution to find the first critical error

• What happens: The model compares its steps to the trusted solution, listing correct steps and identifying the first wrong one that breaks the logic and isn’t later fixed.
• Why this step exists: Credit assignment requires pinpointing where things first go off the rails.
• Example: It finds that declaring all roots have absolute value 1 is false because zero is a valid root with absolute value 0.

Step 3: Propose a one-step intervention

• What happens: The model writes a short replacement for that first wrong step, with guidance for what to do next but without revealing the final answer.
• Why this step exists: We need a minimal fix that preserves earlier correct steps and encourages a correct continuation.
• Example: ‘Instead of claiming all roots lie on the unit circle, check whether zero can be a root by evaluating f(0).’

Step 4: Counterfactual continuation (testing the fix)

• What happens: Concatenate the original prefix (all steps before the error) and the intervention, then let the model continue to the end.
• Why this step exists: To see if the one-step fix really steers the rest toward a correct final answer.
• Example: With the new step inserted, 1.56% of continuations reach the correct answer versus about 0.07% without it (about 22x better).

Step 5: Filter and collect interventions

• What happens: Keep interventions that succeed at least once across multiple continuations; discard any that leak the final answer.
• Why this step exists: Ensures the dataset teaches real corrective patterns, not memorization or shortcuts.
• Example: From thousands of hard problems, about 1,076 interventions pass the filter.

Step 6: SFT design choices (the secret sauce)

• Clone the prefix + intervention, exclude the suffix: Training on the suffix (the rest of the correct solution) narrows exploration too much and hurts later RL. Training on the prefix + intervention localizes learning to ‘what to do at the moment of mistake.’
• Keep only interventions that worked at least once: This raises quality and improves post-SFT performance.
• Why it matters: These choices maximize coverage (how many problems see at least one success) and keep the model’s next-token entropy close to baseline, which stabilizes RL.
• Example data effect: Cloning prefix + intervention (no suffix) solved far more training problems than including the suffix.

🍞/🥬/🍞: RL post-training (e.g., GRPO)

• What it is: A standard outcome-reward RL phase to further improve pass@1 using the model patched by SFT.
• How it works: Sample multiple rollouts per problem; reinforce the ones that end correctly; discourage the ones that don’t.
• Why it matters: RL converts the higher pass@k (more correct samples somewhere in the batch) into higher pass@1 (get it right on the first try).
• Anchor: After InT, RL reward rises faster and the zero-advantage ratio drops (more problems yield at least one correct rollout).

Step 7: Practicalities and ablations (what breaks without each part)

• Without instruction-following: The model may ignore the verification prompt or fail to output the intervention in the required format, leading to fewer usable interventions and lower coverage.
• Without references: Interventions are weaker; performance drops notably versus having a trusted solution for comparison.
• Without filtering: You collect noisy or leaky fixes; SFT quality falls and gains shrink.
• Using larger models to propose interventions: Quality rises; coverage and accuracy increase.

The secret sauce in one breath: Use the model’s easier skill (verification) to fix its harder skill (generation), keep the fix minimal (one step), train only on the prefix + fix to stay on-policy and low-entropy, then let standard RL finish the job.

The test: Do self-proposed interventions help solve problems the base model could not? Researchers measured two things: (1) average success rate (accuracy) when continuing from different prefixes, and (2) coverage, the number of distinct problems for which at least one continuation is correct.

Key head-to-heads when continuing rollouts:

• From error step (no fix): about 0.071% accuracy; coverage around 29 out of 334 problems.
• From prior prefix (resample next step): about 0.073% accuracy; coverage around 31 out of 334.
• From prefix + intervention: about 1.56% accuracy; coverage 80 out of 334. That’s roughly a 22x jump in accuracy and much broader coverage.

Competition (alternative uses of references):

• Hints: Give a partial prefix of a trusted solution to guide early steps. Helpful but limited when the first error occurs deep in the trace.
• Interventions: Fix the first wrong step mid-trajectory. Stronger for credit assignment and pairs well with hints.
• Combined: Hints + interventions did best on a subset where hint-only or intervention-only struggled alone.

Surprising findings and ablations:

• Instruction-following matters: The instruction-tuned 4B-Instruct model produced more valid interventions than a similar-sized reasoning-tuned model that sometimes failed to follow the verification/format instructions.
• Bigger proposer helps: A 30B instruct model proposing interventions improved accuracy and solved more problems than the 4B proposer.
• References matter: Proposing interventions without a reference solution still helped over naive baselines but was clearly weaker than with references.

Pass@k and on-policy stability:

• After SFT on interventions, pass@k (for k from 16 to 1024) rose both on training and test splits. This means the patched model samples correct solutions more often, giving RL more to reinforce.
• Likelihood analysis showed InT traces are the most on-policy (lowest negative log-likelihood under the base model) compared to self-reflections, R1 think/summaries, or human/Gemini references. Training on on-policy data keeps next-token entropy low and avoids destabilizing the model.

Zero-advantage relief and RL gains:

• Starting RL from the InT-patched model yielded higher average reward and a much lower zero-advantage ratio (fewer problems with no correct rollouts at all) than starting from the base or from models SFT’d on full references or self-reflections.
• This means RL could finally learn from many previously hopeless problems.

Benchmark scoreboard (context):

• IMO-AnswerBench (Olympiad-level, curated by medalists): Base 4B-instruct scored about 11.68%. After InT + RL, it reached about 25.62% (roughly +14 points), outperforming larger open-source models like gpt-oss-20b on this set.
• Across hard benchmarks (IMO-AnswerBench, HMMT Nov 2025, AMO-Bench, Apex Shortlist), InT + RL delivered the strongest overall results, with average scores beating standard RL and hint-guided RL, and much better than SFT on reference solutions.

Make the numbers meaningful:

• 1.56% vs 0.07% is like going from almost never getting it right to getting about 1–2 right per hundred tries on the hardest cases—big progress when most attempts fail.
• Coverage jump from 29 to 80 out of 334 means nearly triple the number of problems become learnable.
• A ~14-point gain on a medalist-curated IMO set with a 4B model is like a small team beating a bigger team because it learned to fix its exact mistake instead of retraining its entire playbook.

Unexpected but helpful:

• Excluding the suffix during SFT (training only on prefix + intervention) was better than including the entire corrected solution. Including full suffix shrank exploration and hurt downstream RL.
• Short interventions (usually under 200 tokens) versus very long full rollouts (often ~7k tokens) kept training close to the base distribution and avoided entropy spikes.

Limitations:

• Dependence on reference solutions: The method assumes access to a trustworthy solution for verification. Without it, interventions are weaker. Future work may train stronger verifiers to reduce or remove this need.
• First-error focus: Fixing the first critical error often suffices, but some problems may contain multiple subtle, interacting errors.
• Instruction-following requirement: If the base model struggles to follow the verification prompt or output format, intervention quality drops.
• Domain scope: The paper targets math reasoning; applying InT to code, proofs, or multi-modal tasks likely needs careful prompt and verifier design.

Required resources:

• An instruction-tuned base model that can follow a verification-and-intervention prompt reliably.
• A corpus of hard problems with reference solutions (human or trusted model-generated, then filtered).
• Compute for: generating initial traces, proposing interventions, testing continuations (dozens of samples per problem), then modest SFT and short RL runs.

When not to use:

• No references and no reliable verifier: If you cannot trust step-level verification, proposed interventions may mislabel steps.
• Very short problems: Full-outcome RL or vanilla SFT may be simpler and enough.
• Creative writing or open-ended tasks: ‘First critical error’ is ill-defined; InT is best for objective, checkable reasoning.

Open questions:

• Can we train a strong verifier that doesn’t need a reference solution, so the model can self-improve autonomously?
• How to extend from single mistakes to interacting chains of errors, or to proof-style solutions that require strict step validation?
• How to apply InT in continual learning with memory or multi-turn settings where the ‘first error’ may be in a past summary, not the current step?
• Can InT combine with data generation (e.g., models proposing new training problems) for a fully self-improving loop?

Big picture: InT trades expensive step-reward modeling and branched rollouts for a simple, scalable routine: self-verify, fix one step, train on that fix, then RL. It is not a magic wand, but it makes the most of failed attempts and turns them into focused lessons the model can actually learn.

Three-sentence summary:

• InT lets a model compare its own failed solution to a reference, find the first critical error, and write a one-step intervention to fix it.
• Training only on the prefix plus that intervention keeps learning on-policy and low-entropy, creating a stable, effective starting point for standard RL.
• The result is strong performance gains on very hard math benchmarks, including about a 14-point jump on IMO-AnswerBench with just a 4B model.

Main achievement:

• A simple, compute-friendly recipe for credit assignment that turns failed rollouts into targeted lessons without training a separate process reward model or using branched rollouts.

Future directions:

• Train verifiers so references are not required; extend to proofs, code, and multi-turn tasks; explore fully autonomous loops where models generate problems, verify steps, and improve themselves continuously.

Why remember this:

• Because it shows that small, precise fixes beat broad, blunt updates. By teaching a model to correct exactly where it went wrong, we keep what works, fix what doesn’t, and make RL learn from problems that used to give no signal at all.