Likelihood-Based Reward Designs for General LLM Reasoning

🍞 Hook: You know how it’s easier to learn when your teacher gives you helpful hints instead of only saying “right” or “wrong”?

🥬 The Concept: Before this paper, many models learned to reason with a “right/wrong” score. That means the model writes a step-by-step chain-of-thought (CoT), then gets a 1 if the final answer is correct and 0 if not. How it works: (1) The model tries a reasoning path; (2) it gives a final answer; (3) a checker (verifier) says 1 or 0; (4) the model learns from that one bit. Why it matters: If the model rarely gets exactly right answers early on, the reward is almost always 0, so learning is slow and fragile.

🍞 Anchor: Imagine practicing math and only hearing a buzzer for “wrong” without seeing which parts were close—you’d improve very slowly.

🍞 Hook: Imagine learning to write essays. There’s no single “correct” sentence to match, so who decides if it’s good?

🥬 The Concept: Some tasks are verifiable (like 2+2=4) and others are non-verifiable (like a detailed explanation). How it works: (1) Verifiable tasks can be checked automatically; (2) non-verifiable tasks can’t be graded as simply right/wrong; (3) this makes binary rewards hard to use outside math/code. Why it matters: A training method that only works when you can verify exact correctness won’t help for long answers, stories, or proofs.

🍞 Anchor: Checking a multiple-choice quiz is easy; judging a history essay isn’t so clear.

🍞 Hook: You know how teachers sometimes grade by “how close” you are, not just right/wrong?

🥬 The Concept: Likelihood-based rewards give points based on how likely the model thinks the reference answer is after its reasoning. How it works: (1) For each question, there’s a reference answer (the continuation in data); (2) the model writes its CoT; (3) we measure the probability (or log-probability) of the reference answer given that CoT; (4) higher likelihood = higher reward. Why it matters: This gives a smooth, dense signal even when you can’t do exact right/wrong checks.

🍞 Anchor: It’s like partial credit: the closer your final sentence is to the teacher’s version, the more points you get.

🍞 Hook: Think of whispering a long sentence and asking someone to repeat it perfectly; the chance of an exact match shrinks with length.

🥬 The Concept: Plain probability (without logs) collapses on long answers because exact-match probabilities get tiny. How it works: (1) Multiply tiny token probabilities together → super tiny overall probability; (2) reward becomes near zero; (3) learning flatlines. Why it matters: Non-verifiable, long-form answers need a reward that doesn’t vanish with length.

🍞 Anchor: If the game is “say my 200-word paragraph exactly,” no one scores points—so no one learns.

🍞 Hook: You know how in class you learn word-by-word, sentence-by-sentence?

🥬 The Concept: Log-probability aligns with pretraining (next-token log-likelihood). How it works: (1) Pretraining teaches the model to assign high log-likelihood to the right next token; (2) using log-prob of the whole answer keeps the training signal consistent; (3) it avoids vanishing by adding logs (tiny numbers become manageable sums). Why it matters: Consistency with pretraining means stable learning and better perplexity.

🍞 Anchor: It’s like practicing scales on a piano and then using the same skills in a song.

The World Before: LLMs improved at reasoning using CoT and RL with 0/1 rewards, mainly in math/code where answers are easy to check. The Problem: Binary rewards are sparse, sensitive to verifier design, and don’t extend to long-form tasks. Failed Attempts: Intrinsic/self-judging rewards often underperform verified correctness; plain probability rewards (like VeriFree) struggle on long outputs. The Gap: We needed a single, scalable reward that works whether or not we can verify. Real Stakes: With a unified reward, we can train reasoning on everyday tasks—explaining, summarizing, and teaching—without special checkers or fragile signals.

🍞 Hook: Imagine coaching a team using a scoreboard that shows not only win/lose but also how close each play was to scoring.

🥬 The Concept: The key insight is simple: reward the model by the log-probability of the reference answer after its chain-of-thought. How it works: (1) The model writes its reasoning (CoT); (2) we compute log p(reference answer | question, CoT); (3) use that number as the RL reward; (4) optimize with standard policy-gradient tricks (like leave-one-out baselines). Why it matters: This one signal works for short, checkable answers and for long, uncheckable ones—and it doesn’t vanish with length.

🍞 Anchor: It’s like awarding points for how “on track” the final play was, not just whether the team won the whole game.

Multiple Analogies:

1. Thermometer analogy: Binary reward is “fever or no fever.” Log-prob is an actual temperature—continuous and informative.
2. Map reading: Exact-match probability is like demanding you step on every identical tile; log-prob is like rewarding you for getting close to the destination, even with small detours.
3. Graded rubrics: Instead of all-or-nothing grades, use a rubric that gives partial credit based on closeness; logs make tiny credits add up sensibly.

Before vs After:

• Before: CoT RL needed verifiers; performance good in math/code but didn’t transfer to long-form. Probability rewards often flatlined on long answers.
• After: Log-prob rewards match or beat RL on verified tasks and match SFT on long-form; perplexity improves; the same objective covers both worlds.

Why It Works (intuition, not equations):

• Logs turn tiny products into manageable sums, preventing rewards from collapsing on long answers.
• The reward matches pretraining’s next-token log-likelihood, so optimization is stable and familiar to the model.
• Log-prob discourages overconfidence in wrong answers, improving perplexity (less “confidently wrong”).

Building Blocks (mini-concepts, each as a sandwich):

🍞 Hook: You know how you list steps to solve a puzzle? 🥬 The Concept: Chain-of-Thought (CoT) is the model’s written steps. How it works: It prints thoughts, then an answer. Why it matters: CoT helps with hard reasoning, and we can score how helpful the steps are by checking the final answer’s likelihood. 🍞 Anchor: Like showing your long division steps before the final number.

🍞 Hook: Imagine giving points based on how likely a guess is to be right. 🥬 The Concept: Likelihood-based reward scores how probable the reference answer is given the CoT. How it works: Compute probability (or log-probability) after the CoT; use as reward. Why it matters: Provides a dense training signal without a dedicated verifier. 🍞 Anchor: Near-miss answers still earn points and guide learning.

🍞 Hook: Long sentences make exact repeats hard. 🥬 The Concept: Log-probability avoids vanishing rewards for long outputs. How it works: Sum token log-probs instead of multiplying tiny probs. Why it matters: The signal stays strong enough to learn. 🍞 Anchor: Counting steps instead of multiplying small chances.

🍞 Hook: Sports teams compare plays within a game. 🥬 The Concept: Leave-One-Out (RLOO) reduces noise. How it works: Compare each CoT’s reward to the others for the same question. Why it matters: Stable training with less randomness. 🍞 Anchor: Scoring a play relative to teammates’ attempts on the same drill.

🍞 Hook: Two ways to read a score: per game or per play. 🥬 The Concept: Per-answer vs per-token averaging. How it works: Average log-probs per whole answer or per token. Why it matters: Controls how much long answers weigh during training/evaluation. 🍞 Anchor: Averaging the grade for the whole essay vs each sentence.

At a high level: Question → Model writes a Chain-of-Thought (CoT) → Compute log-probability of the reference answer given that CoT → Use that as the reward → Update the model with RL (RLOO) → Repeat.

Step-by-step (like a recipe):

1. Collect data with reference continuations.

   • What: Use datasets with prompts and answers (math, code, long-form). For long-form, the “answer” is the reference continuation in the data.
   • Why: We need a target continuation to score likelihood; no verifier required.
   • Example: From MATH, a problem and its correct short answer; from Alpaca, a prompt and its long response.

2. Format prompts to elicit CoT.

   • What: Use an instruction template that asks the model to think inside <think>…</think> and then give <answer>…</answer>.
   • Why: Separates reasoning from the final answer; makes it easy to truncate for scoring.
   • Example: “Solve the problem. Think first in <think>…</think>. Then give just the final answer in <answer>…</answer>.”

3. Generate groups of CoTs per question (group size G).

   • What: For each prompt, sample G independent CoTs (e.g., G=32 for verifiable, G=4 for non-verifiable).
   • Why: Comparing CoTs for the same question reduces noise (RLOO/GRPO-style advantages).
   • Example: For a math problem, produce 32 thought traces that may differ in steps and length.

4. Compute the log-probability reward.

   • What: Reward = log p(reference answer | prompt, CoT). Optionally average per token (AvgLogProb) to normalize by length.
   • Why: Dense, stable signal aligned with pretraining; avoids vanishing on long answers.
   • Example: If the answer is “42,” compute the model’s log-likelihood of “42” after each CoT and use that as the reward.

5. Estimate advantages with RLOO (leave-one-out).

   • What: For each CoT in the group, subtract the average reward of the other CoTs for the same question.
   • Why: Stabilizes gradients by judging each attempt relative to its peers.
   • Example: If one CoT earns higher log-prob than the others, it gets a positive advantage.

6. Update the model.

   • What: Apply a policy-gradient step combining: (a) the RL term that nudges toward higher-reward CoTs, and (b) a direct supervised-like term on the answer (from the log-prob gradient).
   • Why: The RL term learns better CoTs; the supervised-like term reinforces the answer tokens in context.
   • Example: The model slightly increases chance of producing the higher-reward CoT patterns next time.

7. Decode and evaluate.

   • What: Measure success rates (greedy and sampled), log-prob metrics (per-answer/per-token), perplexity, and CoT length.
   • Why: Success shows correctness; log-prob/perplexity show confidence calibration; CoT length shows reasoning behavior.
   • Example: On MATH, track greedy accuracy and perplexity over training steps.

8. Monte Carlo for log-of-expectation.

   • What: True log p(answer | prompt with CoT marginalization) is log E_z p(answer|z). Approximate with MC by averaging over sampled CoTs, then take log (MC32 better than MC1).
   • Why: The log of an average cannot be computed exactly without summing over all CoTs; MC keeps it practical.
   • Example: For each question, average probabilities over 32 CoTs and then take log.

What breaks without each step:

• Without CoT formatting, the model’s thoughts and answers get mixed; scoring becomes noisy.
• Without likelihood reward, training in non-verifiable settings can’t proceed reliably.
• Without RLOO, gradients are high-variance; learning is unstable.
• Without MC, log-of-expectation is biased by single-sample underestimates.

Concrete data example:

• Prompt: “Find the value of x: 3x + 5 = 17.”
• CoT A: “Subtract 5 both sides: 3x=12. Divide by 3: x=4.” → High log p(“4”).
• CoT B: “Try 3: 3*3+5=14; not 17. Try 4: 12+5=17.” → Also high but maybe slightly lower. RLOO favors the higher one.

Secret sauce (what’s clever):

• Using log-prob aligns post-training with pretraining, improving perplexity and stability.
• It avoids vanishing rewards on long outputs, unlike plain probability.
• It does not need sampling of final answers to score (compute-friendly), only a single forward pass on the reference answer given the CoT.
• Same recipe works for both verified and non-verified domains, simplifying pipelines.

Mini sandwiches for related methods:

🍞 Hook: Two ways to count chances: raw chance vs log-chance. 🥬 The Concept: VeriFree uses raw probability p(answer|CoT); JEPO uses a grouped log-mean-exp across CoTs. How it works: VeriFree scores expected exact-match; JEPO tightens the log-of-expectation estimate with multiple samples. Why it matters: VeriFree can flatline on long answers; JEPO helps estimation but adds compute. 🍞 Anchor: Counting tiny raindrops (probability) vs using a rain gauge that sums steadily (log-prob).

The Test: The authors measured how well different rewards train reasoning across two verifiable math sets (MATH, DeepScaleR) and two long-form, non-verifiable sets (Alpaca, NuminaProof). They reported:

• Success rate (greedy and sampled)
• Log-prob metrics (per-answer/per-token), plus perplexity
• CoT length over training

The Competition: Baselines included Supervised Fine-Tuning (SFT) and Base RL with 0/1 rewards. Variants tested: Log-prob, AvgLogProb, Probability (VeriFree), AvgProb (RLPR-like), and JEPO.

Scoreboard with context:

• Verifiable (MATH, Qwen-3B): Greedy success ~56.84% with log-prob vs 55.85% base RL—like edging from a solid B+ to an A-. With temperature-1 sampling, all methods dip, and log-prob no longer leads; greedy decoding shows the true gains.
• Perplexity shines: On MATH (Llama-3B), log-prob achieves ~2.21 perplexity vs base RL ~13.87—like moving from guessing wildly to being calmly confident. Probability (without logs) sits in-between but much worse than log-prob.
• DeepScaleR shows similar patterns: log-prob-family gets strong greedy success and far better perplexity than base RL and probability-only rewards.
• Non-verifiable (Alpaca, NuminaProof): Log-prob, AvgLogProb, and JEPO match SFT in log-prob/perplexity. Plain probability rewards often flatline due to vanishing exact-match probabilities on long answers.

Surprising findings:

• CoT shortening: Under log-prob rewards, CoTs get much shorter early. In math, they later recover; in long-form, they collapse to very short traces (≈5–10 tokens) and stay there, making behavior similar to SFT.
• Stabilizing CoT (via KL penalties or explicit length rewards) prevents collapse but hurts performance, revealing a trade-off between “long visible thinking” and measured quality.
• Warm-starting (teaching the model to answer in the presence of CoTs before RL) stabilizes CoT somewhat but still doesn’t beat SFT on long-form within the given compute.
• JEPO vs simple log-prob: With greedy decoding, JEPO wasn’t clearly better than basic log-prob despite extra compute; the simpler method often sufficed.

Key numbers (illustrative):

• MATH (Qwen-3B, greedy): Base RL 55.85% vs Log-prob 56.84%.
• MATH (Llama-3B, perplexity): Base RL ~13.87 vs Log-prob ~2.21; SFT ~2.63.
• Non-verifiable (Alpaca, Llama-3B): Log-prob perplexity ~2.56, matching SFT ~2.56; Probability reward struggles.

Takeaways:

• For accuracy on verified tasks, several methods are comparable under greedy decoding; for confidence calibration (perplexity), only log-prob-family shines.
• For long-form, log-prob matches SFT; plain probabilities fail. One reward to rule both worlds: log-prob.

Limitations:

• CoT collapse: Log-prob rewards often shorten CoT dramatically, especially in long-form, and keeping CoT long tends to hurt measured performance.
• Compute sensitivity: JEPO can surpass in theory with more compute (as seen in other work), but here added complexity didn’t pay off within budget.
• Metric trade-offs: Optimizing success rate alone can hurt perplexity; optimizing log-prob helps perplexity but may reduce visible CoT.
• Domain scope: Results shown on 3B-scale Llama/Qwen; effects at very large scales may differ.

Required resources:

• Instruction-tuned base model, datasets with reference continuations, RL infrastructure with group sampling (G up to 32 for verified tasks), and standard optimizers.
• Optional verifiers for checked tasks; not required for log-prob training itself.

When NOT to use:

• If you must preserve long, human-readable CoTs on long-form tasks at all costs (log-prob may collapse CoT length without extra penalties, which then hurt performance).
• If your setup depends on temperature-1 sampling accuracy as the main metric; greedy decoding reveals clearer gains for log-prob.
• If you need maximal gains in unverifiable domains but have very limited compute; log-prob will likely match, not exceed, SFT under modest budgets.

Open questions:

• Can we design rewards that keep CoTs informative and long without hurting performance?
• Is there a compute regime where JEPO (or similar grouped objectives) reliably surpasses log-prob on long-form?
• Do larger models or mixed short/long curricula change the CoT collapse pattern?
• Can we measure or leverage “hidden CoT” inside the network to bridge visible-vs-internal reasoning?
• Is there a smooth path of tasks (short → medium → long answers) that preserves gains across lengths?

Three-sentence summary: This paper shows that rewarding models by the log-probability of the reference answer after their chain-of-thought is a simple, universal signal that works for both verifiable (math/code) and non-verifiable (long-form) tasks. It matches or beats standard RL on accuracy in math while strongly improving perplexity, and it matches SFT on long-form where plain probability rewards fail. Thus, one lightweight, verifier-free objective can train reasoning across many settings.

Main achievement: Establishing log-likelihood rewards as a practical, compute-friendly, and broadly effective CoT fine-tuning method that aligns with pretraining and avoids vanishing rewards on long outputs.

Future directions: Stabilize CoT length without hurting quality; explore larger models and longer training for potential gains beyond SFT on long-form; investigate curricula that blend short and long answers; study hidden CoT mechanisms and better group objectives (e.g., JEPO variants) under higher compute.

Why remember this: When in doubt, use log-prob. It’s the rare training signal that is simple, scalable, verifier-free, and strong across both short, checkable answers and long, open-ended explanations.