Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR

🍞 Hook: Imagine you’re writing answers for a math contest. If the teacher graded you mostly by how short or long your answer is, instead of how correct and clear it is, you might start changing your writing style to match the grading trick. That wouldn’t be fair, right?

🥬 The Concept (RLVR — Reinforcement Learning with Verifiable Rewards):

• What it is: RLVR is a way to train AI where the model tries answers, gets a reward that can be checked (like “correct/incorrect”), and then learns to do better next time.
• How it works:
  1. The model sees a question and writes several possible answers.
  2. A checker (the verifier) scores each answer based on rules (correctness, format, etc.).
  3. The model updates itself to make good answers more likely in the future.
• Why it matters: Without RLVR, models may sound fluent but struggle with tricky reasoning like math or multi-step logic. 🍞 Anchor: Think of a quiz game where each answer can be checked—AI gets a point for right answers and learns to score more points over time.

🍞 Hook: You know how a group of friends can help you judge which solution seems best by comparing them all side by side?

🥬 The Concept (GRPO — Group Relative Policy Optimization):

• What it is: GRPO improves a model by comparing a group of answers to the same question and rewarding those that beat the group average.
• How it works:
  1. Generate G answers to one question.
  2. Score each answer.
  3. Compute each answer’s “advantage” as how much it beats (or loses to) the group average.
  4. Adjust the model so answers with higher advantage become more likely.
• Why it matters: It removes the need for a separate value model and focuses learning on relative quality. 🍞 Anchor: Like a science fair where judges compare projects side-by-side and push the winners’ ideas forward.

🍞 Hook: Imagine grading a whole story by looking at the story as one piece, not by checking each word one by one.

🥬 The Concept (GSPO — Group Sequence Policy Optimization):

• What it is: GSPO is like GRPO but scores and adjusts at the whole-answer (sequence) level instead of token-by-token.
• How it works:
  1. Treat the entire answer as a single unit.
  2. Compare how likely the old model vs. the new model thinks that whole answer is (importance ratio at sequence level).
  3. Clip extreme ratios for stability (so updates don’t explode).
  4. Push up good whole answers; push down bad ones.
• Why it matters: GSPO is more stable, especially for MoE models, where token-level updates can get noisy. 🍞 Anchor: It’s like coaching a basketball team by reviewing each full game tape, not just single plays.

🍞 Hook: Suppose your class starts giving extra credit if your essay is short. Suddenly, students might write super short essays—even if the ideas need more space!

🥬 The Concept (Response length bias):

• What it is: A hidden push in the training math that makes the model prefer shorter (or longer) answers, unrelated to real quality.
• How it works:
  1. In GRPO, averaging over tokens makes each token in short answers count more, so short good answers get stronger boosts.
  2. In GSPO, sequence-level clipping and Clip-Higher prune many gradients, often leaving more short, positive samples to dominate.
  3. Over time, models drift toward shorter replies—even when longer reasoning is helpful.
• Why it matters: If the model “learns” that short equals good, it may stop explaining steps, hurting complex reasoning. 🍞 Anchor: Like grading essays mostly by length; students game the system instead of learning to reason clearly.

🍞 Hook: Picture a school with many teachers, each great at a different subject, teaming up to help you.

🥬 The Concept (Mixture-of-Experts, MoE):

• What it is: A model made of many “experts,” where only some experts are activated for each input.
• How it works:
  1. A router picks a few best experts for the current question.
  2. Those experts process the input.
  3. Their outputs are combined into the final answer.
• Why it matters: It’s efficient and powerful, but training can get unstable if updates swing too hard. 🍞 Anchor: Like calling the math teacher for math and the art teacher for art, instead of having every teacher speak at once.

The World Before:

• AI models were good at sounding fluent but not at careful step-by-step reasoning.
• RLVR helped by rewarding correct, checkable answers—but the training math accidently introduced a bias toward certain answer lengths.

The Problem:

• GRPO and GSPO, while strong, made long answers count less per token (GRPO) and clipped too many gradients in a way that favored short answers (GSPO). This caused “length collapse,” especially in vision-language training, where answers shrank over time.

Failed Attempts:

• Switching from token-level (GRPO) to sequence-level (GSPO) updates improved stability but didn’t remove the core length bias.
• Techniques like Clip-Higher improved some stability aspects but unintentionally amplified the bias toward short positives.

The Gap:

• We needed an objective that is fair to both short and long answers—so models can write as long as needed to reason well, without gaming the training math.

Real Stakes:

• In math, coding, and science questions, solutions need multiple steps. If AI cuts explanations short, it makes more mistakes and is less trustworthy.
• In multimodal tasks (with images), shrinking answers means less exploration of visual clues, hurting performance.

🍞 Hook: You know how race judges sometimes adjust times if the track is longer or shorter, so the scores are fair? Training AI needs that kind of fairness for answer length.

🥬 The Concept (LUSPO — Length-Unbiased Sequence Policy Optimization):

• What it is: LUSPO is a simple change to GSPO that makes training fair to answers of all lengths by scaling each sequence’s loss by its own length.
• How it works:
  1. Keep GSPO’s idea of treating each answer as a whole sequence.
  2. Compute how much to push up or down the sequence (using its reward and importance ratio).
  3. Multiply that push by the answer’s length so long and short answers contribute fairly overall.
  4. Apply clipping as usual, then update the model.
• Why it matters: Without this correction, models learn to shrink answers; with it, models safely explore longer, step-by-step reasoning and perform better. 🍞 Anchor: It’s like grading essays by quality and normalizing for length, so a great short essay and a great long essay are both judged fairly.

The “Aha!” Moment in one sentence:

• If we scale each sequence’s learning signal by its length, we remove the hidden push that punishes long answers, unlocking better reasoning.

Three Analogies:

1. Essay Fairness: Don’t let word count decide the grade—judge ideas fairly by adjusting for length.
2. Team Highlights: When comparing whole games, give equal credit whether the team played 30 or 60 minutes, scaling the evaluation so it’s fair.
3. Marathon vs. Sprint: Normalize scores so long races and short races are judged on performance, not distance.

Before vs. After:

• Before: GSPO often clipped gradients so that short, positive examples dominated. Responses got shorter over time, hurting reasoning (especially in multimodal tasks).
• After: LUSPO restores balance. Response length grows healthily, exploration increases, and benchmarks improve across dense and MoE models.

Why It Works (intuition, not math):

• GRPO averaged token signals, making each token in a short answer louder than a token in a long answer.
• GSPO looked at the whole answer but clipped in a way that often removed more negative tokens and over-weighted short positives.
• LUSPO evens the scales: multiplying by length stops long answers from being muted and prevents short replies from being over-amplified.

Building Blocks:

• Sequence-level thinking (from GSPO): judge the whole answer for stability.
• Verifiable rewards: keep correctness as the north star.
• Length scaling (new): ensure the learning signal is unbiased by length.
• Clipping for safety: keep updates stable.
• Works with both dense and MoE: robust across architectures.

High-Level Recipe (Input → Steps → Output):

• Input: A batch of questions (text or image+text), and a current model.
• Steps:
  1. Sample G answers per question from the old policy.
  2. Score each answer with verifiable rewards (correctness, format, overlong penalty).
  3. Compute group advantages (how much each answer beats the group average).
  4. Compute sequence-level importance ratios (how much the new policy likes that whole answer vs the old one), then clip for stability.
  5. Multiply each sequence’s training signal by its own length (the LUSPO fix).
  6. Average within the batch and update the model.
• Output: A new model that learns to produce clear, correct, and appropriately long reasoning.

🍞 Hook: Think of cooking a big stew; you taste the whole stew (sequence), not each tiny grain of salt (token), and you adjust seasoning fairly whether the pot is big or small (length scaling).

🥬 The Concept (Sequence-level importance weighting and clipping, with length scaling):

• What it is: Measure how much the new model agrees with a full answer, keep extreme signals safe with clipping, and then correct for length.
• How it works:
  1. Whole-answer comparison keeps noise low.
  2. Clipping prevents wild jumps.
  3. Length scaling removes unfair preferences for short answers.
• Why it matters: Without it, training stability fights against reasoning depth. 🍞 Anchor: Like adjusting the spice level of a full pot, then making sure a big pot and a small pot get fair comparisons by portion size.

Each Step in Detail:

1. Sampling responses (G per question)

• What happens: For each prompt, generate multiple candidate answers with temperature and top-p sampling.
• Why it exists: More candidates mean better comparison and a clearer signal of what’s good.
• Example: For a geometry problem, produce 8 different solution attempts.

1. Rewarding answers (verifiable rewards)

• What happens: Compute a reward that includes accuracy (0/1), format (0 or 0.5), and an overlong penalty that softly discourages exceeding a buffer near the max length.
• Why it exists: Rewards define the goal—correct, well-formatted, not absurdly long answers.
• Example: An answer that’s correct, well-formatted, and within length buffer might get 1.5; a wrong, messy, too-long answer might get close to 0 or slightly negative after penalty.

1. Group advantage (relative to peers)

• What happens: For the 8 answers to the same question, compute each answer’s advantage as how much it beats the group average (normalized by group spread).
• Why it exists: Removes the need for a separate value model and focuses learning on “better than your siblings.”
• Example: If your answer scores highest among the 8, its advantage is positive; the worst answer’s advantage is negative.

1. Sequence-level importance ratio and clipping

• What happens: Compare how likely the whole answer is under the new model vs. the old one; then clip the ratio within a safe range (with optional Clip-Higher tweaks).
• Why it exists: Prevents unstable updates, especially in MoE models.
• Example: If the new model over-favors a bad answer, clipping stops a harmful update.

1. Length scaling (the LUSPO fix)

• What happens: Multiply each sequence’s (already clipped) learning signal by the answer’s length.
• Why it exists: Neutralizes the hidden math that punishes long answers and over-rewards short ones.
• Example: A 300-token correct solution and a 100-token correct solution now have fair total influence.

1. Update the model

• What happens: Average signals across sequences and backpropagate to adjust parameters using AdamW with a small learning rate and warm-up.
• Why it exists: This is how the model actually learns from the signals.
• Example: After several steps, the model shows longer, clearer step-by-step reasoning.

Concrete Mini Example:

• Prompt: “Compute 12×13.”
• Candidates:
  • A: “156” (length 1, correct, good format) → reward 1.5
  • B: “First, 12×10=120, 12×3=36, add to get 156. So 156.” (length 22, correct, good format) → reward 1.5
  • C: “Uh… maybe 158?” (length 5, wrong) → reward 0
• In plain GSPO: Short correct A can dominate because of clipping and no length correction; over time, replies may shrink.
• In LUSPO: A and B get equal reward quality, but B’s longer reasoning isn’t muted—length scaling keeps it fairly weighted. C is pushed down.

Training and Implementation Notes (from the paper):

• Models: Qwen2.5-7B-Base (dense), Qwen3-30B-A3B-Instruct (MoE), Qwen2.5-VL-7B-Instruct (vision-language).
• Compute: 8 H800 GPUs for 7B and VL; 4×8 H800 GPUs for 30B MoE.
• Rollout: 128 prompts per batch, 8 responses per prompt; top-p 0.7, temperature 1.0.
• Optimizer and schedule: AdamW, LR 1e-6, linear warm-up 20 steps.
• Max generation length: 32,768 (text), 4,096 (VL); overlong penalty uses a buffer (e.g., 512 for VL, 4096 for text) to discourage only very long outputs.

The Secret Sauce:

• A one-line conceptual change—multiply each sequence’s signal by its own length—unlocks fair learning across lengths, prevents length collapse, and keeps GSPO’s stability benefits.

The Test: What and Why

• They measured three main things: accuracy (did the model get the problem right?), response length (is the model explaining enough?), and validation curves (does training generalize beyond the training set?).
• Why: Longer, thoughtful answers often mean better reasoning—but only if length is used wisely. We want more correct answers and stable training without shrinking replies.

The Competition (Baselines)

• GRPO: Strong performance but token-averaging introduces length bias; unstable for MoE.
• GSPO: More stable via sequence-level updates, but amplifies length bias, sometimes causing length collapse.
• LUSPO: The proposed method removes length bias by length scaling.

The Scoreboard (with context)

• Text-only, Qwen2.5-7B-Base (dense):
  • AIME24: LUSPO beats GSPO by +2.9 points (like moving from a low B to a solid B+).
  • AIME25: +2.7; AMC23 and MATH500 also improve, with average gain about +4.0.
• Text-only, Qwen3-30B-A3B-Instruct (MoE):
  • AIME24: LUSPO +6.9 over GSPO (a big jump—like from a B to an A-).
  • AIME25: +17.1 (very large improvement—like leaping from a C+ to a B+/A- range).
• Vision-language, Qwen2.5-VL-7B-Instruct:
  • MathVista-mini: LUSPO +1.6 over GRPO and +0.5 over GSPO.
  • WeMath and LogicVista: LUSPO +5.1 and +6.0 over GSPO, respectively (major gains on tough reasoning).

Response Length Growth

• LUSPO raises average response length to about 1.5× GSPO across models (e.g., 2611 vs 3940 tokens, dense; 6757 vs 10114 tokens, MoE), showing increased exploration space.
• In multimodal training, GSPO suffers length collapse; LUSPO avoids this and maintains healthy lengths.

Training Dynamics and Validation

• Accuracy reward during training is consistently higher for LUSPO than GSPO (dense, MoE, and VL), meaning better progress per step.
• Validation curves (e.g., avg@32 on AIME24) climb higher with LUSPO, suggesting gains are not just memorization.

Surprising Findings

• GSPO sometimes underperforms GRPO on multimodal benchmarks due to amplified length bias from sequence-level clipping and Clip-Higher.
• A simple length scaling in LUSPO reverses this: stability is preserved while bias is removed.

Bottom Line

• LUSPO provides state-of-the-art optimization among tested methods, giving both longer, meaningful reasoning chains and better scores across diverse settings.

Limitations

• LUSPO improves fairness across lengths, but it still relies on reward design. If rewards overly favor verbosity or extreme brevity, models may drift again.
• Very long sequences increase compute and memory costs; even with fair training, practical budgets may cap length.
• Length scaling interacts with clipping; extreme settings of Clip-Higher or epsilon could still distort signals.

Required Resources

• Substantial compute: training used clusters of H800 GPUs and long max sequence lengths (up to 32K tokens for text).
• Quality verifiers and datasets: performance depends on reliable correctness checks and suitable prompts.

When NOT to Use

• Tasks where strict brevity is essential (SMS replies, on-device assistants with tight latency/memory) may not benefit from promoting longer reasoning.
• If the verifier is noisy or easily gamed, longer answers might ‘farm’ reward without real quality.

Open Questions

• Can we adaptively scale by “useful steps” instead of raw token count (e.g., detect genuine reasoning structure)?
• How should length scaling interact with different clipping schemes or soft-gating methods (e.g., SAPO)?
• What are the best settings for multimodal tasks with varied input complexities?
• Can we prove stronger theoretical guarantees about convergence and bias removal under different data distributions?

Three-Sentence Summary

• The paper finds that GRPO and GSPO have a hidden length bias that skews training toward shorter answers, which can harm complex reasoning and even cause length collapse.
• It introduces LUSPO, a length-unbiased variant of GSPO that multiplies each sequence’s learning signal by its own length, restoring fairness and stability.
• Across dense, MoE, text-only, and vision-language models, LUSPO yields longer, more effective reasoning and higher accuracy than GRPO and GSPO.

Main Achievement

• A minimal, principled fix—length scaling at the sequence level—that neutralizes response length bias and unlocks better reasoning performance.

Future Directions

• Explore adaptive scaling based on reasoning structure, integrate with soft clipping methods, and refine reward design for richer step-by-step verification.
• Extend to domains like code generation, scientific writing, and step-by-step planning, where controlled length matters.

Why Remember This

• It shows that tiny tweaks in the training objective can have outsized effects on how models think. By making training fair to length, we let models explain their thoughts clearly—and that leads to smarter, more trustworthy AI.