ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

🍞 Hook: Imagine you and your friends are planning a trip. There’s no single 'right' plan—lots of good choices depend on budget, time, and what you like. Now imagine trying to grade all those plans with just one number from 0 to 10. That’s tough! 🥬 The Concept: Big language models (LLMs) are moving from answering questions to acting like agents that plan, search, and use tools across many steps. In simple tasks like math or coding, we can check answers exactly and reward the model cleanly. But in open-ended tasks—like trip planning or deep online research—there isn’t one ground-truth answer. So, many RL systems used an 'LLM-as-judge' to give each solution a single score. Unfortunately, these pointwise scores get noisy and squish many decent answers into the same range, making it hard for RL to tell which attempt is truly better. Why it matters: Without a clear learning signal, the agent stops improving, especially on the complex, real-life jobs we want it to handle. 🍞 Anchor: Think of giving five similar essays a grade. If you give all of them an 8 or 9 because they’re all decent, you’re not telling the writers which is actually best—or how to improve.

🍞 Hook: You know how in sports, tournaments decide winners by playing matches, not by rating teams with a single number? 🥬 The Concept: The paper’s key diagnosis is 'discriminative collapse' in pointwise scoring. As the policy improves, its answers get similar, the spread of scores shrinks, and the judge’s random preferences (like loving longer answers) can outweigh real quality differences. This kills the signal-to-noise ratio. Why it matters: RL ends up chasing noise instead of real progress. 🍞 Anchor: If every ice cream flavor gets rated 8/10 but the taster’s spoon size varies randomly, you can’t tell which flavor is actually better.

🍞 Hook: Imagine judging a cake bake-off by tasting, watching the technique, and checking the recipe steps, not just staring at the final photo. 🥬 The Concept: One earlier attempt was to keep pointwise scoring but improve rubrics or add multiple criteria. Others tried coarse comparisons (like win/lose vs. a random reference) or exhaustive pairwise comparisons. These helped but had issues: pointwise stayed noisy; binary comparisons were too rough; and full pairwise comparisons were too expensive (quadratic in group size). The gap: We needed a method that (1) compares answers directly, (2) looks at the full reasoning-and-tools process, and (3) stays affordable to run during training. 🍞 Anchor: It’s like moving from 'give each cake a number' to 'run a fair bake-off bracket that considers taste, texture, and technique'—but fast enough to use every day.

🍞 Hook: Think about planning your own family trip. Tiny differences—like catching a train 15 minutes earlier—can matter a lot. 🥬 The Concept: The paper introduces ArenaRL, which turns single-number grading into a tournament among the model’s own attempts for the same question. It adds a process-aware judge that compares pairs of attempts on steps, tool use, and final answer. Then, a seeded single-elimination tournament provides a high-quality ranking signal with linear cost. Why it matters: The agent gets clear, robust guidance on which ideas and plans are better, so it keeps improving on open-ended, real-world tasks. 🍞 Anchor: Instead of 'everyone got an 8,' the model learns 'this plan beat that plan because it checked opening times and stayed under budget,' which is actually useful feedback.

🍞 Hook: If we want AI to be a helpful trip-planning buddy or a careful internet researcher, it needs to plan steps, use tools, and defend its choices—not just write pretty paragraphs. 🥬 The Concept: The authors also built two full-cycle benchmarks—Open-Travel (with multi-constraint travel tasks) and Open-DeepResearch (with multi-turn search and synthesis). Both include SFT data, RL splits, tools, and automatic evaluation. Why it matters: Public benchmarks that include training and testing let the community reproduce results and push the field forward. 🍞 Anchor: It’s like not just having a final exam, but also the homework, practice tests, and a fair grading guide everyone can use.

Now let’s walk through the key ideas in the right order, so each piece builds on the last:

🍞 Hook: You know how dogs learn tricks with treats? 🥬 Reinforcement Learning (RL): RL is a way for AI to learn by trying actions and getting feedback (rewards), then doing more of what works. Why it matters: Without good rewards, the AI can’t learn the right tricks. 🍞 Anchor: If 'sit' always gets a treat and 'jump on the couch' gets none, the dog (and the AI) figures it out.

🍞 Hook: When everyone’s essays are pretty good, it’s hard to pick a winner. 🥬 Discriminative Collapse: When many answers are all 'good' and scored similarly, small differences vanish and judge noise dominates. Why it matters: RL can’t tell what to copy and what to drop. 🍞 Anchor: If three pizzas all get 8/10, you miss that one had burnt crust and one nailed the timing.

🍞 Hook: Sports don’t crown champions with random numbers—they run matches. 🥬 Tournament-based Relative Ranking: Instead of scoring each answer alone, compare answers within the same group and rank them. Why it matters: Relative judgments are more stable and informative than single numbers. 🍞 Anchor: A mini soccer bracket between classmates quickly reveals who plays better.

🍞 Hook: Choosing between two ice creams is easier than grading each from 1 to 10. 🥬 Pairwise Comparison: Judge two trajectories head-to-head using a rubric, and decide which is better (and by how much). Why it matters: It reduces noise and makes fine differences visible. 🍞 Anchor: A taste test between chocolate and strawberry tells you which you prefer right now.

🍞 Hook: When judging a science fair, you watch the experiment steps, not just the poster. 🥬 Process-aware Evaluation: Don’t just rate the final answer—check reasoning steps and tool use. Why it matters: It rewards real thinking and good actions, not just flashy endings. 🍞 Anchor: The winner is the volcano that worked safely, measured correctly, and explained results clearly.

🍞 Hook: At tournaments, strong teams are seeded so they don’t knock each other out early. 🥬 Seeded Single-Elimination: Use a high-quality 'anchor' attempt for initial seeding, then run a single-elimination bracket to produce a ranking with only O(N) comparisons. Why it matters: Almost the accuracy of round-robin at a fraction of the cost. 🍞 Anchor: Like March Madness, but with smarter seeding so the best teams meet later.

🍞 Hook: Picture a classroom where every essay gets scored 8 or 9. Not much to learn from that, right? What if, instead, we asked, 'Which essay is better and why?' and used mini-tournaments to find out? 🥬 The Concept (Aha! in one sentence): ArenaRL replaces shaky single-number scores with tournament-based, process-aware pairwise comparisons to generate clear, robust learning signals for open-ended agents. Why it matters: This turns fuzzy judgments into solid guidance so agents keep getting better at long, tricky tasks. 🍞 Anchor: It’s like swapping a flat 'B+' for a real playoff that shows which project actually wins and why.

Multiple Analogies (3 ways to see it):

1. Sports league: Instead of rating every team with one number, have them play matches; seed strong teams so the bracket makes sense; reward the ones who advance. Stable, fair, and informative.
2. Cooking show: Judge dishes head-to-head while watching the cooking steps. You don’t just taste the plate; you check technique and timing—then pick the better chef.
3. Science lab: Students run experiments and share logs. The teacher compares two reports side-by-side, valuing careful setup and accurate measurements as well as the conclusion.

Before vs After:

• Before (Pointwise): Each answer got a single score, often bunched together (like 0.8–0.9), with noise (length bias, sampling randomness) blurring real differences. RL stalled.
• After (Relative Ranking): The agent generates several attempts, compares them pairwise with a rubric (including steps and tool use), then ranks them via a seeded tournament. RL learns from clearer, more truthful advantage signals.

Why It Works (intuition, not equations):

• Relative is easier: People (and LLM judges) are better at choosing between two options than assigning absolute scores to one option.
• Process matters: Looking at reasoning and tool calls stabilizes judging, so the model learns real skills (planning, checking constraints) rather than chasing surface patterns.
• Seeding tames randomness: Using a strong 'anchor' attempt to seed the bracket reduces early upsets where two great answers knock each other out too soon.
• Linear cost: Single-elimination needs only O(N) comparisons but preserves most of the ranking quality of O(N^2) round-robins.

Building Blocks (bite-sized, with the sandwich pattern):

• 🍞 Hook: Picking the best photo from a stack is easier by comparing two at a time. 🥬 Pairwise Comparison: Judge two trajectories head-to-head with a rubric that scores both. Why it matters: Cuts noise and highlights subtle advantages. 🍞 Anchor: Compare two travel itineraries: one checks museum hours and budget; the other forgets. The better one clearly wins.
• 🍞 Hook: Watching how a cake is baked tells you more than just tasting it. 🥬 Process-aware Rubric: Evaluate the chain-of-thought, tool use, and final answer together. Why it matters: Encourages careful steps and good tool choices. 🍞 Anchor: The itinerary that verifies opening times, maps routes, and stays on budget gets the nod.
• 🍞 Hook: Seed the tournament so great teams don’t clash too early. 🥬 Seeded Single-Elimination: Use an 'anchor' plan (greedy decoding) to pre-rank seeds, then run a single-elimination bracket. Why it matters: High ranking fidelity at low cost. 🍞 Anchor: Anchor says which plans look promising; the bracket confirms who really wins.
• 🍞 Hook: A class playoff reveals a clear ranking fast. 🥬 Tournament-based Relative Ranking: Turn many attempts into a sorted list from best to worst within the same question. Why it matters: Produces clean 'advantages' for RL to learn from. 🍞 Anchor: 8 candidate itineraries enter; one champion plan (and a clear order) emerges.
• 🍞 Hook: Treats motivate tricks only if given for the right moves. 🥬 Rank-to-Advantage Mapping: Convert ranks into stable learning signals (advantages) and update the policy while staying near a safe reference. Why it matters: Prevents wild swings and keeps learning steady. 🍞 Anchor: Reward the champion more than the runner-up, but don’t forget to nudge mid-tier plans in the right direction.

High-level Recipe: Input (a user query) → Generate a group of candidate trajectories → Compare them with a process-aware pairwise judge → Run a seeded single-elimination tournament to rank them → Turn ranks into advantages → Update the policy (with a safety leash) → Output: a better agent.

Step-by-step (what, why, example):

1. Sample a Trajectory Group

• What happens: For one user query (e.g., 'Plan a 3-day Beijing trip under 2000 yuan'), the policy generates N different trajectories (plans), mixing one greedy 'anchor' (deterministic) plus N−1 diverse samples (higher temperature).
• Why this step exists: You need multiple candidates to compare; diversity increases the chance that at least one plan is truly better.
• Example: 8 plans vary in day ordering, transport choices, and budget splits; one is the anchor.

1. Process-Aware Pairwise Judge

• What happens: For any two trajectories, the LLM judge looks at their reasoning steps, tool calls (like map routes or price searches), and final answers, then assigns scores to both. The order of presentation is swapped in a second pass to reduce 'first position' bias; scores are combined.
• Why this step exists: Judging only final answers misses whether the model reasoned correctly and used tools well. Process-aware judging rewards real skill.
• Example: Plan A checks museum hours, uses navigation tools correctly, and stays within budget; Plan B forgets a closing time and overspends. The judge prefers A.

1. Seeding via Anchor-Based Ranking

• What happens: Each exploratory plan is compared once with the anchor. The anchor’s average score and each plan’s score produce a preliminary order (seeds).
• Why this step exists: Random brackets can cause early knockouts of strong plans. Seeding with a decent prior reduces bad luck.
• What breaks without it: Two excellent plans might face off in Round 1 by chance, distorting the final ranking.
• Example: If the anchor is decent, top seeds are those that consistently beat it; weaker ones fall lower.

1. Seeded Single-Elimination Tournament

• What happens: Build a binary bracket: highest seed vs. lowest seed, and so on. In each match, the pairwise judge decides who advances. Losers are grouped by elimination round; ties within a tier are broken by accumulated average scores from their matches.
• Why this step exists: It creates a nearly round-robin-quality ranking with only O(N) comparisons.
• What breaks without it: Full round-robins are O(N^2) and too slow for online RL; random single-elimination is too noisy.
• Example: Seed 1 cruises to the final; a mid-seed upsets a higher seed thanks to better tool use; the final ranks reflect survival depth and consistent wins.

1. Rank-to-Advantage Conversion

• What happens: Map ranks to quantile-style rewards (top rank ≈ 1.0, bottom ≈ 0.0), standardize within the group, and use a PPO-style clipped objective with a KL penalty to keep the new policy from straying too far from a reference (safety leash).
• Why this step exists: Stable, normalized signals prevent the 'noise explosion' seen when group variance shrinks under pointwise scoring.
• Example: Champion plan gets a strong positive advantage; mid-tier plans get small nudges; the worst gets negative feedback.

1. Policy Update

• What happens: Optimize the policy to make the good trajectories more likely next time, while penalizing big jumps away from the reference.
• Why this step exists: Encourages steady, reliable improvement without collapsing behavior.
• Example: The model increases the probability of checking hours before proposing a route and reduces the chance of ignoring budget.

Other Tournament Topologies (and why seeded single-elimination wins):

• Round-Robin (gold standard): Everyone plays everyone (O(N^2)). Very accurate but too expensive for online training.
• Anchor-Only Ranking: Linear cost but can’t compare two exploratory samples directly, losing resolution among mid-tier plans.
• Double-Elimination: More forgiving than single-elimination, but without good seeds it underperforms the seeded variant at similar cost.
• Swiss-System: O(N log N) with dynamic pairing. Reasonable, but empirically not as strong as the seeded single-elimination under the same budget.

Secret Sauce (why this is clever):

• Seeding with a strong anchor gives a low-bias prior that organizes the bracket well.
• Bidirectional pairwise judging cancels position bias and reduces noise.
• Process-aware rubrics reward reasoning quality and tool discipline, not just final polish.
• Linear-time tournaments make comparison-based RL feasible at scale.
• Group size is a knob: larger N increases exploration diversity and improves learning signals.

Concrete mini-walkthrough:

• Query: 'Plan a one-day Shanghai trip for a family of four, under 800 RMB, with a river view at sunset.'
• Generate N=8 plans (1 anchor + 7 samples).
• Seed: Compare each sample vs. the anchor once; compute preliminary seeds.
• Tournament: Seed 1 vs. 8, 2 vs. 7, etc.; winners advance; losers tiered by round.
• Ranking: Champion > runner-up > semifinal exits > quarterfinal exits (ties broken by average match score).
• RL update: Convert ranks to advantages and run a clipped, KL-regularized update.
• Next time: The agent more reliably uses 'around search' near the Bund after 10 PM, checks prices, and ensures a sunset view—all within budget.

The Test (what and why):

• The authors evaluate ArenaRL on two new, realistic, end-to-end agent benchmarks: Open-Travel (multi-constraint itinerary planning with tool calls) and Open-DeepResearch (long-horizon web search, reading, synthesis, and reporting). They also test on open-ended writing benchmarks.
• Key metrics: win rate in pairwise LLM-judge evaluations (how often ArenaRL’s output beats a strong baseline) and valid generation rate (how often the model completes the task without running out of context or failing).

The Competition (who vs. whom):

• RL baselines: GRPO and GSPO trained with pointwise scalar rewards (same judges and rubrics for fairness), plus an SFT-only model.
• Closed-source models: GPT-4o, Grok-4, Gemini-2.5-pro, and Claude-3.7-Sonnet.

Scoreboard with Context:

• Tournament topology study (Open-Travel): Seeded single-elimination reached an average win rate of 32.5%, nearly matching the round-robin 'gold standard' of 32.9%, while using only O(N) comparisons. That’s like scoring A- when the perfect but too-slow method scores A.
• Main results (Open-Travel): ArenaRL achieved 41.8% average win rate vs. GRPO (16.4%) and GSPO (17.2%), and surpassed the listed closed-source models on this benchmark. Think of it as jumping from mid-class rankings to the honor roll.
• Main results (Open-DeepResearch): ArenaRL hit 64.3% average win rate with a 99% valid generation rate, versus SFT’s 32% valid rate (and baselines even lower). That’s like finishing almost every marathon you start—and winning many of them—while others often can’t reach the finish line.
• Open-ended writing: Across three public writing benchmarks, ArenaRL delivered the best overall mean, beating GRPO by ~6.7 points and GSPO by ~7.3, and surpassing some closed-source models. It’s not just better at tools and plans; it also writes more clearly and coherently.

Surprising (and interesting) findings:

• Seeding matters a lot: Seeded single-elimination sometimes even outperformed round-robin on specific subtasks, suggesting that seeding filters noise and prevents strong candidates from colliding too early.
• Scaling group size helps: Increasing N (e.g., from 8 to 16) noticeably boosted performance—more diverse attempts mean better chances to find and learn from standout trajectories.
• Consistency with humans: LLM-vs-human agreement reached ~74%, implying ArenaRL’s improvements aren’t just overfitting to a particular judge.
• Cold-start resilience: Even without supervised warm-up, ArenaRL’s relative ranking signal drove steady improvements from 'can’t do it at all' to strong performance on a travel subtask.

Business-side validation:

• On quantifiable POI search, ArenaRL-tuned models improved search accuracy by 75–83% over baseline.
• On open-ended planning queries (e.g., ambiance preferences and logistics), core metrics rose from 69% to 80%—evidence that ArenaRL’s learned planning skills transfer to real user needs.

Limitations (clear-eyed view):

• Seeding quality matters: If the greedy 'anchor' is weak or biased, initial seeds may mislead the bracket. The method reduces but doesn’t eliminate this risk.
• Judge dependence: The approach relies on a strong LLM judge and well-designed rubrics. If the judge has systematic biases (e.g., length preference), that can leak into training—though bidirectional scoring helps.
• Compute and latency: Pairwise judging and tournaments are more expensive than single scores per sample (even if O(N) overall). Very large groups or extremely long contexts can still be costly.
• Domain coverage: The process-aware rubric and tool feedback were tuned for travel and research domains. New domains may need updated rubrics and tool interfaces.
• Extremely objective tasks: For tasks with exact, automatic ground truth (like math unit tests), tournament ranking may be unnecessary overhead compared to direct verifiable rewards.

Required Resources:

• Models and judges: A capable base model (e.g., Qwen3-8B) and a strong judge (e.g., Qwen3-Max or similar) to provide reliable pairwise signals.
• Compute: Multi-GPU training (the paper uses H20 GPUs) for SFT and RL phases; memory for long-context reasoning and tool traces.
• Data and tools: SFT data for cold-start (optional but helpful), RL query sets, and access to real or simulated tools (maps, search, tickets) with logging.

When NOT to Use:

• Deterministic, verifiable tasks (math/code with unit tests) where pointwise ground-truth rewards are flawless and cheap.
• Ultra-low-resource settings where even O(N) pairwise comparisons per group are too expensive.
• Safety-critical contexts without extra guardrails; better combine with alignment checks and human oversight.

Open Questions:

• Smarter seeding: Could ensembles or small learned rankers replace the single anchor to reduce seed bias further?
• Judge robustness: How to mix judges (or human-in-the-loop spot checks) to reduce systematic biases and improve generalization?
• Theory: Formal analysis of signal-to-noise improvements under relative ranking and how group size and topology interact.
• Multimodal extension: Bring the same ideas to agents handling text, images, maps, and audio together.
• Dynamic group sizing and curricula: Adapt N and tournament depth over training to balance compute and learning progress.

Three-sentence summary: ArenaRL upgrades RL for open-ended agents by swapping noisy, pointwise scores for process-aware, pairwise comparisons organized as a seeded single-elimination tournament. This delivers robust, low-cost (linear-time) relative rankings that turn into clear learning signals, avoiding discriminative collapse. Across travel planning, deep research, and open-ended writing, ArenaRL beats strong baselines and several closed-source models, with especially large gains in valid long-context completions.

Main achievement: Showing that tournament-based relative ranking—with process-aware judging and smart seeding—can scale comparison-driven RL to real-world, long-horizon agent tasks, matching the fidelity of exhaustive comparisons at a fraction of the cost.

Future directions: Sharpen seeding with small learned rankers or judge ensembles, extend to multimodal agents, study theory of ranking SNR, and add human-in-the-loop safety checks. Explore adaptive group sizes and hybrid topologies for even better efficiency-accuracy trade-offs.

Why remember this: It reframes how we train open-ended agents—from 'grade each answer with one shaky number' to 'stage fair mini-competitions that reward real reasoning and good tool use.' That shift makes complex planning and research agents more practical, reliable, and ready for everyday use.