Experiential Reinforcement Learning

🍞 Top Bread (Hook): You know how after a tough math quiz, a good way to get better is to look at the problems you missed, figure out why, and write down a tip so you don’t make the same mistake again? That little reflection makes the next quiz go much better.

🥬 Filling (The Actual Concept: Reinforcement Learning)

• What it is: Reinforcement Learning (RL) is when an AI learns by trying actions and receiving rewards or no rewards, like a game scoring system.
• How it works:
  1. The model tries something.
  2. The environment gives a score (reward) after seeing the whole try.
  3. The model tweaks its strategy to get more reward next time.
• Why it matters: Without RL, an AI can only copy examples (imitation) and can’t adapt well when rules aren’t pre-taught. 🍞 Bottom Bread (Anchor): Like training a dog: sit gets a treat, jump on the couch gets nothing. Over time, the dog learns the rewarded behavior.

🍞 Top Bread (Hook): Imagine a video game that only tells you “win” or “lose” at the end, with no hints. That’s frustrating and slow to learn.

🥬 Filling (The Actual Concept: Sparse Rewards)

• What it is: Sparse rewards are rare or delayed signals (often just at the end) telling you if you succeeded.
• How it works:
  1. The agent plays through many steps.
  2. It only gets a 1 (win) or 0 (lose) at the end.
  3. It must guess which earlier steps were good or bad.
• Why it matters: Without frequent hints, the agent struggles to connect actions to outcomes—learning becomes slow and unstable. 🍞 Bottom Bread (Anchor): In FrozenLake or Sokoban, you only get a point if you finish perfectly; there’s no “warm/cold” feedback along the way.

🍞 Top Bread (Hook): Think about planning a trip with several bus changes. One wrong transfer can ruin the whole plan.

🥬 Filling (The Actual Concept: Multi-Step Reasoning)

• What it is: Multi-step reasoning means solving problems that need a chain of decisions where early choices affect later ones.
• How it works:
  1. Consider the current state.
  2. Predict how an action changes the state.
  3. Repeat across many steps to reach a goal.
• Why it matters: If the agent can’t reason across steps, tiny mistakes snowball into failures by the end. 🍞 Bottom Bread (Anchor): In Sokoban, pushing a box into a corner early can make the puzzle impossible later.

🍞 Top Bread (Hook): After a soccer game, great players watch clips, reflect on mistakes, and update their playbook. That reflection turns losses into lessons.

🥬 Filling (The Actual Concept: Experiential Learning)

• What it is: Experiential learning is the human-like cycle of experience, reflection, and trying again with a revised plan.
• How it works:
  1. Do something (experience).
  2. Think about what worked or failed (reflection).
  3. Form better rules (conceptualize).
  4. Try again (experiment) and repeat.
• Why it matters: Without reflection, you just repeat trial and error, learning slowly from scattered signals. 🍞 Bottom Bread (Anchor): A student checks a wrong algebra step, writes “combine like terms first,” and aces the next problem.

The world before: Large language models (LLMs) got good at copying patterns from examples (supervised fine-tuning). Then came RL with verifiable rewards (RLVR), which lets models improve through interaction by optimizing a numeric score. But in many real tasks, rewards are delayed and sparse. The model must somehow guess which part of a long attempt caused success or failure. This makes training unstable (tug-of-war updates), sample-inefficient (lots of flailing), and especially tough when a task needs multi-step reasoning.

The problem: Plain RL turns all feedback into a single number. That number is helpful but vague—like a teacher who only writes “B-” with no comments. The model has to implicitly discover how to fix itself, which is slow and noisy.

Failed attempts:

• Pure imitation (SFT): good at following examples, bad at fixing new mistakes or adapting to new environments.
• RLVR alone: learns, but often via back-and-forth randomness when signals are sparse.
• Reflection at inference-time only: can help, but makes deployments slower and riskier, and improvements aren’t “baked into” the base policy.

The gap: We need a way to turn that vague reward into explicit lessons right inside training—and then store those lessons so the model acts better next time without needing extra reflection.

Real stakes: This matters for everyday AI helpers—like a tutor that improves how it explains, a household robot that learns not to bump into walls, a search assistant that checks sources before answering, or a code helper that stops repeating the same bug. Turning feedback into durable lessons makes AI safer, faster, and more reliable in the tasks we actually care about.

🍞 Top Bread (Hook): Imagine learning piano with a smart practice routine: play once, think about what tripped you up, play again using your note-to-self, then memorize the better version so next time you play it right from the start.

🥬 Filling (The Actual Concept: Experiential Reinforcement Learning, ERL)

• What it is: ERL is a training method that adds an experience–reflection–consolidation loop inside RL so the model can correct itself before updates, then internalize the fix.
• How it works:
  1. First attempt: the model tries to solve the task.
  2. Feedback: the environment returns a score and text feedback.
  3. Self-reflection: the model writes a brief, structured “how to fix it” note.
  4. Second attempt: the model retries, guided by its reflection.
  5. Reinforce: reward the attempts and the helpful reflection.
  6. Internalize: distill successful second attempts so the base model can do them without reflection later.
• Why it matters: Without ERL, the model treats reward like a vague grade; with ERL, it turns that grade into concrete, reusable advice. 🍞 Bottom Bread (Anchor): Like shooting a basketball: take a shot, notice it was short, tell yourself “use more arc,” take a second shot, then practice that better form until it’s automatic.

🍞 Top Bread (Hook): Think of a school routine: do homework (experience), write corrections (reflection), and then study the corrected version (consolidation) so test day is smoother.

🥬 Filling (The Actual Concept: Experience–Reflection–Consolidation Loop)

• What it is: A mini learning cycle where the model improves within an episode, then makes the improvements stick across episodes.
• How it works:
  1. Experience: produce Attempt 1 and get feedback.
  2. Reflection: summarize what to change.
  3. Consolidation: succeed in Attempt 2 and distill that behavior into the base model.
• Why it matters: Without consolidation, the model needs to re-reflect every time, slowing deployments and making improvements fragile. 🍞 Bottom Bread (Anchor): Solve a maze, jot “don’t enter dead ends,” solve it faster next time, then memorize that rule so you start with the better route by default.

🍞 Top Bread (Hook): After a chess loss, you might write: “I keep developing the queen too early—fix: develop minor pieces first.” That short note is powerful.

🥬 Filling (The Actual Concept: Self-Reflection)

• What it is: The model writes a short, structured guide on what to change next.
• How it works:
  1. Read the attempt and the feedback.
  2. Extract the mistake pattern.
  3. Write a specific, actionable rule.
  4. Use it to guide the second try.
• Why it matters: Without explicit reflection, the model relies on chance to stumble on fixes. 🍞 Bottom Bread (Anchor): “Box can’t be pulled—only pushed. Don’t push into corners.” That one-liner can save many Sokoban levels.

🍞 Top Bread (Hook): Imagine keeping a tiny playbook of winning tips you’ve discovered.

🥬 Filling (The Actual Concept: Memory Persistence)

• What it is: A simple cross-episode memory that stores good reflections so they can be reused.
• How it works:
  1. If Attempt 2 succeeds above a threshold, keep the reflection in memory.
  2. Next time, condition new reflections on this memory.
• Why it matters: Without memory, each episode re-learns the same lessons from scratch. 🍞 Bottom Bread (Anchor): “Don’t step on C tiles; A is you; B is the goal” becomes a starter kit for FrozenLake, helping new boards right away.

🍞 Top Bread (Hook): After you learn a better way to tie your shoes, you don’t need to think through every step—you just do it.

🥬 Filling (The Actual Concept: Internalization Mechanism / Distillation)

• What it is: Teach the base model to produce the improved second-attempt behavior directly from the original input.
• How it works:
  1. Take successful second attempts.
  2. Train the model to output those answers without the reflection context.
  3. Over time, the model bakes in the better behavior.
• Why it matters: Without internalization, reflection is always needed at runtime, adding cost and latency. 🍞 Bottom Bread (Anchor): After practicing “don’t push boxes into corners,” the agent starts Sokoban levels acting that way immediately, no extra hint needed.

Multiple analogies (same idea, different views):

• Teacher’s feedback: First draft → teacher’s margin notes → revised draft → you memorize the fix for future essays.
• Cooking: Bake batch 1 → taste and write notes (“more salt, less time”) → bake batch 2 → add the new recipe to your cookbook.
• Sports: Try play → watch replay and write a cue (“cut left sooner”) → try again → drill the cue until it’s habit.

Before vs After:

• Before (RLVR): Reward is an endpoint number; the policy must guess internal corrections via random exploration.
• After (ERL): Feedback spawns an explicit reflection that shapes a second try right away; wins are distilled so the base model retains them.

Why it works (intuition): ERL creates an intermediate teaching signal (reflection) that narrows exploration to promising fixes. Gating uses reflection only when needed (failed or suboptimal attempts), keeping training stable and on-policy enough. Memory reuses good rules; distillation removes runtime overhead. Together, this reduces the guesswork of credit assignment and accelerates learning where rewards are sparse.

Building blocks:

• First attempt and feedback (experience)
• Reflection with optional memory (reasoned correction)
• Second attempt (local improvement)
• RL updates on attempts and reflection (align with reward)
• Internalization/distillation (make it stick)
• Gating (only reflect when helpful)
• Memory persistence (carry good rules forward)

Overview: At a high level: Input task → Attempt 1 → Environment feedback → Reflection (if needed) → Attempt 2 → RL updates (on attempts and reflection) → Internalization (distillation) → Optional memory update.

Step-by-step (with what, why, and example):

1. Sample a task

• What happens: Pick x from the dataset/environment (e.g., a Sokoban board, a FrozenLake grid, or a HotpotQA question).
• Why this step exists: We need a concrete episode to learn from.
• Example: Sokoban board with A=agent, B=box, C=goal, E=wall, D=floor.

1. Attempt 1: y(1) ~ πθ(·|x)

• What happens: The model generates a first solution—an action (Up/Down/Left/Right) or an answer with tool calls (HotpotQA).
• Why this step exists: It creates the “experience” to reflect on; without it, there’s nothing to improve.
• Example (FrozenLake): The agent proposes “Right.”

1. Feedback and reward: (f(1), r(1))

• What happens: The environment returns textual feedback (e.g., “You hit a wall” or “Puzzle not solved yet”) and a scalar reward (e.g., 1 for success else 0).
• Why this step exists: It’s the only ground truth signal about how well the attempt went.
• Example (Sokoban): “The agent did not move (likely hit a wall). Reward: 0.0.”

1. Gated reflection

• What happens: If r(1) < τ (failed or suboptimal), the model writes a reflection Δ conditioned on (x, y(1), f(1), r(1), memory m). If r(1) ≥ τ, skip reflection to stay on-policy and avoid overfitting.
• Why this step exists: Reflection turns vague feedback into explicit, actionable corrections. Gating prevents destabilizing over-reliance on off-policy second attempts and stops reward hacking on already-successful cases.
• Example (Sokoban): Reflection: “Don’t push into walls. Move around to line up behind the box; only push if the space behind is free.”

1. Attempt 2: y(2) ~ πθ(·|x, Δ)

• What happens: The model tries again, guided by the reflection.
• Why this step exists: It tests whether the reflection is actually helpful; without this, reflections don’t translate into behavior.
• Example (Sokoban): Now the agent moves around the box first, then pushes toward goal.

1. Feedback and reward for Attempt 2: (f(2), r(2)); reward for reflection r̃ ← r(2)

• What happens: The environment evaluates Attempt 2; we assign the reflection the same reward as the second attempt (if it led to success, it was good advice).
• Why this step exists: It directly ties reflection quality to outcomes; without it, the model could write pretty but useless notes.
• Example (FrozenLake): “Reached the goal! Reward: 1.0.” The reflection that suggested “avoid C tiles and plan a path to B” gets r̃ = 1.0.

1. Memory update (optional but important)

• What happens: If r(2) > τ, store Δ in memory m.
• Why this step exists: Good rules become priors for future reflections; without memory, each episode re-discovers the same tips.
• Example: Add “Only push boxes when the next cell is free” to memory.

1. RL updates on attempts and reflection

• What happens: Use a policy-gradient objective (e.g., GRPO) to update πθ using the advantages from rewards for y(1), Δ, and y(2). Apply standard stabilizers (clipping, KL regularization, importance sampling) as in modern RL for LLMs.
• Why this step exists: Reinforcement aligns probabilities with better outcomes; without it, the model can’t preferentially choose better behaviors.
• Example: Increase likelihood of the improved Sokoban sequence; decrease likelihood of wall-bumping moves.

1. Internalization (distillation)

• What happens: Train the base model to output y(2) directly from x (no reflection context), but only when r(2) > 0. This selective supervised step “bakes in” successful corrections.
• Why this step exists: To remove reflection overhead at deployment and preserve gains. Without it, the model would need to reflect every time.
• Example: After distillation, on a new Sokoban board, the agent naturally circles to align pushes instead of guessing.

Worked mini-example (FrozenLake):

• x: 4×4 grid, A=you, B=goal, C=holes, D=ice.
• y(1): “Right”; f(1): “You fell into a hole.” r(1)=0.
• Reflection Δ: “Avoid C tiles; plan a safe path D→D→D to reach B; when blocked, try a different corridor.”
• y(2): “Up, Up, Right, Right” (staying on D tiles);
• f(2): “Reached the goal.” r(2)=1 → store Δ in memory; reinforce y(2) and Δ; distill so next time the model plans safe paths without needing Δ.

What breaks without each step:

• No reflection: Second tries don’t improve reliably; the model keeps guessing fixes.
• No gating: Training drifts off-policy; models can overfit instance-specific tricks, hurting generalization.
• No memory: Lessons don’t accumulate; repeated re-learning slows training.
• No internalization: Deployments are slower and brittle; improvements vanish when reflection is absent.
• No RL updates: The model won’t shift probabilities toward better strategies.

Secret sauce (why it’s clever):

• Reflection converts sparse terminal rewards into structured, immediate guidance within the same episode.
• Gating focuses compute on failures, stabilizing optimization and preserving on-policy learning.
• Memory reuses proven fixes; distillation cements them, so the base policy improves with no extra inference cost.
• Together, ERL reduces blind exploration and speeds up durable learning.

The test: The authors evaluated ERL against a strong RL baseline (RLVR) on three settings: two sparse-reward grid worlds (FrozenLake and Sokoban) and a tool-using reasoning task (HotpotQA). The main measurements were final reward (how often it succeeds), learning efficiency (how fast reward improves over time), and optimization dynamics (reward before and after reflection within a training step). Compute was matched fairly: RLVR got 10 rollouts per prompt; ERL used roughly half as many per attempt but with two attempts and a reflection, balancing total cost.

The competition: Baselines included:

• RLVR: Standard reinforcement learning with verifiable rewards (no explicit reflection or memory).
• ERL w/o Memory: ERL but without saving reflections across episodes.
• ERL w/o Reflection: Two attempts but no structured reflection (just reuse context and a generic retry instruction).

The scoreboard (with context):

• Sokoban (hard long-horizon planning): Qwen3-4B-Instruct jumps from 0.06 to 0.87 with ERL (a huge leap—like going from an F to an A). Olmo3-7B-Instruct rises from 0.04 to 0.20.
• FrozenLake (must infer rules from scratch): ERL improves rewards notably (e.g., Qwen from 0.86 to 0.94; Olmo from 0.39 to 0.66), like raising your average from a B to an A-.
• HotpotQA (tool-augmented QA with denser feedback): Gains are smaller but steady (up to +11%), like nudging a B- to a solid B+.

Learning speed: Plots of reward versus wall-clock time show ERL reaches higher rewards earlier across tasks and models. This means reflection concentrates updates on promising fixes, cutting down on wasteful trial-and-error.

Within-episode gains: Reward lines for ERL before vs after reflection show that the second attempt (post-reflection) consistently scores higher than the first attempt and also beats RLVR at the same point in training. That proves reflections provide actionable corrections immediately, not just over long horizons.

Ablations (what matters most):

• Full ERL > No-Memory > No-Reflection in most cases. Removing reflection causes the biggest drop: without explicit “what to fix,” the second try isn’t reliably better. Removing memory slows convergence because lessons don’t accumulate.
• Surprising finding: In Olmo3-7B on Sokoban, the no-memory variant slightly outperformed full ERL. Likely cause: early reflections in a complex environment were sometimes noisy; saving them could spread bad tips. This suggests memory might need quality gates or retrieval filtering in very tricky settings.

Takeaways:

• ERL is most helpful when rewards are sparse and planning is long-horizon (Sokoban, FrozenLake).
• It still helps with tool-using QA, just with smaller margins (as rewards are denser and structure clearer).
• Reflection is the star; memory and distillation make improvements durable and reusable; gating keeps training stable and honest.

Limitations:

• Reflection quality: If the model’s reflective notes are wrong or vague, they can misguide Attempt 2. In complex or stochastic tasks, early reflections may be noisy.
• Memory contamination: Saving low-quality reflections can spread bad habits. This showed up in one Sokoban setting, where no-memory slightly won. Better memory gating or retrieval (e.g., relevance filters) can help.
• Training overhead: ERL uses more steps per episode (attempt→reflection→attempt), though compute was matched in experiments. At inference, there’s no added cost thanks to internalization.
• Reward dependence: ERL still needs clear reward signals (even if sparse). If rewards are misleading or encourage shortcuts (reward hacking), reflections may learn the wrong lessons—gating helps, but reward design still matters.
• Scope: Results cover grid control and a QA setting. Very long-horizon, partially observable, or safety-critical physical tasks may need stronger memory, better credit assignment, and robust evaluation.

Required resources:

• RL training stack (e.g., rLLM), policy-gradient optimizer (GRPO), KL control, importance sampling.
• Serving infra (vLLM, FlashAttention) for efficient rollouts.
• GPUs (experiments used 8×H100), datasets/environments (procedurally generated FrozenLake/Sokoban; HotpotQA with dense retrieval via FAISS), and reward checkers.

When NOT to use:

• Tasks with very dense, low-noise feedback where classic RL already learns fast; ERL’s extra structure may add complexity without big gains.
• Settings with unreliable or hard-to-define rewards; reflections could latch onto spurious cues.
• Highly non-stationary environments where old reflections quickly go stale, unless memory is designed for rapid refresh.

Open questions:

• Smarter memory: retrieval-augmented memory with quality scores; forgetting outdated or low-confidence reflections.
• More than two attempts: adaptive budgeting—allow 0, 1, or more retries based on uncertainty or potential gain.
• On-policy distillation: reverse-KL or policy-matching approaches to improve stability and reduce off-policy drift.
• Safety: auditing reflections for harmful shortcuts; detecting and preventing reward hacking.
• Generalization: applying ERL to robotics, coding agents, web navigation, and multi-agent social settings with richer, longer horizons.

Three-sentence summary: ERL adds a reflection-and-retry step inside RL training so the model can turn sparse feedback into specific corrections, try them immediately, and then internalize what works. This transforms trial-and-error into learn-and-improve, speeding training and producing policies that keep their gains without extra runtime cost. Across hard control tasks and tool-using QA, ERL consistently improves both learning efficiency and final performance over strong RL baselines.

Main achievement: Converting delayed, scalar rewards into structured, actionable reflections inside training—and then distilling those improvements so the base policy remembers them.

Future directions: Build smarter memory with retrieval and confidence gating, explore multiple adaptive retries, use on-policy distillation for stability, and expand to longer, more complex tasks like robotics and web-scale agents. Safety tooling for reflections and rewards will help prevent shortcuts and ensure robust behavior.

Why remember this: ERL shows that giving models a chance to think about their own mistakes during training—and to keep the fixes—turns vague feedback into lasting skill. It’s a practical recipe for making agents that learn faster, explore smarter, and act better when it really counts.