MOA: Multi-Objective Alignment for Role-Playing Agents

🍞 Hook: You know how a good actor must remember their character’s backstory, use the right voice, and still follow the script during a long scene? If they only focus on one thing, like voice, they might forget the story or miss their cues.

🥬 The Concept (Role-Playing Agents): Role-playing agents are AI chatbots that pretend to be a specific character across many turns while following instructions and sounding like that character.

• How it works: They read your message, recall persona traits (like favorite phrases, knowledge limits), follow rules, and answer in the right tone.
• Why it matters: Without balancing all these parts, the agent might sound generic, break character, or ignore instructions. 🍞 Anchor: Imagine a chatbot acting as a medieval blacksmith: it should talk like one, know blacksmith facts, avoid modern slang, and still answer your multi-step requests.

🍞 Hook: Imagine studying for a test only by copying answers from a key. You might get similar questions right, but you’ll struggle with new ones.

🥬 The Concept (Supervised Fine-Tuning, SFT): SFT teaches a model by showing lots of example chats and having it imitate the given answers.

• How it works: Feed question–answer pairs, train the model to predict the answer tokens, repeat many times.
• Why it matters: Without careful variety, the model overfits “surface clues,” becomes less creative, and struggles when questions or personas shift. 🍞 Anchor: If all your practice questions use bullet points, your answers may always be in bullets—even when a character should speak in flowing sentences.

🍞 Hook: Think of a puppy learning tricks by getting treats for good behavior and no treat for mistakes.

🥬 The Concept (Reinforcement Learning, RL): RL trains a model by sampling answers and scoring them with rewards, then nudging the model toward higher-reward behaviors.

• How it works: Generate several answers, score each, compare to the group, and update the model to prefer better ones.
• Why it matters: Without good rewards and variety, RL can chase the wrong patterns (like “longer is always better”) or get stuck. 🍞 Anchor: If the judge always gives more points to longer answers, the model may ramble to get higher rewards.

The world before: Most role-playing agent training used SFT on synthetic persona dialogues. This improved basic skills but had two problems: (1) It latched onto shallow patterns (like keywords) and missed deeper persona traits. (2) It reduced output diversity—when you turned up the “creativity knob,” the model still sounded same-y, making RL exploration weak.

The problem: Role-play is multi-objective. Answers must balance multiple goals simultaneously—basic dialogue quality, persona knowledge boundaries, and style compliance. Optimizing one can hurt another. For example, cramming more facts can help knowledge scores but break the character’s tone. Standard RL methods (like GRPO) often collapse different scores into one mixture and treat all improvements as equal, which hides conflicts.

Failed attempts: Some tried keyword-based rewards and direct RL transfer from reasoning tasks. But keywords miss style and human-likeness, and single-number rewards can label conflicting outputs as equally “good,” confusing the policy. Others bumped up sampling temperature on an SFT model but saw tiny diversity gains.

The gap: We lacked a training method that (1) handles several fine-grained, possibly conflicting rubrics at once, (2) encourages diverse high-quality rollouts, and (3) prevents reward hacking from judge models.

Real stakes: Better RPAs matter in customer support, learning companions, story NPCs, and content creation. You want agents that sound consistently in-character, know when to refuse, follow nested instructions, and adapt across scenes. Without this balance, users lose trust: the “wizard” suddenly speaks like a modern blogger; the “doctor” spills info beyond their role; or the agent follows step one and forgets step two.

🍞 Hook: Imagine you’re juggling three balls—knowledge, style, and instructions. If you try to push all three higher at once without watching which one is actually moving, you’ll drop something.

🥬 The Concept (Why this paper exists): The paper proposes a multi-objective RL framework—MOA—that learns from many rubrics at once, picks the goal that’s improving fastest as the pivot, removes misleading samples, and boosts exploration with thought steps and off-policy guidance.

• How it works: Track reward trends per dimension, select a pivot goal, filter rollouts that hurt it, and combine on-policy and strong off-policy samples.
• Why it matters: It keeps the training signal clean and focused, grows diversity, and prevents the model from gaming the judge. 🍞 Anchor: If style is rising this week but persona knowledge lags, MOA focuses updates on style, ignores “style-breaking” samples, and still listens to a stronger model’s example to avoid bad habits.

🍞 Hook: You know how coaches watch which part of a player’s game (speed, stamina, accuracy) is improving fastest, then tailor practice around that area to build momentum?

🥬 The Concept (MOA in one sentence): MOA is a reinforcement-learning method that aligns role-playing agents across multiple, sometimes conflicting, rubrics by dynamically focusing on the most promising dimension and filtering out rollouts that would derail it, while boosting variety and quality with thought-augmented rollouts and off-policy guidance.

• How it works (recipe):
  1. Generate several answers per prompt, first “thinking in character,”
  2. Score each answer on multiple fine-grained rubrics (dialogue basics, persona knowledge, style),
  3. Track recent reward trends and pick a pivot dimension (the one improving beyond its trend),
  4. Drop rollouts that score poorly on the pivot but look good elsewhere (they’re misleading),
  5. Compute weighted advantages and update the model; mix in one strong off-policy answer to stabilize and prevent hacking.
• Why it matters: Without focusing and filtering, the agent can label contradictory samples as equally good, slowing or reversing progress on key goals. 🍞 Anchor: If the agent is getting better at “style” this session, but some long, fact-stuffed answers break the tone, MOA ignores those and pushes answers that keep the style consistent.

Three analogies:

• Music mixer: MOA turns up the volume on the track that is currently syncing best (pivot), mutes conflicting notes (filtering), and peeks at a pro’s recording (off-policy) to stay on beat.
• Classroom: The teacher focuses on the topic students are just starting to grasp (pivot), discards misleading study guides (filter), and shows an exemplar solution (off-policy) to clarify standards.
• Chef: Taste the dish across salty/sweet/spicy, lean into the flavor that’s balancing well (pivot), remove ingredients fighting that balance (filter), and sample a master chef’s spoon (off-policy) to avoid bad trends.

Before vs After:

• Before: Single-score RL blended goals; conflicting rollouts slipped through; SFT-based models explored poorly.
• After: MOA sees per-goal trends, concentrates on the best-improving goal, discards deceptive samples, and expands exploration with in-character thinking and a strong model’s example.

Why it works (intuition):

• Momentum matters: If one goal is rising faster than its recent trend, the gradients there are often strong and reliable; pushing along that direction speeds learning.
• Noise control: Samples that look good overall but harm the pivot goal inject noise and confuse policy updates; removing them stabilizes optimization.
• Exploration with guardrails: Thought-augmented generation increases diversity and structure; off-policy examples anchor quality and reduce judge gaming (like “longer is always better”).

Building blocks (each with a sandwich explanation):

🍞 Hook: Imagine you track your running speed each day and notice today’s speed is above your usual trend. 🥬 The Concept (Pivot Dimension Selection): Choose the goal whose current score beats its recent trend the most and give it extra weight.

• How it works: Keep a short history per rubric, fit a simple trend line, compute the “residual” (today minus trend), softmax these residuals into weights, pick the max as pivot.
• Why it matters: Focusing updates where progress is easiest compounds gains and avoids spreading effort too thin. 🍞 Anchor: If “style” jumped more than expected today, train more on style right now.

🍞 Hook: You know how a bad example can steer you off-course even if it looks shiny? 🥬 The Concept (Conflict Rollouts Elimination): Filter out answers that hurt the pivot goal even if they score well on others.

• How it works: Keep only rollouts that are not dominated when sorting by pivot score and overall weighted score (a relaxed partial order).
• Why it matters: It removes misleading positives that would nudge the model in the wrong direction for the current focus. 🍞 Anchor: Discard a verbose answer that breaks the knight’s voice (pivot=style), even if it shows lots of facts.

🍞 Hook: Before you speak in a play, you quietly remind yourself who you are and what you want. 🥬 The Concept (Thought-Augmented Rollout): The model writes a brief, role-aware plan before answering.

• How it works: Use a prompt that elicits feelings, background, goals, and plan; then produce the in-character reply.
• Why it matters: It boosts structure, persona consistency, and exploration beyond SFT habits. 🍞 Anchor: As a pirate, the model privately thinks “I’m bold and cheeky; don’t reveal modern terms; keep it swashbuckly,” then answers.

🍞 Hook: You can learn a lot by watching a champion’s move even if you didn’t play that turn. 🥬 The Concept (Off-Policy Guidance): Mix one answer from a stronger model into the batch for scoring and comparison.

• How it works: Add a high-quality sample to the group so advantage estimation sees a broader quality range.
• Why it matters: It reduces reward hacking, adds diversity, and stabilizes updates. 🍞 Anchor: Include one GPT-4o answer among your samples to calibrate what “great” looks like.

High-level flow: Input prompt (with persona and multi-turn context) → Generate multiple rollouts with in-character thinking (on-policy) and include one strong off-policy sample → Score each rollout on several fine-grained rubrics (LLM-as-judge) → Track recent reward trends, compute residuals, select pivot dimension and weights → Remove rollouts that conflict with the pivot → Compute weighted advantages and update the policy (GRPO-style) → Repeat.

Step-by-step details (with sandwiches for key concepts):

1. Prepare diverse rollouts

• What happens: For each input, sample a group of answers. First, the policy model generates a private “thought” in role (feelings, background, goals, plan), then the final in-character reply. Also add one off-policy sample from a stronger model to the group.
• Why this exists: SFT models often produce low-diversity samples; adding thought increases structure and variety, while off-policy gives a strong anchor and prevents drifting into easy-to-game patterns.
• Example data: For a detective persona asked “Who might have moved the vase?”, the thought notes tone (calm, observant), avoids revealing modern terms, then the answer reflects that style.

🍞 Hook: Before jumping into a maze, you sketch a path in your head. 🥬 The Concept (Thought-Augmented Rollout): Briefly plan in character, then respond.

• How it works: Use a prompt template like: “I am {persona}… I’m thinking… So, I'm planning…” then produce the answer.
• Why it matters: Encourages persona-consistent structure and rich exploration. 🍞 Anchor: A knight silently plans “speak formally, avoid slang, refuse out-of-scope knowledge,” then replies nobly.

🍞 Hook: Watching a pro’s move can prevent you from practicing the wrong thing. 🥬 The Concept (Off-Policy Guidance): Include one high-quality answer from a stronger model in each group.

• How it works: Combine 15 on-policy samples with 1 off-policy sample; use them all for advantage estimation.
• Why it matters: Increases diversity and reduces reward hacking (e.g., overly long answers winning unjustly). 🍞 Anchor: The group includes a top-tier example that calibrates what the judge rewards for “style” without rambling.

1. Score with fine-grained rubrics

• What happens: Each rollout is scored along multiple dimensions (Basic Dialogue, Persona Knowledge, Style Compliance) using a strong LLM-as-judge with carefully designed prompts.
• Why this exists: Keywords alone can’t capture tone or persona boundaries; rubrics measure nuances needed in role-play.
• Example data: A concise, accurate, first-person reply scores high on Basic Dialogue; an answer that refuses beyond-knowledge questions scores high on Persona Knowledge; consistent tone, jargon, and phrasing score high on Style.

🍞 Hook: Like grading a speech on clarity, accuracy, and style—separately. 🥬 The Concept (LLM-as-Judge Rubrics): A strong model assigns separate scores per rubric using detailed instructions.

• How it works: Provide conversation, persona, response, and rubric-specific criteria; get a 0–1 score per dimension.
• Why it matters: Separating goals exposes conflicts and gives clearer training signals. 🍞 Anchor: The judge rates “knightliness” (style), “answer correctness,” and “persona boundaries” independently.

1. Track trends, pick a pivot, set weights

• What happens: Keep a short buffer of recent average scores per dimension. Fit a simple trend line per dimension to get “expected today.” Compute the residual (actual minus trend) for each. Convert residuals into weights via softmax; choose the largest as the pivot.
• Why this exists: Emphasizes goals currently gaining momentum; such goals often have strong gradients—updates there pay off fastest.
• Example: If Style’s residual is +0.07 while Knowledge is +0.01 and Basic is −0.02, Style becomes the pivot and gets the highest weight.

🍞 Hook: If your math skill just jumped this week, practicing more math today compounds progress. 🥬 The Concept (Pivot Dimension Selection): Weight dimensions by how much they beat their short-term trend; pick the biggest as pivot.

• How it works: Residuals → softmax weights; pivot = max weight.
• Why it matters: Focused training avoids diluting effort across conflicting goals. 🍞 Anchor: Style improving fastest? Prioritize style this batch.

1. Remove conflicting rollouts

• What happens: Sort rollouts along two axes: pivot score and overall weighted score. Keep a largest subset where each kept sample isn’t dominated by another with both higher pivot and higher weighted score (a relaxed partial order). Zero out the advantage of excluded rollouts.
• Why this exists: Prevents misleading positives that look good globally but sabotage the current pivot goal, stabilizing updates.
• Example: A long answer with high knowledge but low style is dropped during a style-focused step.

🍞 Hook: When practicing free throws, you ignore shots that looked flashy but broke your form. 🥬 The Concept (Conflict Rollouts Elimination): Keep only samples that don’t underperform on both pivot and weighted total.

• How it works: Find the largest non-conflicting subset using a simple ordering procedure.
• Why it matters: Reduces noise so the policy clearly learns what helps the pivot. 🍞 Anchor: During a style-focused step, remove any answer that flattens the character’s voice.

1. Compute advantages and update the model (GRPO-style)

• What happens: Combine the per-dimension scores using the current weights; standardize across the group to form advantages; set advantage to zero for filtered-out rollouts; optimize a GRPO-like clipped objective.
• Why this exists: Group-relative normalization stabilizes updates; clipping avoids too-large steps; zeroing excluded samples enforces the filter.
• Example: Three rollouts have weighted scores 0.75, 0.55, 0.30; they get high, medium, low advantages, except the filtered one gets zero.

🍞 Hook: You grade on a curve within the class so improvements are relative to peers. 🥬 The Concept (GRPO-style Update): Use group-relative, clipped advantages for stable learning.

• How it works: Normalize within the sampled group; clip update ratios; apply weights from the pivot step.
• Why it matters: Prevents unstable jumps and keeps learning focused. 🍞 Anchor: The best in-group style answer gets a strong positive push; filtered samples don’t affect the update.

Secret sauce: The combo of trend-based pivoting (focus where progress compounds), conflict filtering (remove misleading positives), and diversity boosters (thought rollout + off-policy guidance) produces faster, steadier gains than uniform multi-goal mixing. It’s like practicing the part of your game that’s heating up, ignoring bad examples that look shiny, and learning from a champion’s clip each session.

The test: The authors evaluated MOA on two challenging public role-play benchmarks—PersonaGym and RoleMRC—because they cover diverse personas, styles, scenes, and multi-turn, nested instruction-following. They used LLM-as-judge scoring (GPT-4o) to fairly compare models across fine-grained rubrics.

The competition: MOA was compared to (1) strong closed-source models (GPT-4o, Claude-3.7), (2) SFT-only baselines trained on large synthetic persona datasets, and (3) RL baselines like GRPO and RLOO. They tested across multiple base models (Qwen3-1.7B/8B, Llama-3.1-8B) to show generality.

Scoreboard with context:

• PersonaGym: MOA with an 8B base achieved an average around 4.75 across five 1–5 scales (EA, LH, PC, TC, AJ), which is basically “A-level” when top closed models are in the high 4s. For example, MOA (GRPO) posted EA≈4.84, LH≈4.81, PC≈4.40, TC≈4.79, AJ≈4.92. That means it’s nearly matching GPT-4o and even outperforming Claude-3.7 on Action Justification.
• RoleMRC: MOA beat GPT-4o by about 21% on average across binary accuracy metrics (KR, SC, NI, MT, IP). That’s like jumping from a class average B- to a solid A when others stay around B.
• Stability and scaling: MOA consistently outperformed vanilla GRPO and RLOO on the same bases, and the gains held across smaller (1.7B) and instruction-tuned (Llama-3.1-8B-Instruct) models.

Surprising findings:

• Plain GRPO starting from SFT struggled, even with high sampling temperature—evidence that SFT’s low diversity can choke exploration.
• Adding in-character thinking (MOA-t) already helped; adding multi-objective pivot+filter (full MOA) helped even more.
• Off-policy guidance boosted early quality (MOA-o started higher) but without thinking, improvement slowed later; the full combo (thinking + pivot/filter + off-policy) grew fastest overall.

Meaning of the numbers:

• PersonaGym’s near-5 scores signal strong persona faithfulness, safe behavior, and coherent justifications.
• RoleMRC’s big lift, especially in Multi-turn and Instruction Priority, shows MOA’s advantage at complex, nested directions—exactly the sort of juggling role-play needs.

Takeaway: MOA doesn’t need hand-crafted data tricks to get multi-dimensional gains. It uses the rubrics themselves as learning targets, focusing updates where momentum is strongest, and keeps training honest via thought and off-policy guidance.

Limitations:

• LLM-as-judge cost: Scoring multiple dimensions with a strong judge (e.g., GPT-4o) is computationally and monetarily expensive compared to simple, rule-based rewards.
• Self-scoring not explored: Letting the model partially self-judge could reduce costs but risks bias; the paper leaves this for future work.
• Domain generality: MOA shines in role-play, but hasn’t been validated on math/coding; conflicts and rubrics differ there.
• Prompt sensitivity: Thought prompts require careful design; poor templates may lower early quality before benefits emerge.
• Reward hacking risk persists: Off-policy guidance reduces hacking, but any judge-based system remains partially gameable if prompts drift.

Required resources:

• Multi-GPU training (e.g., 8×A100-80GB) to handle group sampling (G≈16), long completions (~1200 tokens), and frequent judge calls.
• Access to a strong judge model (e.g., GPT-4o) during training and evaluation or a high-quality open alternative.

When not to use:

• If you need ultra-low-cost training with simple, verifiable rewards (like unit tests in code), MOA’s judge-heavy loop may be overkill.
• If persona/style don’t matter and you only need factual QA, simpler RL or SFT may suffice.
• If you can’t allow thought traces in generation (even privately during training), you’ll lose a key diversity lever.

Open questions:

• Can we distill the judge into a lighter, domain-adaptive scorer without losing nuance?
• How well does pivot+filter transfer to other multi-objective domains (e.g., reasoning accuracy vs brevity vs calibration)?
• Can we automatically learn the thought template style per persona to further enhance in-character planning?
• How robust is MOA if rubrics disagree systematically (e.g., style judges subtly prefer longer text)?
• Can partial on-device judging and caching reduce cost while keeping training signal fresh?

Three-sentence summary: MOA is a reinforcement-learning framework for role-playing agents that optimizes several fine-grained, sometimes conflicting rubrics at once. It dynamically focuses on the fastest-improving goal (pivot), filters out conflicting rollouts, and boosts diversity and stability with thought-augmented rollouts and off-policy guidance. This delivers state-of-the-art or near-SOTA performance on PersonaGym and a roughly 21% average gain over GPT-4o on RoleMRC using only 8B-scale models.

Main achievement: Showing that dynamic multi-objective alignment—with pivot selection and conflict filtering—combined with structured thinking and off-policy examples, can train RPAs that balance persona knowledge, style, and complex multi-turn instruction-following without handcrafted data tricks.

Future directions: Build lighter, reliable judges (or self-judging with safeguards), extend MOA to domains like reasoning and coding where goals can also conflict (accuracy vs brevity vs safety), and auto-tune thought prompts per persona. Explore online curriculum designs that adapt pivot temperature, history length, and filter strictness as training evolves.

Why remember this: MOA reframes role-play training from “one score fits all” to “many goals, learned smartly,” proving that focus (pivot), clarity (filter), and variety (thought + off-policy) can lift smaller models to challenge much larger, closed systems across rich conversational skills.