EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience

🍞 Hook: Imagine you’re learning to use a new computer game. If you only watch old recordings of other people playing, you’ll pick up some moves, but you won’t feel the timing, the mistakes, or the surprises that happen live.

🥬 The Concept (Reinforcement Learning): Reinforcement Learning is a way for an AI to learn by trying things and getting feedback about how well it did.

• How it works:
  1. The AI does an action.
  2. The world changes and gives a score (good/bad).
  3. The AI repeats, aiming for higher scores.
• Why it matters: Without it, the AI only copies and never truly practices. 🍞 Anchor: A typing tutor that praises you when you type a word correctly and nudges you when you miss a letter is using the idea behind Reinforcement Learning.

🍞 Hook: You know how teachers make practice worksheets so you can learn even when there’s no test that day?

🥬 The Concept (Data Generation): Data Generation means creating new practice problems for the AI to learn from.

• How it works:
  1. Decide a topic.
  2. Create tasks that match the topic.
  3. Feed them to the AI to practice.
• Why it matters: Without fresh practice, the AI gets stuck on what it already saw. 🍞 Anchor: Making new math problems with different numbers helps you learn the pattern, not just memorize answers.

🍞 Hook: When you bake cookies and one tray burns, you don’t just sigh—you ask why.

🥬 The Concept (Error Analysis): Error Analysis is studying what went wrong so we can fix it next time.

• How it works: Find the first wrong step, figure out the cause, and plan a better move.
• Why it matters: Without it, mistakes repeat and progress stalls. 🍞 Anchor: If your cookies burned because the oven was too hot, you lower the temperature next time.

🍞 Hook: Imagine a treasure hunt with a referee who checks if you actually found the treasure and not just a shiny rock.

🥬 The Concept (Validation Mechanisms): Validation Mechanisms are strict checks that tell us if the AI truly finished the task.

• How it works: Write clear rules, run a checker, and accept only verified wins.
• Why it matters: Without validation, the AI can ‘look’ correct but actually be wrong (reward hacking). 🍞 Anchor: A spelling bee judge who only counts the word correct if all letters are right is a validator.

🍞 Hook: Think of a school with many classrooms where students practice at the same time instead of taking turns in one room.

🥬 The Concept (Asynchronous Rollouts): Asynchronous Rollouts let many practice runs happen at once without waiting in line.

• How it works: Start many sessions, collect results independently, and keep the trainer busy with new data.
• Why it matters: Without this, learning is slow and wastes compute. 🍞 Anchor: Many soccer teams scrimmaging on different fields speeds up how fast the whole club improves.

🍞 Hook: When following a recipe, tiny timing differences matter—like when to take cookies out before they overbake.

🥬 The Concept (Task Dynamics): Task Dynamics are the little cause-and-effect rules that decide what happens after each click or keypress.

• How it works: The environment changes with each action; timing and order matter.
• Why it matters: Ignoring dynamics makes long, multi-step tasks crumble. 🍞 Anchor: Clicking ‘Save’ before the file exists won’t work; the order of steps changes the outcome.

🍞 Hook: A flight simulator lets pilots practice safely before flying real passengers.

🥬 The Concept (Simulation Environment): A Simulation Environment is a safe, pretend world where the AI can practice computer use.

• How it works: It shows screens, accepts actions, and updates like a real computer.
• Why it matters: Practicing in the real world can be slow, risky, or inconsistent. 🍞 Anchor: A virtual desktop that opens apps, loads files, and reacts to clicks is a simulation environment.

🍞 Hook: Solving a mystery when you can only see part of the scene takes careful thinking.

🥬 The Concept (POMDP): A POMDP is a math way to say, ‘You don’t see everything, but you still must choose the next best move.’

• How it works: Keep a history of what you saw and did; guess the hidden parts; pick the next action.
• Why it matters: On a computer, the AI only sees the screen, not every hidden system setting. 🍞 Anchor: The agent views screenshots (partial info) and must still choose sensible clicks and keys.

The World Before: For years, computer-using AIs learned by imitating ‘static’ recordings of humans clicking and typing. That helped them copy common moves but failed on long, tricky chores like ‘find a PDF, compare numbers in Excel, email a report, and upload a chart.’ Why? Because long tasks depend on Task Dynamics, timing, and feedback from the environment—things that don’t show up in still, fixed data.

The Problem: Scaling with more static data was giving smaller and smaller gains. Agents needed to practice live, see results, and correct themselves—like a student who needs real homework and quizzes, not just answer keys.

Failed Attempts:

• More imitation: memorized patterns but didn’t generalize to new screens.
• Fuzzy rewards: language-only checks could be tricked (looked right, wasn’t right).
• Small simulators: too few runs, too slow, too brittle.

The Gap: We were missing a way to create tons of trustworthy, checkable practice and a training loop that turns both successes and failures into learning—at scale.

Real Stakes: Better computer-using AIs can help everyone: office workers automating reports, teachers preparing lesson slides, doctors organizing patient forms, and people with disabilities interacting more easily with complex software. To reach that, we need fast practice, hard checks, and steady improvement—exactly what EvoCUA builds.

🍞 Hook: Imagine a music school that can compose new pieces, instantly hire judges to check if you played them right, and schedule thousands of practice rooms so students improve every day.

🥬 The Concept (Verifiable Synthesis Engine): The Verifiable Synthesis Engine creates new computer tasks and builds tiny programs (validators) that can automatically check if the end result is correct.

• How it works:
  1. Define realistic tasks from basic skills (like ‘sort data’ or ‘format text’).
  2. Generate the task and a validator that runs in a sandbox.
  3. Only keep tasks whose validators work reliably.
• Why it matters: Without verifiable tasks, agents can ‘fake’ success or learn from noisy signals. 🍞 Anchor: ‘Make a chart in Excel’ plus ‘a checker that opens the finished file and confirms the chart exists with the right labels’ is a verifiable pair.

🍞 Hook: Think of a gym that never closes, with endless treadmills ready, so no athlete has to wait.

🥬 The Concept (Scalable Interaction Infrastructure): This is the giant system that runs tens of thousands of practice computers in parallel and feeds back results to the trainer.

• How it works: Asynchronous rollouts, fast schedulers, and stable virtual desktops collect experience nonstop.
• Why it matters: Without scale, improvement crawls; with scale, you can learn from a river of fresh attempts. 🍞 Anchor: Hundreds of virtual desktops spin up in seconds, each trying to complete tasks like ‘download and rename a file’ and reporting success or failure.

🍞 Hook: When you learn piano, you keep the parts you can already play and focus practice on the tricky bars you keep missing.

🥬 The Concept (Evolving Learning Paradigm): The Evolving Learning Paradigm is a loop that learns from experience—boosting the good patterns and directly fixing the bad ones.

• How it works:
  1. Cold start: teach basic moves and reasoning format.
  2. Rejection Sampling Fine-Tuning: keep only clean, successful runs to sharpen good habits.
  3. Reinforcement Learning with step-level preferences: find the first mistake in failed runs and teach the fix right there.
• Why it matters: Without both sides (wins and mistakes), agents plateau or overfit. 🍞 Anchor: The agent saves ‘what worked’ as recipes and rewrites the ‘tricky steps’ after it finds exactly where it went off-track.

Three analogies for the core idea:

1. School analogy: The system writes new quizzes (synthesis), hires fair graders (validators), runs many classrooms at once (infrastructure), and updates lesson plans based on class results (evolving learning).
2. Sports analogy: It schedules many scrimmages (rollouts), keeps highlight reels of great plays (successful trajectories), studies turnovers to fix them (step-level error correction), and repeats.
3. Cooking analogy: It invents new dishes (tasks), uses a taste tester with strict recipes (validators), runs many ovens (infrastructure), and refines the cookbook by keeping successes and correcting failed steps (evolving learning).

Before vs After:

• Before: Agents copied and hoped. Rewards were fuzzy. Practice was small and slow.
• After: Agents practice at scale with verifiable checks, learn faster from both wins and failures, and grow reliably across app types.

Why it works (intuition):

• Verifiable targets remove guesswork, so signals are clean.
• Massive parallel practice fills the experience pool quickly, so the agent sees edge cases.
• Step-level correction attacks the root cause at the first wrong turn, preventing cascade failures later in long tasks.

Building Blocks (with mini ‘sandwich’ intros):

• 🍞 Hook: You know how a teacher shows both the question and the answer key? 🥬 The Concept (Validator): A small program that checks if the final state meets the goal; Why it matters: it turns ‘looks right’ into ‘is right.’ 🍞 Anchor: An auto-checker opens the saved Excel and confirms the totals match.
• 🍞 Hook: Photo albums grow one picture at a time. 🥬 The Concept (Experience Pool): A big bucket of recent practice runs (good and bad) to train from; Why it matters: keeps learning current and on-policy. 🍞 Anchor: Today’s 10,000 rollouts are sampled for tonight’s training.
• 🍞 Hook: First-day lessons set the tone. 🥬 The Concept (Cold Start): A small, diverse set teaches actions and reasoning format; Why it matters: stabilizes output and reduces confusion later. 🍞 Anchor: The agent learns ‘how to think out loud’ and how to click, type, and wait correctly.
• 🍞 Hook: Coaches keep best drills, toss the sloppy ones. 🥬 The Concept (Rejection Sampling Fine-Tuning): Keep only high-quality successful steps; Why it matters: sharpens good habits and removes noisy moves. 🍞 Anchor: From five tries, keep the clean win with minimal extra clicks.
• 🍞 Hook: Fix the first wrong fork on a hike, not after you’re lost. 🥬 The Concept (Direct Preference Optimization at the step level): Prefer the corrected step over the mistaken step right where they diverged; Why it matters: prevents long chains of errors. 🍞 Anchor: At the first mis-click, teach ‘open File menu’ instead of ‘random click.’
• 🍞 Hook: Sometimes you need to pause and think. 🥬 The Concept (Reflection at the next step): Train the agent to stop, check the screen, and plan recovery after an error; Why it matters: boosts robustness. 🍞 Anchor: After a failed paste, the agent re-reads the menu and chooses the right command.

Put together, EvoCUA’s aha is simple: If you can endlessly create fair, checkable practice and fix mistakes exactly where they start, you can steadily evolve reliable computer-using agents.

At a high level: Instruction → Verifiable Synthesis Engine (task + validator) → Scalable Sandbox Rollouts → Experience Pool → Evolving Learning (Cold Start → Rejection Sampling Fine-Tuning → Step-Level Preference RL) → Updated Policy → Repeat.

Step 1. Verifiable Synthesis Engine

• What happens: The system constructs realistic tasks from basic app skills, injects real resources (docs, images), and co-generates a validator that can programmatically confirm success.
• Why it exists: Without strict validators, agents can ‘pass’ with shortcuts or surface-only matches.
• Example data: Task: ‘Create a bar chart of monthly sales in Excel and save as Sales_Chart.xlsx.’ Validator: a script opens the file, confirms a bar chart exists with correct data range and title.

🍞 Hook: You know how a puzzle booklet includes both easy and hard puzzles? 🥬 The Concept (Generation-as-Validation): Create tasks and validators together so every task is checkable by design; Why it matters: avoids vague, uncheckable goals. 🍞 Anchor: A generated ‘rename file’ task ships with a checker that confirms the exact filename and location.

Step 2. Scalable Interaction Infrastructure

• What happens: Thousands of virtual desktops (QEMU-KVM inside Docker) run in parallel; an async gateway and scheduler keep sessions flowing; rendering, input, and fonts are calibrated for determinism.
• Why it exists: To turn compute into massive, clean experience quickly and reliably.
• Example data: 10,000 sessions attempt mixed tasks (web search, PDF edit, spreadsheet ops). Logs store screenshots, thoughts, and actions.

🍞 Hook: A busy airport needs many gates and a fast tower. 🥬 The Concept (Asynchronous Rollouts): Many practice flights (sessions) take off and land independently; Why it matters: removes waiting lines and keeps GPUs/CPUs busy. 🍞 Anchor: While one task loads a web page, others continue editing slides.

Step 3. Experience Pool

• What happens: Recent trajectories (successes and failures) plus their tasks and validators are stored for training.
• Why it exists: Keeps training close to the current policy (on-policy), reducing drift.
• Example data: For a ‘merge cells’ task, the pool holds several attempts—one clean success, two near misses, one clear failure.

🍞 Hook: Today’s practice shapes tomorrow’s lesson. 🥬 The Concept (On-Policy Data): Training on what the current model actually produces; Why it matters: prevents learning from outdated habits. 🍞 Anchor: If the model now prefers a new shortcut, the pool contains those attempts.

Step 4. Cold Start (small, diverse SFT)

• What happens: Teach the model the Action Space and the Thought Space so it can act and ‘think out loud’ consistently.
• Why it exists: A stable format for reasoning and complete actions prevents confusion in long tasks.
• Example data: The model learns actions like key_down/Key_up for Shift+Click, scroll, wait, terminate-with-evidence; and thought schema: clarify goal, observe screen, self-check, reflect, then terminate.

🍞 Hook: Learning instrument basics before symphonies. 🥬 The Concept (Action Space): The allowed ‘moves’ like clicks, keys, drag, scroll, wait, and terminate; Why it matters: Missing moves (like triple-click or Shift-hold) make some tasks unsolvable. 🍞 Anchor: Selecting a whole word fast requires triple-click; selecting a range requires Shift+Click. 🍞 Hook: Saying your plan out loud helps catch mistakes. 🥬 The Concept (Thought Space): A structured inner monologue: clarify, observe, check, reflect, then decide; Why it matters: Grounds actions in what’s actually on-screen. 🍞 Anchor: ‘I see the File menu; I’ll click it to export as PDF.’

Step 5. Rejection Sampling Fine-Tuning (RFT)

• What happens: Generate multiple attempts per task; keep only the clean, successful trajectories and mask redundant steps.
• Why it exists: Successes prove capability but often include extra clicks; filtering keeps only the crisp pattern.
• Example data: From five tries to ‘insert chart,’ keep the 12-step clean one, drop the 18-step noisy ones; mask out unnecessary double-opens.

🍞 Hook: Coaches keep the best drill form. 🥬 The Concept (Dynamic Compute Budgeting): Spend more tries on borderline-hard tasks, fewer on easy ones; Why it matters: Maximizes learning per compute. 🍞 Anchor: If success rate hits the threshold at 4 tries, don’t waste 12.

Step 6. Step-Level Preference RL (DPO and Reflection)

• What happens: For a failed run, find the Critical Forking Point—the first step that diverged from a success under similar screen state—then prefer the corrected thought+action. Also train the next step to pause, reflect, and recover.
• Why it exists: Long tasks collapse from one early mistake. Fixing that spot prevents cascades.
• Example data: At step 7, instead of clicking an icon-like ad, pick the real ‘Download’ button; at step 8, reflect and verify the proper file saved.

🍞 Hook: Fix the first wrong turn on a maze map. 🥬 The Concept (Critical Forking Point): The earliest step where the bad path split from a good one; Why it matters: Early fixes save the whole plan. 🍞 Anchor: Choosing the wrong menu branch at step 3 ruins steps 4–20; fix step 3. 🍞 Hook: Voting for the better move. 🥬 The Concept (Direct Preference Optimization): Train the model to prefer the corrected step over the wrong step; Why it matters: Teaches clear, local fixes. 🍞 Anchor: Prefer ‘Open File’ over ‘Random click at (120,400).’ 🍞 Hook: Stop, look, and listen. 🥬 The Concept (Reflection Step): After an error, pause to reassess the screen and plan recovery; Why it matters: Builds robustness to surprises. 🍞 Anchor: If paste failed, re-check selection and try the right shortcut.

Step 7. Iterate

• What happens: Updated policy drives new rollouts; the loop continues, steadily expanding skills and stability.
• Why it exists: Software and tasks vary; iteration chases the moving frontier.
• Example: New tasks appear (PDF tables, web pop-ups), and the loop learns them.

Secret Sauce

• Generation-as-Validation removes reward ambiguity.
• Massive asynchrony turns compute into rich, diverse practice quickly.
• Step-level preferences attack errors at their roots.
• Deterministic, high-fidelity sandboxes reduce random glitches so learning signals stay clear.

🍞 Hook: Think of a school contest where every team solves real computer chores, and strict judges check the final files—not just how confident the team sounds.

🥬 The Concept (OSWorld Benchmark): OSWorld is a test where agents complete real desktop tasks in a controlled environment, and success is judged by verifiable checks.

• How it works: Present tasks, limit steps, run the agent, and use validators to confirm completion.
• Why it matters: It measures true end-to-end computer use, not just talk. 🍞 Anchor: ‘Find data, make a chart, save the file’ only counts if the saved file truly contains the correct chart.

The Test: EvoCUA is evaluated on OSWorld with a strict maximum of 50 steps per task. This emphasizes precise execution, not just endless retries. The main metric is Success Rate (Pass@1): did the agent solve the task correctly on its first attempt?

The Competition: EvoCUA is compared to strong open-weight and closed-weight agents, including OpenCUA-72B (open), UI-TARS-2 (closed), and frontier general models like Claude Sonnet 4.5.

The Scoreboard (with context):

• EvoCUA-32B: 56.7% Pass@1 at 50 steps.
  • This beats the prior open-source best, OpenCUA-72B (45.0%), by +11.7 percentage points—a jump like moving from a B- to an A-.
  • It also tops a strong closed-weight system, UI-TARS-2 (53.1%), by +3.6 points.
  • Compared to its own base model, Qwen3-VL-32B-Thinking (41.0% at 100 steps), EvoCUA adds +15.7 points with half the step budget, signaling sharper, less wasteful execution.
• EvoCUA-8B: 46.1% at 50 steps, beating Step-GUI-8B (40.2%), showing that the evolving approach helps even smaller models punch above their weight.
• Top closed models still lead at longer step budgets, but the gap narrows: EvoCUA-32B trails Claude-4.5-Sonnet (58.1% at 50 steps) by only 1.4 points.

Surprising Findings:

• Efficiency matters: Many baselines need up to 100 steps to peak; EvoCUA hits strong results at 50, implying better decision quality and fewer extra clicks.
• Smaller can beat bigger: EvoCUA-8B surpasses a 72B specialized model, highlighting that training recipe and verifiable experience can outweigh raw parameter count.
• Pass@k nuance: Even with deterministic sampling, GUI randomness (latency, rendering) changes outcomes, so higher k measures both model diversity and environment robustness—not just ‘trying more answers.’
• General capability trade-off: On some general vision-language tests, the EvoCUA-32B variant trained from a ‘thinking’ base saw declines due to a mismatch with non-thinking general data used in post-training. Aligning data style (thinking vs non-thinking) is important.

🍞 Hook: You know how practicing the same drill many times makes you faster and cleaner?

🥬 The Concept (Experience Scaling): As EvoCUA’s training adds more high-quality experiences, performance climbs.

• How it works: Round-by-round RFT scales from 20k to 1M samples, stacking gains.
• Why it matters: Shows the approach scales with more verifiable practice. 🍞 Anchor: Round 1 (+2.61), Round 2 (+6.79), Round 3 (+8.12) percentage-point gains over baseline for OpenCUA-72B with multi-round RFT.

Ablations (what made the difference):

• Unified Action Space: +4.84 points—having the right moves (like Shift+Click, triple-click) enables tasks that were otherwise impossible.
• Cold Start: +2.62—just enough examples to stabilize reasoning and formatting.
• RFT: +3.13—keeping only the clean successes makes patterns crisp.
• Step-Level DPO and iteration: +3.21 then +1.90—fixing first mistakes and repeating the loop steadily extends capability.

Takeaway: Verifiable tasks, massive asynchronous practice, and step-level corrections work together. The result is fewer wasted steps, stronger first-try success, and consistent gains across model sizes.

Limitations:

• Remaining gap to top closed systems and human-level reliability, especially in messy, long-tail app behaviors (pop-ups, network delays, unusual file formats).
• Some general benchmarks dipped when the post-training data style didn’t match the base model’s ‘thinking’ behavior; data alignment matters.
• Environment stochasticity (fonts, rendering, latency) still causes variance; even perfect logic can stumble if the screen shifts slightly.
• High compute and infrastructure needs: thousands of sandboxes, validators, and logging at scale are non-trivial.

Required Resources:

• A robust sandbox cluster (QEMU-KVM in Docker), async gateways, and fast schedulers.
• Storage and logging for screenshots, thoughts, and actions.
• Validator libraries and a synthesis engine to produce task+checker pairs.
• GPUs/accelerators for continuous fine-tuning and large-scale rollouts.

When NOT to Use:

• If you cannot run verifiable validators (e.g., tasks without a clear, checkable final state), you lose the clean signal this method relies on.
• If you can’t operate many parallel sandboxes, progress will be slow and the evolving loop may underperform simpler imitation.
• If your app domain disallows automation (security policies, legal constraints), you should not deploy such agents.

Open Questions:

• How far can step-level preference learning go before we need full online RL with advanced credit assignment (like STEPO) at scale?
• What are the best recipes for data style alignment (thinking vs non-thinking) to preserve general vision-language ability while boosting GUI skills?
• How to model and train for environmental uncertainty directly—so agents learn to expect and overcome rendering shifts and latency spikes?
• Can we auto-discover and add missing high-value actions (like new shortcuts) as tasks evolve, without manual design?
• What is the best way to combine human-in-the-loop feedback with verifiable validators to catch rare evaluator blind spots?

Three-Sentence Summary: EvoCUA turns compute into reliable computer-using skill by generating verifiable tasks, practicing them at massive scale inside stable sandboxes, and evolving the policy with a loop that keeps the best patterns and fixes first mistakes. This approach reaches 56.7% success on OSWorld with 50 steps, surpassing the best open-source baseline and strong closed alternatives. The evidence shows a scalable, general path forward: verifiable synthesis + asynchronous rollouts + step-level preference learning.

Main Achievement: Unifying a Verifiable Synthesis Engine, Scalable Interaction Infrastructure, and an Evolving Learning Paradigm that learns from both wins and failures—at the exact step where things go wrong—establishes a new open-source state of the art for native computer use.

Future Directions: Scale up online RL (e.g., STEPO) to further reduce extra steps and increase robustness; better align general ‘thinking’ datasets to preserve broad VLM skills; model environment uncertainty explicitly; and expand action spaces and validators to cover more real-world edge cases.

Why Remember This: EvoCUA shows that with fair, checkable practice and the right kind of step-level corrections, agents can steadily master real computer use—not just talk about it—opening the door to trustworthy digital helpers across everyday apps.