STEP3-VL-10B Technical Report

🍞 Hook: You know how a great student isn’t always the tallest kid in class? Sometimes the most thoughtful kid wins the science fair by being smart about their plan, not by being the biggest.

🥬 The Concept: Multimodal Foundation Model

• What it is: A multimodal foundation model is an AI that understands and combines different kinds of information, like pictures and words, to answer questions or solve problems.
• How it works:
  1. Look at the image to spot important details, 2) Read the text to catch the question, 3) Connect what it sees and reads, 4) Produce a helpful answer.
• Why it matters: Without this, the AI might either see but not understand the words, or read but miss what’s in the picture—so it would get confused on real tasks like homework images or charts. 🍞 Anchor: When you show the model a photo of a worksheet and ask, “What is the area of the triangle on the page?”, it finds the triangle, reads its measurements, and gives the right number.

The World Before: Big-name multimodal models kept getting larger to get smarter. Giant systems (think: hundreds of billions of parameters) showed great skills in reading charts, recognizing small text in images, solving math tied to pictures, and following human instructions. But they needed huge computers, lots of memory, and plenty of electricity, which made them hard to deploy in real life.

The Problem: Smaller models under 10B parameters were often fast and cheap, but were called “efficient yet limited.” They could miss tiny letters in a photo, mix up object locations, or get stuck on tricky math that used both words and pictures. People wanted a compact model that could still do tough perceptual and reasoning tasks without a giant compute bill.

🍞 Hook: Imagine training for a decathlon by practicing running, jumping, and throwing together instead of training them in isolated silos. Your body learns how the parts work together.

🥬 The Concept: Unified Pre-training

• What it is: Teach vision and language together from the start on a huge, high-quality mix of images and text so they form a shared understanding.
• How it works:
  1. Feed image–text data at scale, 2) Let both vision and language parts update together (fully unfrozen), 3) Include many task types (OCR, grounding, counting, VQA, math), 4) Encourage tight coordination between seeing and saying.
• Why it matters: If you only teach them separately, they never fully sync. Later steps have to “glue” them together and that glue can be weak. 🍞 Anchor: The model learns to read menu photos and answer questions like “How much is the burger?” without bolting reading and reasoning together as an afterthought.

Failed Attempts: Common shortcuts—like freezing the vision backbone, pre-training vision and language separately, or doing only small supervised finetuning—often led to brittle perception and weak cross-modal reasoning. Another miss: focusing solely on sequential “chain-of-thought” without considering that perception may benefit from parallel exploration.

🍞 Hook: Picture a coach who first teaches you core skills, then practices with feedback, and finally runs drills that reward good choices.

🥬 The Concept: Post-training (SFT + RL)

• What it is: After pre-training, the model is refined with teacher examples (Supervised Finetuning) and then learns by trial-and-reward (Reinforcement Learning).
• How it works:
  1. Show correct solutions to shape behavior (SFT), 2) Let the model try and score it with rewards (RL), 3) Repeat to improve precision, reasoning, and alignment.
• Why it matters: Without this, the model stays generic, sometimes right but not reliably careful, helpful, or safe. 🍞 Anchor: The model first studies worked math solutions, then practices on new problems and earns points for correct, well-explained answers.

The Gap: What was missing was a smart recipe that lets a 10B model keep up with the giants: (1) fully unfrozen, unified pre-training across 1.2 trillion multimodal tokens; (2) scaled RL with strong verifiable and preference rewards; and (3) test-time parallel thinking (PaCoRe) that gathers and cross-checks multiple visual hypotheses before deciding.

🍞 Hook: You know how teams split up to search different areas for a lost item, then compare notes to find it faster?

🥬 The Concept: Parallel Coordinated Reasoning (PaCoRe)

• What it is: A way to think in parallel at test time by generating many candidate thoughts, then coordinating and synthesizing them into a single, better answer.
• How it works:
  1. Spin up multiple reasoning attempts in parallel, 2) Collect their different ideas, 3) Cross-check for agreement, evidence, and errors, 4) Produce a final, stronger solution.
• Why it matters: Without PaCoRe, the model might miss rare clues or get stuck in one line of thinking. 🍞 Anchor: For “How many birds are in this crowded photo?”, several parallel tries count different regions; the final step merges them into one accurate total.

Real Stakes: A compact yet powerful multimodal model means better homework help on a laptop, accurate document reading on a phone, improved accessibility tools that read signs and papers, smarter coding from UI screenshots, and more reliable on-device assistants. It reduces costs, broadens access, and still nails tough tasks like spatial reasoning, math tied to images, OCR, and GUI grounding—all without needing a warehouse of servers.

The “Aha!” Moment in one sentence: With the right training recipe—unified pre-training, scaled reinforcement learning, and parallel coordinated reasoning—a 10B multimodal model can rival or beat much larger systems in perception and reasoning.

Three Analogies:

1. School Orchestra: Instead of teaching strings and winds separately and hoping they sync later, you rehearse the whole orchestra together from day one (unified pre-training), then polish with a conductor’s guidance (SFT), add performance scoring (RL), and finally record multiple takes and mix the best parts (PaCoRe).
2. Detective Squad: One detective studies the case history (pre-training), a mentor shows solved cases (SFT), the team practices and gets graded on catching mistakes and proving claims (RL), and for a big case they send out many detectives to explore different leads at once before agreeing on the final theory (PaCoRe).
3. Sports Playbook: Players learn core skills together (pre-training), drill plays with a coach’s examples (SFT), run scrimmages where points reward smart moves (RL), and during a match, multiple spotters call out options in parallel and the captain chooses the best combined play (PaCoRe).

Before vs After:

• Before: Small models often had good speed but shallow perception or wobbly reasoning. They were seen as practical but limited.
• After: STEP3-VL-10B shows that unfreezing and jointly training vision-language, followed by strong RL and parallel test-time reasoning, lifts a compact model to frontier-level performance on OCR, grounding, math-in-vision, GUI grounding, and more.

Why It Works (intuition, not equations):

• Joint Growth: Training the eyes (Perception Encoder) and the voice/brain (Qwen3-8B decoder) together forms shared concepts—like learning “a bar in a chart” both visually and linguistically—so connections become natural, not patched later.
• Rewarded Improvement: RL’s verifiable checks reward not just right answers, but also consistent steps; preference rewards nudge the model toward clear, safe, and helpful behavior even when there isn’t a single “right” answer.
• Parallel Safety Net: PaCoRe hedges against missing a clue by exploring many views in parallel, then coordinating a careful synthesis. This is especially powerful for dense images, spatial puzzles, counting, and tricky math.

🍞 Hook: Imagine a camera that’s already tuned to describe what it sees in words.

🥬 The Concept: Perception Encoder

• What it is: The visual backbone that turns images into features already friendly to language understanding.
• How it works:
  1. Break images into patches and extract features, 2) Compress tokens with a projector so they’re efficient to process, 3) Keep fine details using multi-crop (global + local views).
• Why it matters: If the eyes don’t capture the right small details (like tiny text or thin lines), the thinker can’t reason correctly. 🍞 Anchor: Reading the tiny price on a menu photo or the small axis labels on a graph becomes reliable because the encoder preserves those details.

🍞 Hook: Think of the decoder as the storyteller who explains what the eyes saw.

🥬 The Concept: Qwen3-8B Decoder

• What it is: The language model head that turns multimodal features into fluent, logical text answers.
• How it works:
  1. Receive visual-language tokens, 2) Attend to relevant parts, 3) Generate step-by-step thoughts and final answers, 4) Learn to follow instructions and formats.
• Why it matters: Without a strong decoder, even perfect vision features won’t become clear, correct explanations. 🍞 Anchor: Given a chart image and a question like “Which month had the highest sales?”, it cites the month and explains how it read the bars.

🍞 Hook: You know how you learn faster when practice has immediate feedback?

🥬 The Concept: Reinforcement Learning (RL)

• What it is: A training method where the model tries answers and gets rewards for being correct, consistent, and helpful.
• How it works:
  1. Generate multiple answers, 2) Check them with verifiers and preference models, 3) Score and learn from the results, 4) Repeat many times.
• Why it matters: It pushes the model beyond imitation to mastery—handling edge cases, formatting carefully, and avoiding lucky guesses. 🍞 Anchor: For a geometry photo, the model earns higher rewards when it both points to the right triangle and shows valid steps to compute its area.

🍞 Hook: A teacher’s worked examples can set a strong baseline before free practice.

🥬 The Concept: Supervised Finetuning (SFT)

• What it is: The model studies high-quality example solutions to shape its initial behavior before RL.
• How it works:
  1. Curate good multimodal prompts and answers, 2) Filter out low-quality data and leaks, 3) Train on text-first, then balanced text+vision data, 4) Learn long-context formatting.
• Why it matters: SFT teaches style, structure, and core reasoning patterns so RL starts from a capable base. 🍞 Anchor: The model learns the habit of showing steps for math or using precise coordinates for GUI grounding from teacher examples.

🍞 Hook: Instead of just one brain thinking longer, imagine many brains thinking at once and debating.

🥬 The Concept: Parallel Coordinated Reasoning (PaCoRe)

• What it is: At inference, the model creates several parallel chains of thought and then synthesizes them into a single, stronger answer.
• How it works:
  1. Launch 16+ parallel rollouts, 2) Gather different hypotheses, 3) Cross-check agreements and spot contradictions, 4) Write a final, verified solution.
• Why it matters: This boosts recall on visual details and reduces errors in complex reasoning. 🍞 Anchor: For a cluttered UI screenshot, parallel paths each locate and describe different buttons; the final answer identifies the exact button to click and why.

Building Blocks:

• Language-aligned Perception Encoder + projector with spatial downsampling to keep visual tokens efficient but informative.
• Qwen3-8B decoder to reason and generate text with long context windows.
• Unified, fully unfrozen pre-training over 1.2T tokens mixing knowledge, education, OCR, grounding, VQA, and GUI data.
• Two-stage SFT that first focuses on text reasoning, then balances multimodal reasoning.
• Scaled RL: hundreds of iterations of verifiable-reward RL for correctness, then RLHF for preference alignment and safety.
• PaCoRe to scale test-time compute via parallel exploration and coordinated synthesis.

High-level pipeline: Input (image + text) → Perception Encoder + projector + multi-crop → Qwen3-8B decoder (pre-trained jointly) → SFT Stage 1 (text-heavy) → SFT Stage 2 (balanced multimodal) → RL with verifiable rewards (RLVR) → RL from human feedback (RLHF) → PaCoRe training for coordinated parallel synthesis → Output (step-by-step reasoning + final answer).

Step A: Unified, Fully Unfrozen Pre-training

• What happens: The visual encoder and the language decoder train together across 1.2T tokens of multimodal data. A projector compresses visual tokens (16× spatial downsampling), and a multi-crop strategy provides a global 728×728 view plus local 504×504 crops to capture both context and fine details. Simple positional cues (newline per row + 1D RoPE) encode spatial structure.
• Why this exists: If vision and language are glued late, they often don’t align tightly. Jointly updating both halves creates shared concepts like “table cell,” “axis tick,” or “bounding box,” which makes later reasoning more reliable.
• Example: On document pages, the model learns to track text lines, headings, and tables so later OCR questions like “What’s the total in the bottom-right cell?” become straightforward.

🍞 Hook: Like choosing your study materials—textbooks, worksheets, and practice exams—that teach both basics and tricky cases.

🥬 The Concept: Multimodal Data Mixture

• What it is: A curated blend of knowledge-rich web image–text pairs, education problems (K–12 to professional), OCR (image-to-text/code, document parsing), grounding and counting, VQA, and GUI tasks.
• How it works:
  1. Gather and clean web data, 2) Balance long-tail concepts with CLIP clustering, 3) Generate high-fidelity synthetic charts/code, 4) Add GUI captions, Q&A, trajectories, grounding, and OCR coordinates.
• Why it matters: Wide, well-balanced coverage teaches the model to read, locate, count, reason, and act across many real scenarios. 🍞 Anchor: The same model can read a scientific diagram, answer a history question about an illustration, count cars in a street photo, and click the right button in a software UI.

Step B: Supervised Finetuning (Two Stages)

• What happens: Stage 1 is text-dominant (9:1 text:vision) to set a strong logic and formatting base with long contexts (up to 128k). Stage 2 balances (1:1) to deepen multimodal reasoning and instruction-following.
• Why this exists: A consistent, teacher-led foundation reduces flukes and stabilizes later RL.
• Example: The model learns to always show math steps for clarity and to output structured coordinates or boxes for grounding.

🍞 Hook: Practicing a game with a scoreboard that rewards fair play and correct moves.

🥬 The Concept: Reinforcement Learning (RL) with Verifiable Rewards (RLVR)

• What it is: The model practices on tasks with ground truth, earning rewards for correct and consistent answers.
• How it works:
  1. Try 16 rollouts per prompt, 2) Score exact perception outputs (IoU/distance) and math/logic via a strong model-based verifier that checks equivalence and process consistency, 3) Learn from rewards across hundreds of iterations.
• Why it matters: This teaches precision—point exactly here, count correctly, derive valid math—not just plausible text. 🍞 Anchor: For “Click the ‘Submit’ button on this UI,” the model is rewarded only when its predicted box tightly matches the real button’s location.

🍞 Hook: Some questions don’t have a single right answer, like “Explain this chart clearly.”

🥬 The Concept: RL from Human Feedback (RLHF) with Preference Rewards

• What it is: The model learns preferences on open-ended tasks using a reward model (GenRM) and guardrails.
• How it works:
  1. Compare multiple candidate answers to teacher references, 2) Reward clarity, helpfulness, and reasoning quality, 3) Penalize hallucinated citations, language mismatches, and overconfidence, 4) Encourage calibrated, safe answers.
• Why it matters: This aligns the model with human expectations beyond strict correctness. 🍞 Anchor: When summarizing a busy infographic, the model is rewarded for accurate, concise summaries with proper caution about uncertain parts.

Step C: Scaling Test-time Compute with PaCoRe

• What happens: The model is further trained to coordinate parallel thoughts. During inference, it runs many SeRe (sequential reasoning) rollouts in parallel (e.g., 16), collects their hypotheses, and synthesizes a final answer. Training uses carefully filtered prompts that remain challenging even with parallel context, preventing trivialization.
• Why this exists: Some perception tasks suffer from “length diminishment”—shorter, sharper policies are better—but can still miss rare details. Parallel exploration boosts recall and lets the model cross-check itself before answering.
• Example: For spatial puzzles (rotated views, 3D cues), different rollouts focus on different regions or transformations; synthesis captures the best of each.

🍞 Hook: Think of PaCoRe like many teammates proposing counts in different parts of a crowd photo.

🥬 The Concept: Synthesis and Cross-checking

• What it is: A structured step where the model weighs multiple candidate answers, rewards agreements backed by evidence, and flags contradictions.
• How it works:
  1. Serialize parallel messages into a synthesis context, 2) Identify consensus regions and conflicts, 3) Revisit the image for targeted checks, 4) Output a validated final.
• Why it matters: It cuts down on misses and overconfident mistakes, especially in dense, detail-heavy images. 🍞 Anchor: Counting overlapping objects becomes more accurate when parallel paths each handle a segment and the synthesizer resolves overlaps.

Secret Sauce:

• Fully unfrozen joint training knits vision and language tightly.
• RL with both strict (verifiable) and soft (preference) rewards sculpts correctness and helpfulness.
• PaCoRe turns extra test-time compute into better answers by structured parallel exploration and synthesis.
• Smart data construction (e.g., GUI trajectories with action grounding) turns perception into actionable intelligence.

The Test: The team evaluated STEP3-VL-10B on over 60 benchmarks spanning multimodal reasoning (MMMU, MathVista, MathVision), recognition and VQA (MMBench, MMStar), OCR and document understanding (OCRBench, AI2D), spatial reasoning (BLINK, All-Angles-Bench), counting (CountQA), GUI grounding (ScreenSpot-V2, OSWorld-G), plus text-only exams, math (AIME 2024/2025), coding (LiveCodeBench), instruction following, and subjective quality.

The Competition: Comparisons include strong 7B–10B open models (GLM-4.6V-Flash 9B, Qwen3-VL-Thinking 8B, InternVL-3.5 8B, MiMo-VL-RL-2508 7B), as well as much larger systems (GLM-4.6V 106B, Qwen3-VL-Thinking 235B) and top proprietary models (Gemini-2.5 Pro, Seed-1.5-VL).

The Scoreboard with Context:

• MMBench (recognition and understanding): 92.2% (EN/CN average ~92.17%), like scoring an A that ties or beats many bigger models.
• MMMU (broad multimodal reasoning): 80.11% with PaCoRe—think moving from a solid B to an A- through parallel thinking.
• MathVision (multimodal math): 75.95% with PaCoRe—significantly ahead of many peers; PaCoRe adds roughly +5 points over sequential mode.
• AIME2025 (text math): 94.43%—like acing a very hard math test; this is remarkable for a 10B model.
• OCRBench (text-in-image): 86.75%—frontier-class document intelligence for a compact model.
• GUI Grounding: ScreenSpot-V2 92.61%, OSWorld-G 59.02%—strong actionable perception.
• Spatial Understanding: Big gains with PaCoRe on All-Angles-Bench (+7.5%) and SpatialViz-Bench (+6.5%).
• Coding: LiveCodeBench 75.77%—strong text-only coding ability preserved despite heavy multimodal training.

Surprising Findings:

1. No Trade-off Penalty: Unlike many VL models that lose text strength when trained on images, STEP3-VL-10B preserves strong text-only performance (e.g., AIME and coding).
2. Emergent Spatial Skill: Even without hyper-specialized spatial data curation, spatial reasoning is strong—suggesting unified pre-training and scaled RL foster generalizable spatial concepts.
3. PaCoRe Helps Both Worlds: Reasoning-heavy tasks (MathVision, DynaMath) and perception-heavy tasks (counting, OCR, spatial) all benefit from parallel synthesis.
4. Length Dynamics: In perception tasks, better policies got shorter (more confident, fewer wandering tokens), while reasoning tasks benefited from longer chains or parallel coverage—matching the paper’s “length diminishment” vs “sequential scaling” insight.

Human Alignment and Usability:

• Instruction following and subjective metrics show high human preference for the model’s answers, indicating practical readiness.
• Behavioral regularization during RL reduces hallucinated citations and overconfident claims, producing more trustworthy outputs.

Big Picture: A compact 10B model consistently outperforms many peers and holds its own against 10–20× larger systems and even top proprietary models when boosted with PaCoRe. This flips the old belief that “only bigger is better,” showing that training strategy and test-time teamwork can close most of the gap.

Limitations:

• Compute at Inference: PaCoRe improves accuracy but uses extra test-time compute to run many parallel thoughts. For ultra-low-latency or tiny-device settings, sequential mode may be preferable, but with somewhat lower accuracy.
• Data Hunger: Unified pre-training used 1.2T tokens—a carefully curated, massive corpus. Reproducing this without strong data pipelines can be hard.
• Perception Traces: The paper notes that perceptual reasoning traces (like glance–focus–verify) are underrepresented in typical training data, so the model relies on PaCoRe at test time to simulate them.
• Edge Cases: Extremely fine details (e.g., ultra-tiny fonts, heavy occlusions) or unusual layouts may still trip the model compared to even larger frontier systems.

Required Resources:

• Training: Large-scale GPU/TPU clusters for trillion-token pre-training, then long-context SFT and 1k+ RL iterations.
• Inference: For PaCoRe, more compute for parallel rollouts and long contexts (up to ~131k tokens). Sequential mode needs less.
• Data: High-quality multimodal datasets (OCR, GUI trajectories, charts/code renderings, grounding) and robust filtering.

When NOT to Use:

• Ultra-tight latency or severe memory budgets where parallel rollouts are impossible and exactness is not critical.
• Domains with fast-changing, private, or regulated content where training and verification data are unavailable or restricted.
• Tasks needing physical interaction without the necessary action grounding or simulation feedback (beyond current GUI scope).

Open Questions:

• Can we distill PaCoRe’s “many-brains” traces into a single fast brain (compress System 2 into System 1) to keep accuracy while cutting latency?
• How far can verifiable rewards scale into video, 3D, and robotics where the “right answer” is an action that changes the world?
• What’s the best recipe to generate and incorporate perceptual reasoning traces during training, not just at test time?
• Can unified pre-training plus RL generalize robustly to fully embodied agents, where physics supplies the ultimate ground truth?
• How to optimally balance sequential depth vs parallel width for different task families (math vs OCR vs spatial)?

Three-sentence summary: STEP3-VL-10B is a compact, open multimodal model that learns vision and language together on 1.2T tokens, then is sharpened by scaled SFT and RL. At test time, it uses Parallel Coordinated Reasoning (PaCoRe) to explore many hypotheses in parallel and synthesize stronger answers. As a result, it matches or surpasses much larger models across perception, spatial reasoning, OCR, math-in-vision, and even text-only math and coding.

Main Achievement: Showing that smart training (fully unfrozen, unified pre-training + robust RL) and smart inference (PaCoRe) can let a 10B model achieve frontier-level multimodal intelligence without just scaling parameters.

Future Directions:

• Distill PaCoRe’s parallel traces into fast, single-pass intuition to reduce latency while keeping accuracy.
• Expand verifiable RL to richer modalities (video, 3D) and embodied environments where physics provides clean rewards.
• Enrich data with explicit perceptual reasoning traces, enabling the model to internalize glance–focus–verify behaviors.

Why Remember This: STEP3-VL-10B turns the old “bigger is always better” idea upside down by proving that better recipes and teamwork-at-inference can rival size. It opens the door to affordable, powerful multimodal assistants that read documents, reason about charts and diagrams, navigate interfaces, and solve tough math—on everyday hardware.