Innovator-VL: A Multimodal Large Language Model for Scientific Discovery

🍞 Top Bread (Hook): You know how a great science fair project often needs pictures (like diagrams) and words (like labels and steps) to make sense together? If you only had one or the other, it would be hard to understand the whole story.

🥬 Filling (The Actual Concept): What it is: Innovator-VL is a multimodal large language model (MLLM) built to understand both images and text, especially for science tasks like reading charts, decoding chemical diagrams, or analyzing microscope images. How it works (story of the world before, problem, failed attempts, the gap, and why we care):

1. The World Before:

• AI could caption images or answer simple questions, but struggled with science images that have tiny symbols, labels, and exact rules.
• Many models did well on general picture tasks (like “What color is the bag?”) but not on careful scientific reasoning (like “Which chemical group is here and how does it react?”).

1. The Problem:

• Scientific multimodal tasks need three hard things at once: precise vision (read the tiny parts), step-by-step logic (reason over multiple steps), and domain knowledge (facts from chemistry, physics, biology, etc.).
• Existing open models often fell short when all three were needed together, so their science answers were unreliable.

1. Failed Attempts:

• Some teams tried massive, domain-specific pretraining—collecting huge science-only datasets. That helped, but was extremely expensive, hard to reproduce, and sometimes made the model worse at normal tasks.
• Others used pipelines with many hidden tricks. Results were good but hard for the community to repeat or build upon, slowing progress for everyone.

1. The Gap:

• What was missing was a clean, transparent, and efficient recipe that shows: with smart data selection, better visual understanding, and principled training, you can get strong scientific reasoning without billions of science-only examples.

1. Real Stakes (Why it matters):

• In daily life and research, science is everywhere: reading medical charts, checking lab results, understanding patents, and analyzing satellite or microscope images.
• A reliable, efficient, and reproducible scientific MLLM helps students learn STEM concepts, supports scientists in reading literature, and speeds up discovery—without needing supercomputers or secret data.

Why it matters: Without a model like this, we either overspend on giant datasets, accept lower accuracy, or sacrifice general abilities for science performance—none of which is good for real-world science use.

🍞 Bottom Bread (Anchor): Imagine a chemistry student snapping a photo of a reaction scheme from a paper. Innovator-VL can read the symbols, connect them with text, and explain what’s happening step by step—accurately and efficiently—without needing a warehouse full of extra data to learn that skill.

🍞 Top Bread (Hook): Imagine building a winning LEGO robot not by adding more bricks, but by choosing the right pieces and following a clear plan. You get more power with less clutter.

🥬 Filling (The Actual Concept): What it is: The key insight is that careful design and high-quality, well-targeted data—plus a transparent training recipe—can unlock strong scientific multimodal reasoning without massive domain-specific pretraining. How it works:

1. Use a vision encoder (RICE-ViT) that sees fine-grained scientific details (like symbols and labels).
2. Compress image tokens with PatchMerger so the language model can think efficiently.
3. Start from a strong general language brain (Qwen3-8B-Base) and align images to it.
4. Fine-tune with three curated data types: general instructions, chain-of-thought reasoning, and high-quality scientific datasets.
5. Finish with reinforcement learning (GSPO) guided by a reward system that checks both structure (<think> and <answer> tags) and correctness. Why it matters: Without this approach, you either drown in data or lose accuracy; with it, you get state-of-the-art science reasoning that still works great on everyday vision tasks.

Multiple analogies (same idea, 3 ways):

• Swiss Army Scientist: Instead of carrying 100 tools (massive data), carry a smart few: a sharp microscope (RICE-ViT), a neat organizer (PatchMerger), and a wise brain (Qwen3-8B). You’re lighter and still ready for any job.
• Orchestra Conductor: Good music needs the right players and a clear score, not more instruments. RICE-ViT plays details, PatchMerger harmonizes them, Qwen3 conducts, and RL polishes timing and clarity.
• Cooking with Quality Ingredients: Fresh, carefully measured ingredients (curated data) and a precise recipe (transparent pipeline) beat tossing everything in a pot (indiscriminate scaling).

Before vs After:

• Before: Models needed huge special datasets, struggled to generalize, and often became less capable at general tasks.
• After: Innovator-VL achieves top science reasoning with fewer than five million scientific samples, stays strong at general vision, and is fully reproducible.

Why it works (intuition, no equations):

• Better Seeing: RICE-ViT pays attention to the small parts science needs (regions, symbols, OCR-like text), so the language part doesn’t carry the whole burden.
• Better Packing: PatchMerger removes token clutter, letting the model think more with less.
• Better Teaching: SFT gives examples of how to follow instructions and reason; RL rewards not only correct answers but also clean, structured thinking. Together, they turn potential into reliable performance.

Building blocks (each with a sandwich explanation):

• 🍞 You know how magnifying glasses help you read tiny map labels? 🥬 RICE-ViT: What it is: A vision encoder that captures region-level details. How: It adds region-aware layers so the model notices local structures (like symbols, arrows, axes). Why: Without it, tiny scientific details get lost. 🍞 Anchor: Reading a chemistry diagram where one short line changes the whole meaning.

• 🍞 Imagine packing a suitcase smartly so you carry less but keep the important stuff. 🥬 PatchMerger: What: A learned token compressor for images. How: It merges many visual patches into a smaller set of representative tokens. Why: Without it, reasoning costs explode and slows everything. 🍞 Anchor: Turning a 100-piece puzzle into a 20-piece summary that still shows the whole picture.

• 🍞 Think of borrowing a brilliant tutor’s brain to explain what you see. 🥬 Qwen3-8B-Base: What: A strong language model backbone with broad knowledge. How: It takes compressed visual tokens plus text and produces answers. Why: Without a capable language brain, even clear pictures won’t lead to good explanations. 🍞 Anchor: Explaining a microscope image using proper science words and logic.

• 🍞 Like learning the alphabet before writing essays. 🥬 Language-Image Alignment & Mid-Training: What: First align image tokens with words; then inject high-quality general multimodal knowledge. How: Train projector on LLaVA-1.5 558k, then full-parameter mid-training on a curated 85M set. Why: Without alignment and broad practice, the model stumbles when pairing pictures with text. 🍞 Anchor: Matching chart bars to their labels before answering questions.

• 🍞 Think of a coach showing play-by-play strategies. 🥬 Supervised Fine-Tuning (SFT): What: Teach instruction-following, step-by-step reasoning, and scientific understanding. How: Use three data types—general instructions, chain-of-thought/multi-step, and scientific datasets. Why: Without SFT, the model won’t follow directions or think clearly. 🍞 Anchor: Solving a physics diagram question using numbered steps.

• 🍞 Like getting points for both neat work and correct answers. 🥬 Reinforcement Learning (GSPO + Rewards): What: Fine-tune policies using sequence-level optimization and a hierarchical reward. How: GSPO updates whole responses with trust regions; rewards check format tags and correctness (heuristics → symbolic math → LLM judge). Why: Without RL, the model’s good ideas don’t show up consistently. 🍞 Anchor: Producing compact, correct math solutions with a clear final boxed answer.

At a high level: Input (text + one or many images) → RICE-ViT encodes each image at native resolution → PatchMerger compresses visual tokens → Concatenate with text tokens → Qwen3-8B-Base reasons and outputs a response with optional <think> and <answer> sections.

Step-by-step with sandwich explanations and examples:

1. 🍞 Hook: Imagine reading a comic book where each panel is different in size; you want to see each panel clearly. 🥬 RICE-ViT (Vision Encoder): What it is: A region-aware vision transformer that encodes each image at its native size into visual tokens. How: It processes images through layers that attend to regions, capturing fine-grained elements (symbols, text boxes, arrows) and global context simultaneously. Why it matters: Without region awareness, tiny scientific notations or labels can be missed, breaking downstream reasoning. 🍞 Anchor: In a physics diagram with small force arrows, RICE-ViT keeps those arrows distinct so the model knows which way the force points.

2. 🍞 Hook: You know how we summarize a long story into a short book report? 🥬 PatchMerger (Projector): What it is: A learned token compressor that turns many visual tokens into fewer, more meaningful ones. How: It merges related patches into representative tokens using learned weights, keeping semantics while reducing length. Why it matters: Without compression, the language model sees too many tokens, making reasoning slow and memory-hungry. 🍞 Anchor: A 2,000-token scan of a document becomes a compact set of tokens that still preserves the table’s axes, legends, and key numbers.

3. 🍞 Hook: Picture a smart narrator who explains what the pictures mean in clear sentences. 🥬 Qwen3-8B-Base (Language Model): What it is: The reasoning and generation core that reads text plus compressed visual tokens. How: It takes the concatenated tokens and predicts step-by-step reasoning (<think>) and a final answer (<answer>). Why it matters: Without a strong language backbone, even perfect visual features won’t produce accurate, explainable answers. 🍞 Anchor: Given a bar chart and a question, it explains which bar is highest and why, then states the final value.

4. 🍞 Hook: Before solving puzzles, you first learn the matching rules between pieces. 🥬 Language-Image Alignment (Pre-training stage 1): What it is: Train the projector so image features align with the language embedding space. How: Use LLaVA-1.5 558k pairs to guide the projector to produce tokens the LLM understands. Why it matters: Without this, the words and pictures “speak different languages.” 🍞 Anchor: Matching label “NaCl” to the correct salt crystal image snippet.

5. 🍞 Hook: After learning the alphabet, you read many stories to grow general knowledge. 🥬 High-quality Mid-Training (Pre-training stage 2): What it is: Full-parameter training on a curated 85M multimodal corpus (diverse, concept-balanced). How: Use sources like COYO, Obelics, LAION-CN, DataComp, etc., with pseudo-captioning and concept-balanced sampling to improve breadth and robustness. Why it matters: Without broad multimodal practice, the model overfits or misses everyday visual skills. 🍞 Anchor: Seeing many chart types, the model learns legends, axes, and units so later it can read any new chart format.

6. 🍞 Hook: Think of a teacher giving three kinds of homework: general, show-your-work, and science labs. 🥬 Supervised Fine-Tuning (SFT): What it is: Full-parameter instruction fine-tuning to teach following directions and reasoning. How and why (three data groups):

   • General Multimodal Instruction Data: What: 22M samples covering captions, charts, OCR, grounding, etc. How: From LLaVA-OneVision-1.5-Instruct. Why: Builds reliable instruction-following across common tasks. Example: “Describe this infographic’s key points.”
   • Chain-of-Thought & Multi-step Reasoning Data: What: ~15M reasoning-focused examples (Honey-Data-15M) with cleaned formatting (fewer explicit think tags). How: Teach structured, stepwise problem solving. Why: Without this, the model gives shallow answers. Example: Solving a geometry diagram with numbered steps.
   • Scientific Understanding Data: What: Carefully built datasets in three subfields with expert-in-the-loop pipelines. How: • OCSR with E-SMILES: Start with 7M synthetic pairs (rendered structure ↔ E-SMILES), add real patent/paper crops, ensemble confidence, and expert corrections in a loop. • Reaction Understanding (Rxn-like): Parse PDFs, crop schemes, generate Q/A candidates, and have chemists refine plus add hard distractors. • EM Microstructure: Aggregate EM images, clean and crop, add instance segmentation labels and structured descriptions via multi-stage expert workflows. Why: Without quality scientific data, the model won’t grasp real notations, edge cases, and document complexity. Example: Reading a chemical scheme with R-groups and picking the correct product among look-alike options.

7. 🍞 Hook: Good students not only know answers—they present them clearly and efficiently. 🥬 Reinforcement Learning (GSPO + Rewards): What it is: Post-training to elicit consistent, concise, correct reasoning. How:

   • Data: 172K discrepancy-driven RL instances that the model can solve sometimes (high Pass@N) but not consistently (low Pass@1), standardized into a uniform reasoning format.
   • GSPO: Optimize at the sequence level (not token-by-token), aligning the learning signal with how we score full answers; use a trust region for stability.
   • Reward: Combine format reward (proper <think>/<answer> tags or boxed final) and accuracy reward via a cascade: exact/regex match → symbolic math check → LLM-as-judge for open questions. Why it matters: Without RL, the model may have hidden ability but won’t reliably produce the best, concise chain of thought. 🍞 Anchor: For a multi-step algebra word problem with a diagram, the model produces a short, correct reasoning chain and a boxed final number.

Secret Sauce (what makes it clever):

• Region-aware vision plus learned token compression means the LLM gets cleaner inputs.
• A fully transparent, staged recipe ensures alignment first, then breadth, then instruction/CoT, then RL polishing.
• Discrepancy-driven RL focuses learning where it pays off most—turning potential into dependable first-try answers.

Example walk-through (with real-ish data flavor):

• Input: “From this reaction scheme image, which reagent produces the major product?” + the scheme figure.
• RICE-ViT: Encodes symbols, arrows, and reagents.
• PatchMerger: Compresses thousands of visual tokens to a compact set.
• Qwen3-8B: Reads the question and tokens, reasons in <think>, checks equivalents, and outputs <answer> (e.g., “Benzyl bromide”).
• RL shaping ensures the <answer> tag is present and correct, and the reasoning is concise.

🍞 Top Bread (Hook): You know how in a spelling bee, it’s not just how many words you spell right—it also matters how quickly and neatly you answer under pressure.

🥬 Filling (The Actual Concept): What it is: The authors tested Innovator-VL on 37 benchmarks across three groups—General Vision, Math & Reasoning, and Science—comparing it to strong 7B–9B models. How it works (tests, competition, scoreboard with context, surprises):

1. The Test (What they measured and why):

• Accuracy and consistency across many datasets: Can the model understand diagrams, read text in images, solve math with pictures, and handle real scientific data?
• Token efficiency: How many tokens does it use to reason? Does each token pull its weight?
• Balance: Stay strong at general tasks while excelling at science.

1. The Competition (Who/what they compared against):

• Models of similar size: Qwen3-VL-8B, InternVL3.5-8B, Intern-S1-mini (9B), LLaVA-OneVision 1.5-8B, MiMo-VL-7B (SFT/RL), and MiniCPM-V 4.5 (8B).

1. The Scoreboard (with context):

• Overall: Innovator-VL-8B-Thinking averaged 61.83% across all benchmarks—like earning an A- where the class average is closer to a B.
• General Vision: Innovator-VL-8B-Instruct averaged 74.50%, on par with Qwen3-VL-8B (74.71%), while beating others on tasks like AI2D and RealWorldQA. Translation: it sees everyday images as well as the best peers.
• Math & Reasoning: Innovator-VL-8B-Thinking scored 55.41%, topping the field and improving +4.54% over its Instruct version. That’s like moving from a B to a solid A- in tough visual math quizzes.
• Science: Innovator-VL leads with 50.13% (Thinking) and 49.79% (Instruct), well above general models. On especially hard chemistry tests—OpenRxn (57.05%) and MolParse (64.90%)—others often scored below 17%. That’s a landslide win.

1. Token Efficiency (numbers that matter):

• Shorter chains, better answers: On several reasoning benchmarks, Innovator-VL-8B-Thinking used 62–66% fewer tokens than Intern-S1-mini and 18–48% fewer than MiMo-VL-7B-RL.
• Accuracy per token: 1.4–2× higher than MiMo-VL-7B-RL and 3.9–4.3× higher than Intern-S1-mini. Like solving more puzzles with fewer moves.

1. Surprising Findings:

• Less can be more: With fewer than five million targeted scientific samples (not billions), the model achieves top science performance.
• No trade-off trap: Scientific alignment didn’t ruin general vision skills; both coexisted well.
• RL shaped brevity: The reward design didn’t just boost correctness; it taught the model to be concise, improving real-world efficiency and cost.

Why it matters: These results mean researchers, students, and engineers can get high-quality scientific answers without paying heavy compute costs or relying on giant, opaque datasets.

🍞 Bottom Bread (Anchor): Picture a chemistry exam with tricky look-alike options based on a reaction scheme. Innovator-VL consistently picks the right one, writes a short, neat solution, and does it faster—like the student who not only aces the test but finishes with time to spare.

🍞 Top Bread (Hook): Imagine a top student who’s great in many subjects but still has areas to grow, like speaking a new language or playing a new instrument.

🥬 Filling (The Actual Concept): What it is: An honest look at Innovator-VL’s limits, resources needed, when not to use it, and open questions. How it works:

1. Limitations (specific):

• Domain coverage: Performance is strongest where the curated scientific data is rich (e.g., chemistry diagrams, reaction schemes, EM microstructures). Less-covered niches (e.g., some biology subdomains or exotic notations) may still challenge the model.
• Synthetic biases: While synthetic bootstrapping accelerates data creation, it can introduce style biases that differ from real-world images or documents.
• Symbol complexity: Ultra-dense pages with overlapping OCR text, tiny subscripts, or unusual fonts may still cause misreads.
• Open-ended generation: For very long, unconstrained scientific essays, factuality can drift without tool grounding.

1. Required Resources:

• Training: Multi-GPU training infrastructure (as used in the paper with efficient pipelines) to reproduce full results.
• Inference: Modest GPU/CPU for deployed reasoning; thanks to token efficiency, serving costs are lower than many peers.
• Data: Access to the released instruction/RL datasets; scientific SFT subsets benefit from expert-in-the-loop if extended.

1. When NOT to Use:

• Ultra-specialized domains with unique symbols/formats unseen in training and with no quick adaptation data.
• Tasks demanding formal proofs or heavy numerical simulation (without external tools), where dedicated solvers are better.
• High-stakes medical or legal decisions without human review.

1. Open Questions:

• Tool Integration: How much can external tools (chemistry engines, OCR, symbolic solvers) further boost accuracy and safety?
• New Modalities: What gains arrive by adding video, 3D molecular structures, and time-series signals (e.g., spectra, sensor data)?
• Data Recipes: What’s the best mix between synthetic and real data to reduce bias yet keep coverage high?
• Safety & Attribution: How to ensure correct citations to papers/patents and reduce hallucinations in subtle scientific claims?

Why it matters: Knowing these boundaries guides safe, effective use and points the way to the next round of improvements.

🍞 Bottom Bread (Anchor): If you hand the model a rare lab diagram style from the 1970s it never saw, it might stumble—just like a student seeing a brand-new notation. Giving it a few examples or a tool plugin often fixes it.

🍞 Top Bread (Hook): Think of Innovator-VL as a well-trained science teammate who sees clearly, thinks step by step, and explains answers neatly—without needing a giant private library.

🥬 Filling (The Actual Concept):

1. Three-sentence summary:

• Innovator-VL is a transparent, reproducible multimodal model that fuses a region-aware vision encoder, a token-merging projector, and a strong language backbone to solve scientific tasks.
• With fewer than five million carefully curated scientific samples plus principled SFT and GSPO-based RL, it achieves state-of-the-art science results and strong general vision performance.
• Its reward design improves both correctness and conciseness, yielding excellent token efficiency and practical deployment benefits.

1. Main Achievement:

• Proving that smart design and high-quality targeted data can reliably beat brute-force domain scaling for scientific multimodal reasoning—without sacrificing general abilities.

1. Future Directions:

• Add video, 3D molecules, and time-series; integrate external scientific tools and knowledge bases; compress further for edge devices; and expand expert-in-the-loop data recipes across more scientific fields.

1. Why Remember This:

• Innovator-VL shows a clean, repeatable path to scientific AI that is efficient, accurate, and shareable—pointing the community toward progress that is powerful, practical, and fair to reproduce.

🍞 Bottom Bread (Anchor): Just like a carefully coached debate team wins with clear structure and sharp facts—not by shouting louder—Innovator-VL wins with better seeing, smarter training, and focused practice.