BABE: Biology Arena BEnchmark

🍞 You know how in science class you don’t just memorize facts—you look at a picture of a lab result, read the setup, and then explain what it means? That mix of seeing, thinking, and concluding is what real scientists do every day.

🥬 The Concept — Experimental Reasoning:

• What it is: Using experiments plus their context to figure out what’s truly happening.
• How it works:
  1. Look at the evidence (like a gel image or a plot),
  2. Read the setup (what cells, what treatment),
  3. Connect the two to reach a conclusion (e.g., “this protein increased”).
• Why it matters: Without it, you might glance at a band on a gel and jump to the wrong idea because you missed the control or the condition. 🍞 Anchor: Imagine judging a baking contest by only seeing a frosting color. Experimental reasoning is tasting the cake, reading the recipe, and then deciding which is best.

🍞 You know how pushing a light switch can cause the light to turn on? We care about the cause (switch) and the effect (light).

🥬 The Concept — Causal Reasoning:

• What it is: Figuring out if one thing actually makes another thing happen.
• How it works:
  1. Notice patterns (when A happens, B happens),
  2. Rule out coincidences (could it be C?),
  3. Use controls and comparisons to argue A → B.
• Why it matters: Without causal reasoning, we might think two things go together just by luck and make wrong decisions. 🍞 Anchor: If fertilizer makes plants grow taller in tests with good controls, you can say fertilizer causes growth, not just that they appear together.

🍞 Imagine chatting with a super-helpful robot that can read, write, and answer questions.

🥬 The Concept — Large Language Models (LLMs):

• What it is: Computer programs trained to read and generate human-like text.
• How it works:
  1. Learn from lots of text to spot patterns,
  2. Use those patterns to predict the next words,
  3. Follow instructions to solve tasks.
• Why it matters: LLMs can help with science, but only if they truly understand evidence, not just words. 🍞 Anchor: When you ask, “What does this graph mean?”, a good LLM should link the graph to the experiment’s goal and give a sound explanation.

🍞 Think of a school newspaper story that teachers check before it’s printed to make sure it’s right.

🥬 The Concept — Peer-Reviewed Research Papers:

• What it is: Scientific articles checked by expert scientists before publication.
• How it works:
  1. Scientists do a study and write it up,
  2. Other experts review it for quality,
  3. It gets published if it passes checks.
• Why it matters: Using peer-reviewed data makes tests more trustworthy and realistic. 🍞 Anchor: A figure from a peer-reviewed paper is like a trusted map you can use to find the right path in reasoning.

The world before BABE: AI benchmarks in biology often tested narrow skills—like reading a DNA sequence or naming a protein fold. These are useful, but they don’t measure the scientist’s core job: pulling meaning from experiments with proper context. This left a big blind spot: models could ace flash-card facts but stumble when asked, “Given this figure and method, what should we conclude?”

The problem: We lacked a way to evaluate whether AI can integrate evidence (figures, tables), context (methods, controls), and background knowledge (biology concepts) to make causal, research-level conclusions.

Failed attempts: Many benchmarks were sequence-centric or structure-centric; a few were multimodal but still light on deep experimental reasoning. They didn’t require linking evidence to context in multi-step ways, so models could succeed by surface pattern matching.

The gap: We needed a benchmark built from real, peer-reviewed studies that forces models to do scientist-style reasoning: interpret experimental results, connect them to conditions, and make justified, causal claims.

Real stakes: This matters for drug discovery, diagnostics, agriculture, and public health. If an AI misreads an experiment, it could suggest a wrong target or overlook a safety signal. Better evaluation leads to safer, smarter tools that help scientists discover faster and with fewer errors.

🍞 Imagine a science fair where each team must answer three linked questions about one real experiment—some answers depend on previous ones, and some do not. You must both follow a chain and keep separate facts straight.

The “Aha!” in one sentence: BABE tests whether AI can reason like a biologist by answering three-question sets from a single paper, where pairs are labeled as strong-correlation (needs sequential, multi-hop logic) or weak-correlation (independent, parallel extraction).

Multiple analogies:

• Detective case file: One case file, three clues; some clues must be solved in order (strong), others are side facts you keep separate (weak).
• Recipe making: Some steps require the previous step’s result (bake, then frost), while others are optional garnishes that don’t depend on the main dish.
• Map reading: Some routes require passing through specific checkpoints (strong), while other landmarks are optional and independent (weak).

🥬 The Concept — Strong Correlation:

• What it is: When answering one question depends on correctly solving a previous one.
• How it works:
  1. Solve Q1 to get a key fact,
  2. Use that fact to unlock Q2,
  3. Continue the chain to Q3.
• Why it matters: Without strong-correlation testing, we can’t tell if a model can do real multi-step reasoning. 🍞 Anchor: If you must read a Western blot to identify a protein increase (Q1) before deciding which pathway is activated (Q2), that’s strong correlation.

🥬 The Concept — Weak Correlation:

• What it is: Questions from the same paper that don’t depend on each other.
• How it works:
  1. Pull fact A from Figure 2,
  2. Pull fact B from Methods,
  3. Keep them separate without mixing details.
• Why it matters: Without weak-correlation testing, models might blend unrelated facts and get confused. 🍞 Anchor: Identifying a cell line in one panel and a reagent concentration in another—neither requires the other, but both must be correct.

Before vs. After:

• Before: Benchmarks rewarded short, shallow answers and single-hop lookups.
• After BABE: Evaluation checks if models can chain steps correctly when needed and also keep independent parts cleanly separated.

Why it works (intuition): Scientist-style understanding has two gears. Gear 1 (sequential) handles multi-hop logic—each step depends on the last. Gear 2 (parallel) handles keeping multiple facts straight without mixing them. BABE activates and inspects both gears using triplets labeled for strong vs. weak correlation from real papers, so shallow pattern matching isn’t enough.

Building blocks:

1. Single-source focus: All three questions come from one peer-reviewed document, preserving context.
2. Correlation labels: Each adjacent pair is marked strong or weak to diagnose sequential vs. parallel reasoning.
3. Expert pipeline: Domain experts write, review, and refine items to ensure correctness and difficulty.
4. Multimodality: Figures/tables + text mimic real lab interpretation.
5. Analysis tools: BABE observes how models reason (deep reasoning vs. unproductive self-reflection) and how multiple trials change outcomes.

🥬 The Concept — Multi-Hop Reasoning:

• What it is: Solving by linking several facts step-by-step.
• How it works:
  1. Extract Fact 1 (e.g., band intensity up),
  2. Combine with Fact 2 (treatment X applied),
  3. Conclude Cause (treatment X increases protein).
• Why it matters: Many biology conclusions need at least two or three hops; skipping breaks the logic. 🍞 Anchor: From “phospho-protein increased” + “kinase inhibitor removed” to “kinase activity likely drove the increase.”

🥬 The Concept — Chain-of-Thought (CoT):

• What it is: Writing out the reasoning steps clearly.
• How it works:
  1. State evidence,
  2. Connect with logic words (because, therefore),
  3. Reach a supported conclusion.
• Why it matters: Without explicit steps, hidden mistakes sneak in. 🍞 Anchor: Explaining, “Because lane 3 is the loading control and is constant, the higher band in lane 4 means true upregulation.”

🥬 The Concept — Cross-Scale Inference:

• What it is: Connecting different biological levels (molecules → cells → tissues → organisms).
• How it works:
  1. Read a molecular change,
  2. Predict cell behavior,
  3. Anticipate tissue/organism effect.
• Why it matters: Biology spans levels; a model must knit them together. 🍞 Anchor: From a gene mutation (molecular) to neuron firing changes (cell) to movement behavior (organism).

Together, these concepts make BABE a true arena for testing scientist-level reasoning, not just trivia recall.

At a high level: Single research paper → Expert-crafted 3-question set (triplet) → Correlation labeling (strong/weak) → Multi-round review and filtering → Model evaluation (with optional multi-trial inference) → Diagnostic analyses (reasoning behavior and convergence).

Step 1: Source selection (input)

• What happens: Curate recent, peer-reviewed biology papers and authoritative reviews across 12 subfields (e.g., cell biology, genetics, immunology).
• Why it exists: Real experiments and figures force realistic reasoning; toy tasks can be gamed.
• Example: Picking a paper where a CRISPR complex edits RNA and includes figures with gel bands and bar charts.

Step 2: Triplet construction

• What happens: Domain experts write three self-contained, unambiguous questions from one paper, targeting conceptual understanding, method interpretation, and higher-order reasoning.
• Why it exists: Three related questions reveal whether a model can sustain context and handle both sequential and parallel demands.
• Example: Q1: choose the correct RNA ligase; Q2: rank cut sites by cleavage; Q3: predict which guides yield highest reporter activity.

Step 3: Correlation labeling

• What happens: Reviewers label the logical relation between consecutive questions as strong correlation (sequential dependency) or weak correlation (independent extraction).
• Why it exists: Distinguishes multi-hop reasoning (error compounding risk) from parallel retrieval (interference risk).
• Example: If Q2 requires Q1’s answer, Q1–Q2 is strong; if Q3 reads a different figure that doesn’t need earlier answers, Q2–Q3 is weak.

Step 4: Quality assurance and difficulty control

• What happens: Senior experts check factual fidelity, clarity, and answer correctness; ambiguous or too-simple items are revised or removed. A second review round confirms fixes.
• Why it exists: Ensures that questions demand real reasoning, not guessable tricks.
• Example: If a question can be answered by keyword matching, it’s either reworked to require figure interpretation or discarded.

Step 5: Evaluation format

• What happens: Models read the single-source document and answer the three questions. Scores are reported overall and split by strong vs. weak correlation to diagnose strengths.
• Why it exists: A single source tests comprehension and consistency; sub-scores reveal whether a model handles chains, parallel facts, or both.
• Example: A model might score 55% on weak (good at parallel extraction) but only 49% on strong (struggles with multi-hop chains).

Step 6: Reasoning behavior analysis

• What happens: Inspect the model’s inference steps for proportions of deep reasoning, self-exploration, and self-reflection across the trajectory.
• Why it exists: BABE is designed to need deep reasoning; overuse of self-reflection can signal “overthinking loops.”
• Example: High performers keep deep reasoning steady from start to finish; low performers show bursts of reflection without progress.

Step 7: Multi-trial inference and convergence

• What happens: Run multiple independent attempts (N trials) and compare average vs. Best-of-N (BoN). Fit a saturating curve Gain(n) = a·(1 − e^(−bn)) to estimate asymptotic benefit and convergence speed.
• Why it exists: Real reasoning can benefit from diverse trajectories; BoN shows how much improvement is possible by sampling.
• Example: Even top models improve with 4–8 trials but show diminishing returns, indicating inherent task difficulty.

Secret sauce:

• Single-document triplets with explicit strong/weak correlation labels are the key diagnostic. They expose two major failure modes: chain breakage (strong) and context mixing (weak). Adding reasoning-behavior tracking and multi-trial convergence turns BABE into both a scoreboard and a microscope for how models think.

Concrete mini-walkthrough (with data flavor):

• Input paper: OK-seq replication study with RFD curves.
• Q1 (weak): Compute RFD sign in boxed regions (figure reading + definition).
• Q2 (strong with Q1): Pick which figure patterns match initiation vs. termination.
• Q3 (strong with Q2): Predict RFD slope shape at origins and count origin regions in another panel.
• If the model misreads Q1, it cascades into Q2/Q3 errors (strong-correlation failure). If it confuses panels between Q2 and Q3 despite independence, that’s weak-correlation interference.

What breaks without each step:

• No curated sources: Models overfit to trivia; results don’t transfer to real research.
• No triplets: Hard to see error propagation or context maintenance.
• No correlation labels: Can’t diagnose chain vs. parallel weaknesses.
• No QA: Ambiguity muddies whether errors are from the model or the dataset.
• No multi-trial: Miss recoverable gains and convergence behavior.
• No behavior analysis: Don’t learn which reasoning styles succeed.

The test: Measure how well various LLMs answer BABE triplets overall and in the strong vs. weak subsets. Also measure how their reasoning style (deep reasoning vs. self-reflection) and multiple trials (Best-of-N) affect performance.

The competition: Frontier, mid-tier, and baseline models, including OpenAI-GPT-5.1-high, Gemini-3-Pro-Preview-Exp, OpenAI-o3-high, Gemini-2.5-Pro, Claude variants, Qwen variants, Doubao variants, GPT4o, and GLM-4.5-V.

The scoreboard with context:

• OpenAI-GPT-5.1-high: Average 52.31 (strong 51.79, weak 52.86). That’s like barely over half the questions right on a graduate-style, figure-heavy exam—showing the test is tough even for leaders.
• Gemini-3-Pro-Preview-Exp: Average 52.02 (strong 49.05, weak 55.16). Like scoring an A- on parallel fact extraction but a B on multi-hop chains.
• OpenAI-o3-high.code: Average 51.62, balanced strong/weak.
• Mid-tier models: Generally in the 36–45 range, comparable to a challenging test where many items require careful multi-step figure reading.
• Lowest tier: ~20–23 average, indicating major difficulty with BABE’s requirements.

Strong vs. weak differences:

• Some models (e.g., Gemini-3-Pro-Preview-Exp) shine on weak correlation, suggesting strong parallel extraction but shakier multi-hop chaining.
• Others are balanced (e.g., Claude-Sonnet-4.5-thinking-azure, Gemini-2.5-Pro), signaling stable behavior across relation types but not necessarily high absolute accuracy.

Reasoning behavior insights:

• Success correlates with sustained deep reasoning throughout inference. High performers keep a steady cadence of evidence-based steps.
• Overuse of self-reflection without progress hurts accuracy—low performers fall into “overthinking loops,” burning steps while drifting from evidence.
• Early bursts of deep reasoning followed by drop-offs aren’t enough; BABE often requires revisiting premises and integrating new implications evenly until the end.

Multi-trial (Best-of-N) gains and convergence:

• All models improve with more trials; gains rise from N=1 to N=8, then start to saturate.
• Strong models (e.g., GPT-5.1-high, Gemini-3-Pro-Preview-Exp) converge faster with asymptotic gains around 30 points—meaning even robust single-pass reasoners benefit, but not endlessly.
• Some mid-tier models show higher potential gains (>35), implying diverse reasoning paths where BoN can rescue better solutions.
• Practical takeaway: Expect to run 4–6 trials for frontier models and 8+ for others to approach their potential on BABE.

Surprising findings:

• More self-reflection is not always better; quality and timing of reflection matter more than quantity.
• Weak-correlation strength doesn’t guarantee strong-correlation success; chain reasoning is a distinct skill.
• Despite strong general capabilities, top models hover near 52%—a clear sign that experimental reasoning remains a frontier challenge.

Limitations:

• Single-document scope: BABE focuses on reasoning within one paper; cross-paper synthesis is future work.
• Expert-labor intensive: High-quality, multimodal triplets require careful human creation and review.
• Modality variability: Not all items are equally multimodal; some subfields rely more on text than figures.
• Scoring granularity: While strong/weak labels help, even finer-grained dependency maps could capture richer logic chains.

Required resources:

• Access to the curated papers and figures, plus an evaluation harness for triplets.
• Models with enough context window to hold the paper excerpts and questions.
• Compute to run multi-trial inference (4–8+ runs) and to analyze reasoning behaviors.

When not to use BABE:

• If you only need simple fact recall or single-sentence QA.
• If your model can’t process longer contexts or figures/tables.
• If you can’t afford multi-trial runs; single-pass scores understate potential and miss convergence insights.

Open questions:

• Can training on synthetic but carefully designed multi-hop, multimodal curricula close the strong-correlation gap?
• How to guide reflection so it helps (plan–execute–reflect) without looping?
• What is the best way to attribute errors to perception (figure reading), retrieval (context selection), or logic (reasoning step) within BABE?
• Can we extend BABE to cross-paper chains and lab-protocol design while keeping reliability and fairness?

Three-sentence summary: BABE is a biology benchmark built from peer-reviewed studies that evaluates whether AI can integrate experimental results with context to reach scientific conclusions. It uses three-question triplets from a single document, labeled for strong (sequential) and weak (parallel) correlations, to diagnose multi-hop vs. independent reasoning skills. Results show that even top models struggle, and sustained deep reasoning—not excessive self-reflection—drives better performance, with multi-trial inference providing meaningful but saturating gains.

Main achievement: Establishing a research-derived, correlation-labeled, multimodal benchmark that exposes how well (or poorly) models perform true experimental reasoning like practicing scientists.

Future directions: Add cross-paper chains, richer multimodal inputs (e.g., raw gels, spectra), and more granular dependency annotations; explore training strategies that stabilize deep reasoning and productive reflection; and develop tool-augmented settings (code, search, calculators) under controlled rules.

Why remember this: BABE shifts evaluation from trivia to scientific thinking. It measures what matters in the lab—careful reading of evidence, correct chaining of logic, and keeping contexts straight—pointing the way toward AIs that can genuinely accelerate biological discovery.