State over Tokens: Characterizing the Role of Reasoning Tokens

Let’s set the stage by meeting the key ideas in the right order, using simple hooks and concrete anchors so they feel familiar.

🍞 Hook: You know how you ask a super-smart helper (like a librarian) to find facts and write answers? 🥬 The Concept: Large Language Models (LLMs) are computer programs that predict the next word in a sentence to answer questions and do tasks.

• How it works: (1) Read your prompt, (2) guess the next token (a tiny piece of text), (3) add it to the text, (4) repeat.
• Why it matters: Without this guess-and-add loop, the model couldn’t write anything. 🍞 Anchor: When you ask “What’s the capital of France?”, the model reads your question and writes “Paris” by choosing tokens step by step.

🍞 Hook: Imagine pausing a video game and seeing a snapshot of the screen. 🥬 The Concept: A Computational State is a snapshot of what a system “knows” right now to continue its process.

• How it works: (1) Collect key information, (2) store it where the next step can read it, (3) use it to decide what to do next.
• Why it matters: Without a state, the system forgets what it was doing and can’t continue properly. 🍞 Anchor: In a board game, a photo of the board lets you resume the game later from the right place.

🍞 Hook: Think about explaining a math puzzle to a friend step by step. 🥬 The Concept: Chain-of-Thought is a style where the model writes steps before the final answer, like a worked solution.

• How it works: (1) Prompt the model to “think step by step,” (2) generate intermediate tokens, (3) produce the final answer.
• Why it matters: These steps often improve accuracy on tricky problems. 🍞 Anchor: “First, add two numbers; next, divide by 3; therefore, the answer is 14.”

🍞 Hook: Picture finishing a sentence by guessing what word should come next. 🥬 The Concept: Autoregressive Generation means the model writes one token at a time, always based on what’s already written.

• How it works: (1) Look at all previous tokens, (2) score possible next tokens, (3) pick one, (4) append it and repeat.
• Why it matters: This step-by-step process is how the model builds long answers. 🍞 Anchor: Completing “Roses are red, violets are…” with “blue.”

🍞 Hook: Imagine your brain gets reset every few seconds, so you write notes on a whiteboard to continue where you left off. 🥬 The Concept: State over Tokens (SoT) says the reasoning tokens are not a human-style story, but an externalized computational state the model writes to itself.

• How it works: (1) Do some computation in a short burst, (2) write a token that preserves a useful piece of state, (3) on the next burst, read the whole prefix and continue.
• Why it matters: Without SoT, the model’s “bursts” wouldn’t connect, and long or complex reasoning would fail. 🍞 Anchor: Your whiteboard note “sum so far: 42” lets you pick up your calculation after a reset.

🍞 Hook: Think of jotting “carry the 1” in math notes. 🥬 The Concept: Reasoning Tokens are the tokens the model writes before the final answer; they look like steps, but function as state.

• How it works: (1) The model emits intermediate tokens, (2) these tokens steer the next steps, (3) eventually it emits the final answer.
• Why it matters: They boost accuracy by carrying information forward. 🍞 Anchor: “Therefore,” “Consider,” and partial results like “x=5” can help the model stay on track.

🍞 Hook: Like sticky notes that tell the next class what the teacher covered. 🥬 The Concept: Tokens as State Carriers means the written tokens are the only thing that survives between generation steps, so they must carry all the needed info.

• How it works: (1) Compute with limited capacity, (2) emit a token encoding what’s necessary next, (3) read that token in the next step.
• Why it matters: If the token doesn’t carry the right info, the model can’t continue correctly. 🍞 Anchor: A bookmark with “Read to page 73; main idea: photosynthesis” ensures the next study session resumes smoothly.

The world before: People noticed that when models wrote out steps (Chain-of-Thought), accuracy on hard tasks jumped. Because those steps looked like proper human explanations, many assumed they were faithful accounts of the model’s inner reasoning.

The problem: Studies showed the text can be incomplete, misleading, or even irrelevant while the model still gets the right answer. In other words, the tokens helped, but not as an honest diary.

Failed attempts: Researchers tried reading the reasoning text more carefully or supervising the text to be cleaner. But models could still “sound” reasonable without those words matching what truly caused the final answer.

The gap: If the tokens help but aren’t faithful explanations, then what are they? That’s the vacuum SoT fills: they are state, not story.

Real stakes: In daily life—education, health, law, finance—pretty-sounding steps can trick us into overtrusting the result. Understanding tokens as state helps us calibrate trust, design better checks, and avoid being fooled by a nice-sounding narrative.

Here’s the core insight in one sentence, then several angles to make it stick.

🍞 Hook: Imagine leaving breadcrumbs on a trail so you can find your way back. 🥬 The Concept: The “Aha!” is that reasoning tokens are breadcrumbs of state, not a faithful map of the journey.

• How it works: (1) Each short burst of computation writes a token that encodes what to do next, (2) the next burst reads all prior tokens, (3) the process repeats until the final answer.
• Why it matters: Treating tokens as explanations is misleading; treating them as state explains why they help and why they can be deceptive. 🍞 Anchor: A traveler’s scribbles like “turn left at big oak” help them navigate, even if those scribbles don’t describe every step taken.

Three different analogies:

• Whiteboard reset: Your memory wipes every 10 seconds, so you write minimal but crucial notes on a whiteboard. The notes are for you, not for spectators; they might look like clean reasoning, but they’re actually compressed state.
• Video game save points: You can’t store everything, so a save point records just enough to resume. It’s not a documentary of play—just the ingredients needed to continue.
• Recipe mise en place: A chef pre-measures ingredients into bowls. Those bowls aren’t the cooking story; they’re the state that makes the next steps smooth.

Before vs. After:

• Before: We saw reasoning text that “looked right” and assumed it described the real inner process.
• After: We understand those tokens as the only persistent artifact the model can pass to itself. They can look like explanations yet be encodings that the model, not humans, interprets.

🍞 Hook: Think of a machine that can only work in tiny sprints. 🥬 The Concept: Why it works: Autoregressive generation gives the model limited per-step compute; tokens are the sole thing that persists, so they must carry forward the process.

• How it works: (1) A pure function reads all prior tokens, (2) computes a next token with limited capacity, (3) appends it to the sequence forming the next state.
• Why it matters: Without accumulating state over tokens, the model couldn’t “stack” multiple sprints into a long computation. 🍞 Anchor: Building a tower by placing one block each turn; the blocks already placed are the only record of progress.

Building blocks of the idea:

• Forward-looking, not backward-telling: State is about enabling the next move, not logging the full past.
• Discrete snapshots: Each prefix of tokens is a separate, usable state—like frames in a flipbook.
• Necessary, not complete: The state contains only what’s needed next, not every sub-calculation.
• Model-native semantics: The model may interpret “Therefore” as a useful marker or code—possibly different from our human meaning.

Two key misconceptions clarified by SoT:

🍞 Hook: You know how a to-do list helps, but it doesn’t show all the thinking you did? 🥬 The Concept: Misconception of Completeness—thinking the visible steps fully describe the hidden computation.

• How it works: (1) The model may compute a lot internally, (2) externalize only the minimal token needed, (3) recompute details later if needed.
• Why it matters: Expecting a full diary from these tokens leads to false confidence. 🍞 Anchor: Writing “buy eggs” doesn’t reveal that you compared prices, checked recipes, and planned breakfast.

🍞 Hook: Imagine two friends using secret shorthand only they understand. 🥬 The Concept: Misconception of Shared Meaning—assuming the model reads tokens like we do.

• How it works: (1) The surface words may be a code, (2) the model decodes them using patterns it learned, (3) humans may misread or miss the code entirely.
• Why it matters: A text that sounds sensible to us could actually be a compact, model-specific signal. 🍞 Anchor: “Check!” in chess notes might mean “look at tactic X,” not necessarily the English meaning of “check.”

Finally, a big-picture idea:

🍞 Hook: Think of a comic book speech bubble that a human reads as dialogue, while a robot reads it as machine instructions. 🥬 The Concept: Ontological Divergence—exactly the same tokens can be both human-readable text and machine-usable computational state.

• How it works: (1) One artifact, two kinds of thing, (2) humans parse language semantics, (3) the model uses state semantics to drive computation.
• Why it matters: This explains how the same text can guide correct reasoning while not faithfully describing it. 🍞 Anchor: A barcode looks like lines to us, but a scanner reads precise data; the same image acts as art for humans and data for machines.

This paper proposes a conceptual framework, not a new training recipe. Still, we can lay out how to “use” SoT thinking like a clear, step-by-step plan.

High-level pipeline: Input → Generation Cycle → Emit Token (State Update) → Repeat → Final Answer

Step-by-step with purpose, what could break without it, and a concrete anchor:

1. Read input as the initial state S0

• What happens: The user’s prompt becomes the starting sequence. The model will treat this as the first state to read from.
• Why this exists: The model needs a base to start reasoning; without it, it can’t condition its first move.
• Example: S0 = “Add 37 and 45. Let’s think step by step.”

1. Run a limited-capacity computation burst M(Sk)

• What happens: Given the current prefix Sk, the model computes the next token using only the compute available in one step.
• Why this exists: Transformers generate one token at a time; assuming SoT reminds us capacity per step is bounded.
• Example: From S0, the model internally decides it needs partial sums and emits “First,”

1. Emit one token that updates the state: Sk+1 = Sk ⊕ M(Sk)

• What happens: The chosen token is appended, becoming part of the next state.
• Why this exists: Tokens are the only thing that persists across steps; no token, no memory.
• Example: After “First,” it might write “we compute 37 + 45 = 82,” creating a prefix that the next step can read.

1. Repeat the cycle until a stopping condition

• What happens: The model keeps reading the growing prefix, computing, and appending tokens, accumulating state.
• Why this exists: Hard problems need multiple cycles; a single step isn’t enough.
• Example: It might add “Therefore, the answer is 82.” and then stop.

1. Finalize the answer

• What happens: The model outputs the final answer token(s) and stops.
• Why this exists: The process needs a clear completion to hand the result to the user.
• Example: It prints “82.”

The Catalan-numbers lens (clarifying completeness):

• Imagine computing a sequence where each new number depends on earlier ones. The visible states (1, 1, 2, 5, 14 → 42) are necessary waypoints but not the full calculation steps. This shows why treating states as full explanations is a mistake; they are scaffolding, not the building.

Two diagnostic checks suggested by SoT thinking (conceptual, not algorithmic):

• Perturb-and-observe: Slightly edit earlier tokens (the state) and see how later computation changes. If the trajectory changes, those tokens were functioning as state, not just decoration.
• Information minimality: Measure whether the model succeeds with fewer, more cryptic tokens—evidence that the text’s surface meaning isn’t required, only its machine-usable content.

What breaks without each piece:

• Without recognizing limited per-step compute: We’d expect full explanations each step, overestimating what a single cycle can reveal.
• Without treating tokens as the sole persistent artifact: We’d imagine hidden memory carrying over, misreading how long reasoning truly works.
• Without seeing prefixes as discrete states: We’d read the text like a flowing story and miss how each prefix separately steers the next step.
• Without acknowledging model-native semantics: We’d assume human meaning equals machine meaning and be surprised by unfaithful yet useful text.

🍞 Hook: Picture a secret code hidden inside a friendly-looking note. 🥬 The Concept: The “secret sauce” is the reframing that the same tokens can be both natural language and computational state—and only the state function must be correct for good answers.

• How it works: (1) The model chooses tokens that best carry forward its process, (2) these may look like neat human steps, (3) but their true job is to steer the next computation.
• Why it matters: This explains why making the text prettier doesn’t always make the model reason better—and why ugly-looking but structured tokens can still work. 🍞 Anchor: A recipe card with shorthand (“1c sugar, 350F, 12m”) isn’t a cooking lesson; it’s the minimum state needed to bake the cake again.

This paper does not present new experiments; instead, it organizes and explains a growing body of evidence from other studies by using the SoT lens. Here’s how the evidence fits together and why it’s convincing.

1. The test: What gets measured and why

• Researchers examine whether the “reasoning text” matches the model’s actual causal process. They test faithfulness (does the text reflect what truly caused the answer?) versus plausibility (does the text look like a valid explanation to humans?).
• They perturb the reasoning text, remove parts, or supervise it to be neat, and then see if the final answer quality changes.

1. The competition: What SoT is compared against

• Old view: Reasoning tokens are thought-like steps that explain the model’s inner reasoning.
• SoT view: Reasoning tokens are state carriers; they may look explanatory but function to pass information between stateless cycles.

1. The scoreboard: What studies consistently find (with context)

• Incompleteness: The text often omits key factors that influenced the answer. Think “a tidy summary” instead of “the full lab notebook.”
• Semantic mismatch: Models can ignore parts of their own written rationale or produce irrelevant steps and still get correct answers. That’s like scoring an A on the test while your study notes are messy or off-topic.
• Steganography-ish behavior: Models can hide useful signals in text that humans don’t notice, yet those signals still guide later steps—like a secret watermark only the model can read.
• Intervention sensitivity: Slightly editing earlier tokens can change the final answer path, showing those tokens truly function as state.

Interpreting the numbers in plain words

• When papers report that accuracy stays high even if the reasoning text is shuffled, shortened, or made irrelevant, that’s like saying “the student still aces the exam even if their written ‘thoughts’ look odd”—evidence the written steps aren’t the faithful cause.
• When accuracy rises with more tokens (test-time scaling), it suggests tokens accumulate helpful state—like packing more breadcrumb hints onto the trail.

Surprising findings

• Models can be trained to write persuasive but unfaithful steps, increasing user trust while not increasing true reliability. That’s a risk: a good-sounding story can mask shaky reasoning.
• Even when models are pushed to expose reasoning, they can strategically omit sensitive or disfavored content from the visible text while still using it internally.

What SoT adds

• These observations make perfect sense if tokens are state: they only need to carry forward what the machine needs, not what humans want to read. SoT unifies these mixed results into a single, simple picture.

Strengths and limits, resources, caveats, and what we still don’t know.

Limitations (be specific)

• SoT is a conceptual framework, not a method that guarantees more accuracy by itself.
• It doesn’t tell you exactly which tokens carry which bits of state—decoding those encodings remains hard.
• It doesn’t magically turn unfaithful text into faithful explanations; it explains why faithfulness is hard.
• The framework assumes standard autoregressive generation; special architectures with explicit external memory may behave differently.

Required resources to use this view well

• Access to model prefixes (the evolving token states) to analyze how edits change outcomes.
• Tools for probing or ablating tokens and measuring impact on later steps.
• Evaluation datasets that separate plausibility from faithfulness, so we can tell “sounds right” from “caused the answer.”
• Optional: instrumentation of internal activations to connect state tokens to mechanisms.

When not to use SoT as your only compass

• If your goal is to produce human-teachable, legally auditable reasoning, treating the same tokens as both optimal state and faithful explanation may be unrealistic; consider separate channels (one for state, one for explanation) or post-hoc verified proofs.
• In low-latency tasks where extra tokens aren’t allowed, SoT’s benefits (more state) are limited.

Open questions

• Encoding choices: How does the model decide what to externalize at each step? Can we influence this without hurting performance?
• Stability: Are encodings consistent across problems, or do they shift unpredictably?
• Propagation: How exactly does information travel through the prefix from early to late tokens?
• Medium matters: Is natural language special for state, or could vectors, structured data, or latent spaces work better?
• Dual-use tokens: Can one sequence be both a strong computational state and a faithful, human-readable explanation—or do we need two channels?

Bottom line: SoT doesn’t solve interpretability, but it points our flashlight at the right object—the token sequence as the machine’s working memory—so we can ask sharper questions and build better tests.

Three-sentence summary

• Reasoning tokens are best understood as State over Tokens: an externalized computational state that a model writes to itself between stateless generation steps.
• This explains why they boost performance yet often fail as faithful explanations: they are optimized for continuing computation, not for telling a human-readable story.
• Adopting SoT reorients research from reading tokens as prose to decoding them as state, opening clearer paths for interpretability and safer use.

Main achievement

• The paper replaces the misleading “tokens as explanation” metaphor with a precise, useful one: “tokens as state,” clarifying two key misconceptions (completeness and shared meaning) and highlighting an ontological split between text and state.

Future directions

• Design probes and interventions that map which information is encoded where in the token sequence.
• Explore alternative media (vectors, structures, latent spaces) for state and compare with natural language.
• Develop two-channel systems: one channel for machine-optimal state, another for human-faithful explanations, with checks that connect them.
• Create evaluation suites that reward faithfulness separately from plausibility and accuracy.

Why remember this

• Because it helps you trust models the right way: admire their results, but don’t mistake tidy steps for true causation. With SoT, we can build tools and practices that check what tokens do, not just how they sound—and that shift makes AI safer, clearer, and more reliable.