NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation

🍞 Hook: Imagine you’re building a comic book. First you write the words, then you draw pictures, and sometimes you go back and forth—write a bit, draw a frame, write more, draw more. Wouldn’t it be great if one smart helper could do both jobs smoothly in one go?

🥬 The Concept (The world before): AI had two strong but separate helpers. Large Language Models (LLMs) were great at text (thinking, explaining, and following instructions), while Diffusion Models were great at drawing realistic images. But they didn’t live in the same house: they used different tools and spoke different “token” languages. That made doing mixed tasks—like reasoning about a prompt, then drawing, then adding captions—clunky and slow.

🍞 Anchor: If you asked for a short story with pictures between paragraphs, older systems often needed to pass work back and forth between different models, wasting time and sometimes losing the thread.

🍞 Hook: You know how you read a page from left to right, word by word? Some older image AIs tried to “read” images the same way: pixel by pixel, line by line.

🥬 The Concept (The problem): Pure autoregressive (AR) image generation that predicts the next token in raster-scan order (like pixels in a long line) becomes painfully slow as images get larger. A 1024×1024 image could take more than 10 minutes. Also, many visual tokenizers focused only on reconstructing pixels, not on packing high-level meaning into tokens. That made it hard for one model to both understand images and generate them well.

🍞 Anchor: It’s like trying to paint a mural with a needle and also storing paintings using only tiny color dots—fast meaning and big structure get lost.

🍞 Hook: Teams tried to glue two worlds together—text models and image models—like taping two different puzzle boxes to make one super-puzzle.

🥬 The Concept (Failed attempts): Hybrid AR+diffusion systems could generate well, but used two separate representations. That meant extra re-encoding steps, extra complexity, and weaker deep integration. Pure AR systems stayed unified but were too slow at high resolution and missed semantic richness.

🍞 Anchor: Like switching languages mid-sentence—translation slows you down and you may lose subtle meaning.

🍞 Hook: Think of how you draw: first big shapes, then medium parts, then tiny details.

🥬 The Concept (The gap): We needed a single model that could do text and images together efficiently, understand high-level ideas, and still render details quickly—without juggling multiple systems.

🍞 Anchor: The missing piece was a way to generate images in big-to-small steps and to use tokens that capture both meaning and detail.

🍞 Hook: Imagine a camera that can zoom smoothly between a map view and street-level details.

🥬 The Concept (NextFlow’s answer): NextFlow is a unified, decoder-only Transformer trained on about 6 trillion interleaved tokens of text and images. It replaces slow pixel-order generation with next-scale prediction (big shapes to fine details), and uses a dual-codebook tokenizer so visual tokens carry both meaning (semantics) and crisp detail. It adds training stabilizers (scale reweighting, self-correction) and a reinforcement learning step that focuses on early, global structure with prefix-tuning.

🍞 Anchor: The result is a single system that can write, draw, edit, and reason in one continuous flow, generating 1024×1024 images in around 5 seconds—fast enough for interactive use.

🍞 Hook: You know how a good artist lightly sketches big shapes first, then refines details? They don’t paint every pixel from top-left to bottom-right.

🥬 The Concept (Aha! in one sentence): NextFlow makes a single model that treats text and images as one sequence, but draws images scale-by-scale (big to small) instead of pixel-by-pixel, using visual tokens that carry both meaning and detail.

How it works (intuition):

1. Use a dual-codebook tokenizer so image tokens capture semantics (what’s there) and pixels (how it looks).
2. Generate images with next-scale prediction: predict coarse layout first, then add finer scales.
3. Stabilize training so early scales (the layout) get enough attention via scale reweighting and self-correction.
4. Use reinforcement learning with prefix-tuning to improve prompt following by updating mainly the early, layout-deciding steps.

Why it matters: Without this, unified AR models are too slow at high resolution and miss semantic richness; with it, we get fast, coherent, and editable images tightly connected to text reasoning.

🍞 Anchor: It’s like outlining a dinosaur first (big shapes), then adding muscles (medium), then scales and skin texture (fine)—all while explaining what species it is in the same conversation.

Multiple analogies for the same idea:

• City planning: Lay roads (coarse), build blocks (medium), add street signs and flowers (fine).
• Baking a cake: Mix the base batter (coarse), bake layers (medium), frost and add sprinkles (fine).
• Map zoom: Start with a country map (coarse), zoom to city blocks (medium), zoom to specific house numbers (fine).

Building blocks (each introduced with a mini sandwich):

• 🍞 Hook: Imagine a library that stores both the meaning of a book and how the cover looks. 🥬 Dual-Codebook Tokenization: A tokenizer that turns images into two kinds of codes: semantic (meaning) and pixel (appearance). It looks up both and keeps them aligned so tokens are rich in meaning and sharp in detail. Without it, the model struggles to understand images deeply or draw crisply. 🍞 Anchor: A token can tell the model “this is a ‘red apple on a table’” and also “it’s glossy with a hard highlight at this spot.”

• 🍞 Hook: Don’t start coloring every pixel when you haven’t decided where the mountain and the river go. 🥬 Next-Scale Prediction: Generate the image in stages—from big layout to tiny details—rather than pixel order. Predict the next scale’s tokens conditioned on previous coarser scales. Without it, generation at 1024×1024 becomes too slow for practical use. 🍞 Anchor: First a 2×2 grid roughs in sky/ground, then 4×4 adds mountain shapes, and so on, until fine textures appear.

• 🍞 Hook: When building a house, the foundation matters more than the paint. 🥬 Scale Reweighting: During training, give more weight to errors at coarse scales (which decide layout) so the model learns structure well. Without it, fine-detail tokens dominate and layouts degrade. 🍞 Anchor: If early scales are ignored, you get pretty textures in the wrong places.

• 🍞 Hook: If you practice only with perfect puzzle pieces, a tiny mistake on test day can throw you off. 🥬 Self-Correction with Residual Features: During training, slightly perturb codebook lookups so the model learns to fix earlier small mistakes; use residual (per-scale) features rather than accumulated ones to keep inputs simple. Without it, small early errors snowball. 🍞 Anchor: It’s like learning to steer back on course after a small wobble instead of panicking.

• 🍞 Hook: When coaching a sports team, you first train opening moves before finessing finishing touches. 🥬 Prefix-Tuning for RL: Use GRPO reinforcement learning but only update the first m scales (“the prefix”). This focuses the noisy RL signal on the most impactful steps. Without it, learning gets dominated by late, detail steps and high-level alignment suffers. 🍞 Anchor: The model gets much better at following the big idea of your prompt (pose, composition, count) before polishing textures.

Before vs. After:

• Before: Unified AR was slow at high resolution and images lacked semantic density.
• After: NextFlow is fast (≈5s at 1K) and uses rich tokens, so it understands prompts better and draws cleaner.

Why it works (no equations, just logic):

• Coarse-to-fine halves the explosion of token steps, so it’s fast.
• Two codebooks compress both ‘what’ and ‘how,’ so the model can reason and render.
• Training focuses on where structure is decided and practices recovery from small errors.
• RL refines the global picture rather than getting lost in late-stage noise.

High-level recipe: Input (interleaved text and image tokens) → Tokenize (dual-codebook + multi-scale) → Decoder-only Transformer (next-scale prediction with 3D positions) → Optional RL prefix-tuning → Output images/text (plus optional diffusion refinement)

Step A: Dual-Codebook Tokenization

• 🍞 Hook: You know how a museum tag tells you both what a painting shows and how it was made?
• 🥬 What it is: A tokenizer that produces semantic tokens (meaning) and pixel tokens (appearance) for images, across multiple scales. How it works:
  1. A semantic encoder (initialized from a strong vision-language model) extracts high-level features.
  2. A pixel CNN branch captures fine textures.
  3. Both are quantized into codebooks; lookup chooses entries by a weighted meaning+pixel distance.
  4. Multi-scale VQ emits coarse-to-fine tokens; dynamic resolution support keeps coordinates consistent. Why it matters: Without semantic-rich tokens, the model can’t reason well about images; without pixel tokens, details vanish.
• 🍞 Anchor: A token might say “this patch is ‘fur on a cat’s cheek’ and it’s smooth, light-brown, with a soft highlight.”

Step B: Unified Decoder-Only Transformer with Next-Scale Prediction

• 🍞 Hook: A storyteller who can pause to draw a panel before continuing the tale.
• 🥬 What it is: One decoder-only model that predicts both text and visual tokens; text uses next-token, images use next-scale. How it works:
  1. One shared output head for simplicity and stronger sharing.
  2. Visual tokens predicted per scale; each scale’s grid is sampled in parallel at that level.
  3. Cross-entropy loss over a unified vocabulary. Why it matters: Without a single head and unified sequence, interleaved tasks need costly handoffs and lose coherence.
• 🍞 Anchor: In a single chat, the model can write a sentence, then output the coarse image grid, then refine it, then keep writing.

Step C: Multiscale 3D Positional Encoding (3D RoPE)

• 🍞 Hook: Imagine labels on a map that mark left-right (x), up-down (y), and zoom level (scale).
• 🥬 What it is: A position system that encodes (x, y, scale) for image tokens and (t, t, t) for text tokens. How it works:
  1. Normalize spatial coords so different resolutions share the same range.
  2. Add learnable scale embeddings and a sinusoid over the number of scales (scale length) to signal target resolution. Why it matters: Without 3D positions and scale length, the model gets confused across resolutions and scales.
• 🍞 Anchor: A 16×16 grid and a 32×32 grid both map to the same [0, C] range, so no surprises when upscaling.

Step D: Scale Reweighting

• 🍞 Hook: If the skeleton isn’t right, the costume won’t fix it.
• 🥬 What it is: A training loss trick that boosts early scales with fewer tokens. How it works:
  1. Compute a weight per scale based on its grid size, with an exponent α (≈0.9 in practice).
  2. Keep total vision loss constant but redistribute across scales. Why it matters: Without it, fine scales dominate—layouts drift and artifacts appear.
• 🍞 Anchor: After reweighting, mountains stay where they should; details decorate, not dictate.

Step E: Self-Correction with Residual Features

• 🍞 Hook: Practice recovering from small slips makes you steadier.
• 🥬 What it is: Train-time noise that teaches the model to fix earlier missteps. How it works:
  1. During encoding, sample among top-k nearest codebook entries instead of always taking the closest.
  2. Use per-scale residual features (not accumulated features) as inputs, keeping text/vision feature spaces aligned. Why it matters: Without this, tiny early errors amplify; with accumulated features, inputs got too messy.
• 🍞 Anchor: The model learns to nudge a slightly crooked roof back into place at the next scale.

Step F: Reinforcement Learning with Prefix-Tuning (GRPO)

• 🍞 Hook: Teach the opening moves before polishing endgame flair.
• 🥬 What it is: Use Group Reward Policy Optimization but only update coarse scales (the prefix). How it works:
  1. Sample candidate generations in groups and compute normalized advantages per group.
  2. Apply clipped policy updates with scale weights, but freeze later fine-scale policies.
  3. Rewards favor prompt following, alignment, and structure. Why it matters: Without prefix focus, noisy signals in late scales overpower learning of global composition.
• 🍞 Anchor: After RL, the model nails counts, positions, and relations (e.g., “three cups on the left, one on the right”).

Step G: Optional Diffusion Decoder

• 🍞 Hook: After you draw a great comic panel, a finisher inks tiny hairs and fabric weave.
• 🥬 What it is: A refinement module that receives semantic+pixel embeddings and decoded semantic features to enhance details. How it works:
  1. Concatenate visual conditions, project, and feed into a diffusion model; caption goes through text branch.
  2. Use when hyper-real detail (small faces, small text) matters; otherwise leave off to keep exact spatial edits. Why it matters: VQ is efficient but drops some high frequencies; diffusion can put them back.
• 🍞 Anchor: For a tiny sign in the background, the diffusion decoder cleans up the letters.

Concrete mini-example (data flow):

• Input: “A yellow bus in front of a red barn.”
• Tokenize text as usual; tokenize reference images (if any) into semantic+pixel tokens across scales.
• The model writes: text tokens → image scale 1 (rough: yellow blob left, red block right) → scale 2 (bus rectangle, barn roof) → … → fine scales (windows, wheels, wood planks) → optional final caption.
• If needed, run diffusion refinement to sharpen small window frames and barn textures.

🍞 Hook: Think of a school decathlon where one student must be fast, accurate, and creative in many events.

🥬 The Concept (The tests): NextFlow was tested on how well it follows prompts (GenEval, DPG), knows the world (WISE), creates with style and long text (PRISM), edits images (ImgEdit, GEdit-Bench, OmniContext), and handles interleaved text-image storytelling. It also measured how fast it can generate big images and how well its tokenizer reconstructs images.

Why it matters: A unified model should do many things well, not just one.

🍞 Anchor: It’s like scoring high in math, art, and writing—versatility counts.

The competition: Strong diffusion baselines (like SD3, FLUX.1-dev, GPT Image) and other unified AR systems (Janus-Pro, Emu3.5, Bagel, etc.).

Scoreboard with context:

• Speed: NextFlow generates 1024×1024 images in about 5 seconds and uses roughly 6× fewer FLOPs than certain diffusion Transformers at the same resolution—this is the difference between “interactive” and “go make a sandwich.”
• Prompt following (GenEval): NextFlow reaches ≈0.84 with RL, comparable to top-tier systems; that’s like getting an A when class averages clustered around B.
• Detailed prompt coverage (DPG): NextFlow RL hits ≈88+, matching or rivaling strong baselines—think “near-top of the leaderboard.”
• World knowledge (WISE): NextFlow RL gets ≈0.62, similar to best unified systems; it handles cultural/time/space/physics/chemistry better than prior AR-only baselines.
• Aesthetics and long prompts (PRISM): NextFlow RL ≈78.8 overall—competitive with the best and showing strong style and composition.
• Editing (ImgEdit): NextFlow RL ≈4.49 overall—outperforming strong baselines on many sub-tasks, especially Adjust and Remove. On GEdit-Bench, it achieves ≈7.87 overall (great balance of meaning + visual quality). On OmniContext single-subject, SC ≈9.22 (identity preserved very well).
• Tokenizer reconstruction: At 1024, PSNR ≈28 dB—meaning the compressed tokens preserve enough detail for high-quality generations.

Surprising findings:

• Self-correction only helped when using per-scale residual features; with accumulated features it hurt—simpler inputs made recovery easier.
• Mixing 25% text-only data didn’t harm image generation quality during small-scale tests.
• A single shared output head matched or beat dual heads, keeping the architecture neat.

Qualitative demos:

• Interleaved storytelling with images between paragraphs looks coherent.
• In-context editing: The model learns edit patterns from examples and applies them to a new image.
• Chain-of-thought before drawing: Reasoning first improved scores on a controlled WISE test from ≈0.60 to ≈0.70—thinking helped drawing.

Bottom line: NextFlow achieves state-of-the-art or near state-of-the-art among unified models while staying fast enough for real use.

🍞 Hook: Even superheroes have weaknesses and need the right tools.

🥬 Limitations:

• Vector quantization is discrete, so some ultra-fine details are inevitably compressed; that’s why an optional diffusion finisher can help—but it may slightly change tiny structures, which matters for strict local edits.
• Sharing one set of parameters for both text and vision in a 7B model creates a capacity squeeze; balancing both skills is a careful dance.
• Training stability needs the full recipe (scale reweighting, self-correction, 3D RoPE); skipping steps hurts quality, especially at high resolution.

Required resources:

• Large-scale compute (thousands of GPUs) and carefully packed data pipelines to train at multi-trillion-token scale.
• Well-filtered, diverse multimodal datasets (including interleaved video frames for narrative continuity).

When not to use:

• If you require pixel-perfect identity preservation on tiny regions (e.g., micro-text in a sign) and you plan to use the diffusion decoder, be cautious—it can subtly alter fine structure.
• If your task is purely text and extremely long-context, a text-only LLM might be more efficient.

Open questions:

• Can we invent tokenizers with even higher compression and semantic density so sequences get shorter yet richer?
• How far can unified RL (across text and images) push reasoning-before-drawing in real-time settings?
• What’s the best way to scale to MoE architectures while keeping training stable and inference efficient?

🍞 Anchor: Think of NextFlow as a strong all-round athlete today, with clear training plans to become an even better decathlete tomorrow.

Three-sentence summary: NextFlow is a single decoder-only Transformer that treats text and images as one sequence, generating images scale-by-scale with semantically rich visual tokens. A careful training recipe (scale reweighting, self-correction, 3D RoPE) plus reinforcement learning with prefix-tuning makes the model fast, stable, and obedient to prompts. It rivals top systems on generation and editing while enabling interleaved reasoning and creation in one place.

Main achievement: Showing that a unified autoregressive model can be both fast at high resolution (≈5 seconds at 1024×1024) and high quality, thanks to next-scale prediction and dual-codebook tokenization.

Future directions:

• Scale up high-quality multimodal understanding data and native chain-of-thought to deepen reasoning.
• Explore Mixture-of-Experts for capacity without losing efficiency.
• Invent next-gen tokenizers (variable-rate, semantic-aware) to further shorten sequences.

Why remember this: NextFlow turns multimodal creation into a single smooth flow—think first, then draw—helping AI move from separate tools to a unified, versatile creator that can read, reason, and render in one conversation.