ERNIE 5.0 Technical Report

🍞 Hook: Imagine a school talent show where one student can sing, dance, draw, and tell stories—and can switch smoothly between them without getting confused. Wouldn't that be amazing?

🥬 The Concept (Unified Autoregressive Model): It is a way to make a model learn to handle text, images, video, and audio by always predicting the next bit based on what came before. How it works:

1. Turn everything (words, pixels, sounds) into tokens, like Lego bricks.
2. Line them up into a single sequence.
3. Predict the next token (or the next small group), over and over. Why it matters: Without one shared way to learn, the model needs separate parts for each skill, which don’t share knowledge well and often fight each other. 🍞 Anchor: When you ask for “a blue cat playing piano” (text) and then want an image and a little tune to match, the same brain can follow the request step-by-step across all outputs.

The world before: Most multimodal AIs glued a language brain to extra modules for images or audio. Understanding often happened in one place (the language core), while making images or sounds happened in separate generators. This late-fusion setup worked, but it made the model feel like a group project where each teammate turns in a separate page. The pages didn’t always fit together, and improving one part sometimes hurt another (the “ability seesaw”).

🍞 Hook: You know how playing on a soccer team teaches you skills that help in basketball too—like passing and spacing? Sharing is powerful.

🥬 The Concept (Cross-Modal Parameter Sharing): It lets different kinds of data (text, image, audio, video) learn from shared parameters so they help each other get smarter. How it works:

1. Put all modalities into one token space.
2. Train them together so the same parameters see many types of tokens.
3. Let patterns learned in one modality guide another (e.g., structure in text helps layout in images). Why it matters: Without sharing, each modality must reinvent the wheel, making learning slower and less general. 🍞 Anchor: Learning how “left, right, center” works in soccer helps you understand “margin, padding, alignment” when reading a web page screenshot.

The problem: As we add more skills (vision, audio, video), the model needs much more capacity. But if we make the whole network dense and huge, it becomes too slow and costly.

🍞 Hook: Think of a hospital with many doctors but only the few specialists you need see you—saves time and money.

🥬 The Concept (Ultra-Sparse Mixture-of-Experts, MoE): The model has many expert feed-forward blocks, but for each token it activates only a tiny fraction. How it works:

1. A router scores which experts fit a token.
2. Only top-k experts run.
3. Outputs are combined, keeping compute low. Why it matters: Without sparsity, trillions of parameters would be too slow and too expensive to use. 🍞 Anchor: A word about math routes to math experts; a pixel about faces routes to vision experts—fast and specialized.

Old attempts and their limits: Late fusion glued on image/audio decoders, which helped generation but weakened deep sharing; hand-made modality-specific routing in MoEs was fiddly and didn’t scale to 3+ modalities; training many model sizes separately was too expensive.

🍞 Hook: Imagine a traffic cop who doesn’t care if you drive a car, bus, or bike—only where you need to go.

🥬 The Concept (Modality-Agnostic Expert Routing): The router chooses experts based on the token’s content, not on its declared modality. How it works:

1. Tokens from any modality enter the same router.
2. The router sees the token’s features and picks experts.
3. Experts naturally specialize by task demands. Why it matters: Without it, you must guess how many experts each modality needs, which breaks as tasks change. 🍞 Anchor: A text token about music and an audio token about rhythm can share some of the same experts that understand timing.

The gap: Even with a great architecture, deploying one giant model in many places is hard—phones, edge devices, and servers all have different limits. Compressing after training takes extra work and often loses quality.

🍞 Hook: Like baking one big cake that can also be sliced into cupcakes for a picnic—without baking again.

🥬 The Concept (Elastic Training Paradigm): Train the big model and many smaller sub-models together in one run. How it works:

1. Randomly sample sub-models with fewer layers, fewer experts, or fewer active experts (smaller top-k) during training.
2. Optimize the full model and the sampled sub-model in the same step.
3. After training, pick the size you need without retraining. Why it matters: Without elastic training, you must compress separately for every size, wasting time and compute. 🍞 Anchor: One pre-training run yields a family: super-fast for a phone, balanced for a laptop, and full-power for a data center.

Real stakes for daily life: A unified, elastic model means your video editor can understand your spoken instructions, your photo app can reason about layouts, and your phone can run a small but smart version while the cloud runs a bigger one—all speaking the same “brain language.” It means fewer mismatches between understanding and generation: when the model reads a chart, it can also redraw it; when it hears a melody description, it can hum it back. And because only a sliver of experts run each time, it stays efficient enough to be practical.

Aha! Moment in one sentence: Treat everything (text, images, video, audio) as tokens in one line and predict the next group of tokens with a shared, sparse, and elastic brain that learns all skills together.

🍞 Hook: You know how telling a story works whether it’s words, a comic strip, or a song—each unfolds over time.

🥬 The Concept (Next-Group-of-Tokens Prediction): It’s one unified way of predicting the next pieces for every modality—words for text, scale-and-frame chunks for visuals, and codec layers for audio. How it works:

1. Convert any input into tokens.
2. Predict the next token or small group (grouping matches the modality’s structure).
3. Repeat until done, so understanding and generation share the same engine. Why it matters: Without one prediction goal, different modalities drift apart and don’t reinforce each other. 🍞 Anchor: Ask for “a short video of a red ball rolling while a drum beat plays”—the model advances frame-by-frame and beat-by-beat using the same turn-the-page mechanic.

Three analogies to explain the idea:

1. One music conductor: Different instruments (modalities) play from the same score (token sequence). The conductor (the model) cues each section in turn.
2. One recipe book: Whether you’re making cookies (text), pizza (image), or soup (audio), you still follow step-by-step instructions (next tokens).
3. One hiking trail: Hikers (modalities) walk the same path (sequence). The trail markers (positions) guide everyone, even if they walk at different speeds (group sizes).

Before vs After:

• Before: Separate decoders and objectives; improving images might hurt text; routing needed hand-tuned rules.
• After: One objective binds all modalities; a common router lets experts self-organize; learning in one place lifts all boats.

Why it works (intuition, no equations):

• Shared objective aligns the model’s inner language: everything is “what comes next,” so patterns transfer across skills.
• Sparse experts give capacity without cost: only the needed brains wake up, so we can afford a huge pool.
• Modality-agnostic routing discovers structure: experts naturally specialize by task (e.g., counting, layout, rhythm) rather than by rigid modality walls.
• Elastic training keeps every part useful alone: because sub-models practice during training, they stay competent when you later deploy them solo.

Building blocks (with sandwiches):

🍞 Hook: Imagine reading a poster: your eyes notice big titles and small fine print. 🥬 The Concept (Dual-Path Hybrid Representation): Use both CNN-like local detail and ViT-like global meaning before tokenization for better visual understanding. How it works:

1. Extract local (CNN) and global (ViT) features.
2. Use attention to merge patches smartly (not just by squishing with an MLP).
3. Compress to a compact set of visual tokens. Why it matters: Without this, fine details (like small text on charts) get lost and reasoning suffers. 🍞 Anchor: Reading a chart with tiny labels becomes easier when the model keeps both the big picture and the tiny text.

🍞 Hook: Drawing a picture: sketch the big shapes first, then add details. 🥬 The Concept (Next-Frame-and-Scale Prediction, NFSP): Generate images/videos from coarse scales to fine scales and from earlier frames to later ones. How it works:

1. Predict low-res tokens for a frame.
2. Add higher-res scales using what’s already drawn.
3. For video, move to the next frame with a temporal window. Why it matters: Without coarse-to-fine, long sequences pile up errors and blur details. 🍞 Anchor: First draw the outline of a cat, then whiskers, then fur texture—frame by frame.

🍞 Hook: Humming a tune first, then adding instruments. 🥬 The Concept (Next-Codec Prediction, NCP): Generate audio by predicting semantic code first, then residual codes for fine details layer by layer. How it works:

1. Tokenize audio into a semantic code + residual layers (RVQ).
2. Predict the top semantic code.
3. Feed it back, then predict the next residual layer; repeat. Why it matters: Without structured depth-wise steps, audio becomes too long and messy to predict well. 🍞 Anchor: Say the melody (la-la-la), then add drums, then add guitar.

🍞 Hook: One backpack that can shrink or stretch for different trips. 🥬 The Concept (Elastic Training Paradigm): Train the full model and many sub-models together by sampling different depths, widths, and sparsity. How it works:

1. Sometimes train full size; sometimes sample fewer layers or experts or smaller top-k.
2. Backprop once for both.
3. Later, choose the size that fits your device. Why it matters: Without elastic training, every size needs a separate, expensive training or compression pass. 🍞 Anchor: One training run produces a speedy phone-sized model and a powerhouse cloud model that think alike.

At a high level: Raw multimodal inputs → Tokenizers (text, vision, audio) → Unified sparse MoE transformer with modality-agnostic routing → Next-group-of-tokens prediction (understanding and generation) → Optional diffusion refiner for high-res visuals → Outputs (text, image/video, audio).

Step-by-step (with what/why/examples):

1. Tokenize everything

• What happens: Text is BPE/byte-tokenized; images/videos use a causal multi-scale visual tokenizer; audio uses a codec tokenizer with residual vector quantization (RVQ) where the first code captures semantics.
• Why this step exists: The model only understands tokens; choosing compact, meaningful tokens makes sequences shorter and learning steadier.
• Example: A 5-second clip becomes a stream of audio codes; a 256×256 image becomes multi-scale bit-codes.

🍞 Hook: Like printing a map with a grid and time stamps. 🥬 The Concept (Unified Spatiotemporal Rotary Position, Uni-RoPE): Give every token coordinates that work for text, images, and videos. How it works:

1. Text/audio use a single index (time).
2. Visual tokens use (frame t, height h, width w), center-aligned across scales.
3. The same positional math guides attention for all tokens. Why it matters: Without consistent positions, the model can’t align patches across scales or keep frame order straight. 🍞 Anchor: Knowing which comic panel (frame) and which bubble (position) a word belongs to.

1. Unified backbone with ultra-sparse MoE and modality-agnostic routing

• What happens: A shared transformer stack with MoE feed-forward layers routes each token to a tiny set of experts.
• Why this step exists: It gives huge capacity (many experts) while keeping compute low (few active per token) and sharing skills across modalities.
• Example: A math text token and a chart patch may both activate a “counting/layout” expert.

1. Visual understanding and generation

• What happens: For understanding, dual-path hybrid features (CNN + ViT) are merged with attention into compact tokens. For generation, NFSP predicts from coarse to fine within frames and over time for videos.
• Why this step exists: Understanding needs both big-picture and tiny details; generation needs to avoid error snowballs over long sequences.
• Example: Reading tiny chart labels (understanding) and drawing crisp text in an infographic (generation).

1. Audio understanding and generation

• What happens: For understanding, add embeddings from all RVQ levels to form each audio token. For generation, NCP predicts semantic code first, then residual codes layer-by-layer with feedback.
• Why this step exists: Hierarchical audio keeps sequences manageable and separates meaning from timbre/prosody.
• Example: Transcribing speech while also recognizing background sounds; then speaking the answer in a chosen voice.

1. Training recipe and long-context

• What happens: Train in stages (8K → 32K → 128K context) with careful learning-rate and loss balancing; multi-token prediction helps speed and quality; modality losses are reweighted to keep training fair.
• Why this step exists: Longer sequences appear gradually so optimization stays stable; each modality gets its fair share of learning.
• Example: Reading long PDFs or hour-long transcripts without re-tuning positions.

1. Elastic training

• What happens: During pre-training, sample sub-models:
  • Elastic depth: sometimes drop layers.
  • Elastic width: sometimes use a subset of experts.
  • Elastic sparsity: sometimes reduce top-k routing.
• Why this step exists: So multiple deployable sizes are born inside one training run.
• Example: Later, you deploy a 60%-size model to an edge device without retraining.

1. Post-training with stable reinforcement learning

• What happens: After supervised fine-tuning, the model learns with RL using several stability tricks.

🍞 Hook: Like assembling a fair line at a busy water slide, so no one gets skipped and the flow never jams. 🥬 The Concept (Unbiased Replay Buffer, U-RB): Keeps rollouts efficient and unbiased even when some are much longer. How it works:

1. Make a big inference pool and a training pool.
2. While long samples finish, prepare future batches without changing the data order for the current training step.
3. Move only the assigned group to training when complete. Why it matters: Without it, GPUs idle or the model sees too many easy/short examples first, harming learning. 🍞 Anchor: Long math problems no longer block the whole batch, and the difficulty mix stays fair.

🍞 Hook: Like using guardrails so a skateboarder explores tricks without crashing. 🥬 The Concept (MISC – Multi-granularity Importance Sampling Clipping): Calibrates how much to trust off-policy rollouts at multiple levels to avoid entropy collapse. How it works:

1. Compare training and inference policies moment-by-moment.
2. Clip importance weights within safe bounds.
3. Balance exploration (try new paths) and exploitation (keep good paths). Why it matters: Without it, the model locks into narrow, overconfident answers too early. 🍞 Anchor: The model keeps trying different reasoning steps instead of repeating the same shortcut.

🍞 Hook: When a level in a game is already easy for you, you don’t need to grind it forever. 🥬 The Concept (WPSM – Well-learned Positive Sample Mask): Skips over-optimizing already-mastered rollouts to focus on hard ones. How it works:

1. Track each query’s success and policy entropy.
2. If it’s easy and stable, reduce its training weight.
3. Spend gradients on tough, low-reward cases. Why it matters: Without it, the model wastes time on easy wins and stops improving on hard tasks. 🍞 Anchor: Stop practicing the alphabet song; start practicing spelling bees.

🍞 Hook: Like getting a tiny hint for a tough puzzle, then fewer hints as you get better. 🥬 The Concept (AHRL – Adaptive Hint-based RL): Adds partial “think” skeletons to hard queries, then fades them out as learning progresses. How it works:

1. Attach a small fraction of reasoning steps to the prompt.
2. Anneal the fraction based on progress.
3. Transition to full self-reasoning. Why it matters: Without hints, zero-reward zones stall learning on the hardest problems. 🍞 Anchor: A geometry problem starts with “try pairing opposite vertices,” and later you solve it unaided.

1. Infrastructure helpers

🍞 Hook: Imagine a mask-making machine that instantly cuts any pattern you ask for. 🥬 The Concept (FlashMask): A super-fast attention masking system that handles different masks per sample. How it works:

1. Compiles flexible mask patterns efficiently.
2. Works well with long-context and parallelism.
3. Speeds training and keeps kernels consistent. Why it matters: Without fast, flexible masks, multimodal batches slow to a crawl. 🍞 Anchor: Some tokens see forward only (text), some see neighbors both ways (images), and FlashMask makes that easy.

Secret sauces that make it clever:

• One prediction game unites all modalities, so sharing is natural.
• Sparse experts scale capacity without scaling cost.
• Elastic training produces a whole zoo of deployable sizes in one go.
• RL stability tools (U-RB, MISC, WPSM, AHRL) keep learning fair, fast, and creative, even for long, hard, multimodal tasks.

Optional high-res visual refiner

• What happens: A separate diffusion refiner polishes high-res details after the autoregressive backbone sets layout and semantics.
• Why this step exists: High-res detail is costly to model autoregressively; decoupling lets each part do what it’s best at.
• Example: The backbone draws the poster and text boxes; the refiner sharpens edges and textures.

The test: ERNIE 5.0 was evaluated on many tasks: language knowledge and reasoning, coding, multilingual exams, visual reasoning and document understanding, general VQA, video understanding, image/video generation, ASR and audio understanding, and TTS. This checks if one model can be strong and balanced across everything, not just one specialty.

The competition: Comparisons include strong open and proprietary models like DeepSeek V3.2, Qwen, Gemini 2.5/3, GPT-5 (High), Seedream, and specialist visual/audio systems. Baselines show how dedicated or modular models perform when they don’t share everything in one backbone.

The scoreboard with context:

• Language (pre-train): ERNIE 5.0-Base beats strong open baselines on knowledge (e.g., PopQA, HotPotQA), general reasoning (MMLU-Pro, MMLU-Redux), STEM (MATH CoT, GPQA-Diamond), coding (LiveCodeBench v6, CRUXEval), and multilingual (MMMLU), showing it’s like getting an A or A+ when classmates average B+ to A-.
• Language (post-train): ERNIE 5.0 stays competitive with leading models. On instruction following (MultiChallenge, Multi-IF) and agent benchmarks (ACEBench, BrowseComp-zh), it excels—like being a reliable all-rounder rather than hyper-optimized for only one “contest exam.”
• Vision (post-train): On document understanding (ChartQA, AI2D) and robust perception tests (VLMAreBlind), ERNIE 5.0 is among the top performers, often matching or surpassing specialized systems. That’s like scoring top marks in reading charts and spotting tricky visual illusions.
• Image generation: On GenEval, ERNIE 5.0 hits about 90, matching/approaching top commercial systems, meaning strong alignment between prompts and generated images.
• Video generation: On VBench, ERNIE 5.0 leads on semantic alignment and remains strong overall—like making a video that not only looks good but truly follows the request.
• Audio: ASR word error rates are low across Chinese and English benchmarks; voice-chatting benchmarks show solid reasoning; audio understanding scores are strong on acoustic scenes. TTS content consistency (WER) is competitive with generalist omni models.

Efficiency findings that matter:

• Reducing routing top-k to 25% at inference speeds decoding by over 15% with only minor accuracy loss—like running faster without tripping.
• Elastic variants keep near-full performance using only 53.7% activated parameters and 35.8% total parameters, scoring about 75.17 vs 75.55 for the full model—like carrying a lighter backpack but still finishing the hike at nearly the same time.

Surprising findings:

• Even with a single, modality-agnostic router, experts self-specialize by task (e.g., visual generation vs. understanding) rather than rigidly by modality. This emergent behavior shows the router learns to send the right tokens to the right brains without hand-made rules.
• Visual understanding benefits from attention-based fusion (CNN + ViT) more than simple MLP adapters, especially for documents and charts with tiny text.
• Long-horizon generation gets sturdier when the model practices self-correction (bit flips on past tokens) and uses loss reweighting to focus on early predictions.

Takeaway in plain words: ERNIE 5.0 is not just good at many things; it stays good when you make it smaller or faster, and it learns to share its own smarts across skills. Speed-ups don’t crater accuracy, and balanced scores show this isn’t a one-trick system.

Limitations:

• Extremely long generations (very long videos or speeches) can still accumulate errors, even with robustness tricks.
• Trillion-parameter MoE pre-training needs large-scale compute and careful engineering; not every lab can reproduce the full run.
• While balanced, the model may trail top specialized systems on certain bleeding-edge tasks (e.g., the very hardest math contests or ultra-high-fidelity TTS).
• Diffusion refiner adds an extra step for top-tier image/video quality, which complicates deployment a bit for production pipelines aiming for end-to-end autoregression only.

Required resources:

• Multi-node GPU clusters with high-speed interconnects; support for tensor/pipeline/expert parallelism; robust memory management (e.g., FP8, offloading).
• Storage and throughput for large multimodal datasets and long-context training.
• RL infrastructure that keeps training and inference numerically consistent and unbiased.

When NOT to use:

• If you only need a tiny text-only model on a microcontroller, a compact specialized LLM may be simpler.
• If you require the absolute best in one vertical (e.g., studio-grade video or pro TTS), a top specialist might edge out a generalist.
• If your task forbids any external diffusion step but demands ultra-high-res fidelity, a pure diffusion or dedicated AR pipeline might fit better.

Open questions:

• Can we push error-correction further to make ultra-long video/audio generation as steady as short clips?
• How far can modality-agnostic routing go—could experts specialize by abstract skills (like counting or spatial reasoning) even more explicitly?
• What’s the best way to blend diffusion and autoregression inside one backbone without training conflicts?
• Can elastic training expand to hidden-size elasticity and other axes without adding instability?
• How to reduce data and compute costs while keeping the same balanced multimodal quality?

Three-sentence summary: ERNIE 5.0 unifies text, image, video, and audio into one autoregressive model that predicts the next group of tokens for any modality. It scales with a sparse Mixture-of-Experts and a modality-agnostic router, and it trains elastically so many deployable sizes are learned at once. Stable RL tricks keep learning efficient across hard multimodal tasks, resulting in strong, balanced performance.

Main achievement: Showing that a single, natively autoregressive, ultra-sparse, and elastic model can both understand and generate across all major modalities—at production scale—while maintaining efficiency and balance.

Future directions:

• Stronger long-horizon stability for ultra-long video/audio.
• Tighter integration of diffusion and autoregression.
• More axes of elasticity (e.g., hidden size) and smarter automatic expert allocation.
• Even more robust, low-cost multimodal RL with better self-correction.

Why remember this: ERNIE 5.0 is a blueprint for the next wave of AI that doesn’t just talk about pictures or sounds—it reads, reasons, and creates them within one shared brain, and it can shrink or speed up without forgetting what it learned.