Scaling Embeddings Outperforms Scaling Experts in Language Models

🍞 Hook: Imagine you have a giant library with many librarians (experts). At first, hiring more librarians makes it much faster to find books. But after a while, they start bumping into each other, and the walkie-talkies they use (communication) get jammed, so adding more doesn’t help much anymore.

🥬 The Concept (Mixture-of-Experts, MoE): MoE is a model where only a few specialized “experts” are used for each token, so you can have many total parameters but pay computation only for the chosen experts. How it works (simple recipe):

1. Split the model’s big feed-forward part into many experts. 2) A router picks a small set of experts per token. 3) Only those experts run, and their results are combined. 4) Repeat for each layer. Why it matters: Without MoE, scaling would mean paying computation for all experts every time, which would be too slow and expensive. 🍞 Anchor: Like calling only the two librarians who know the right shelf, not the whole staff.

🍞 Hook: You know how a good backpack has useful side pockets you rarely use? Many language models underuse one such pocket: the embedding layer.

🥬 The Concept (Embeddings): An embedding turns each token into a vector so the model can do math on words. How it works: 1) Each token ID looks up a row in a table. 2) That row is a vector that represents the token’s meaning. 3) The model uses those vectors to think and predict next tokens. Why it matters: If embeddings are weak, the rest of the model starts with poor inputs and wastes effort fixing them. 🍞 Anchor: Like turning each word into a little coordinate on a map so the model knows roughly where it lives.

🍞 Hook: Imagine labeling not just single words but short word sequences like “ice cream” so the model instantly knows they belong together.

🥬 The Concept (N-gram Embedding): N-gram embeddings add extra vectors for short spans of tokens (like 2-grams, 3-grams) to capture local context. How it works: 1) For token i, collect recent tokens (pairs, triples, etc.). 2) Hash each n-gram into a table. 3) Look up vectors from several small sub-tables and linearly project them. 4) Sum these with the base embedding (optionally scaled/normalized). Why it matters: Without these local-context vectors, the model must learn common phrases later, which is slower and costlier. 🍞 Anchor: It’s like giving a special tag for “New York” so the model treats it as a place, not “new” + “york.”

🍞 Hook: Picture a team that keeps hiring more specialists. It helps until the hallway gets crowded. What if instead we improve the labels on the books so each librarian grabs the right one faster?

🥬 The Concept (Embedding Scaling): Instead of adding more experts, you grow the embedding system (especially with n-grams) to pack more knowledge into fast lookups. How it works: 1) Add n-gram tables with sub-tables and small projections. 2) Balance how many parameters go to embeddings vs. experts. 3) Train with signal amplification so embeddings don’t get drowned out. 4) Use system tricks so lookups stay fast. Why it matters: Past MoE’s “sweet spot,” more experts give little gain and add I/O cost; bigger embeddings can improve accuracy without heavy compute. 🍞 Anchor: Sharper book labels beat hiring a tenth extra librarian who can’t move freely.

🍞 Hook: You know how you sometimes guess the end of a friend’s sentence because you’ve heard similar phrases?

🥬 The Concept (Speculative Decoding): A fast draft model proposes several next tokens; the big model quickly checks and accepts or rejects them. How it works: 1) Draft several steps. 2) Verify in parallel. 3) Accept what matches; fall back where needed. 4) Repeat to grow the batch and speed. Why it matters: Without it, decoding is strictly one-token-at-a-time and underuses the hardware. 🍞 Anchor: Like rough-sketching a paragraph, then having a teacher quickly mark which parts are correct so you don’t rewrite everything.

🍞 Hook: Imagine sorting mail into many slots using a code. Sometimes two different letters go into the same slot by accident.

🥬 The Concept (Hash Collisions): When hashing n-grams into tables, two different n-grams can map to the same index, mixing their meanings. How it works: 1) Apply a hash function. 2) If two items share an index, they share a vector. 3) Multiple sub-tables and smart vocab sizes reduce clashes. 4) Training tries to untangle shared signals. Why it matters: Too many collisions blur meanings and hurt accuracy. 🍞 Anchor: Like two students sharing one cubby with their names mixed up.

🍞 Hook: Picture planning a trip budget. If you spend everything on snacks (embeddings), you can’t afford tickets (experts).

🥬 The Concept (Parameter Budgeting): Decide how much of the total parameters go to embeddings vs. experts. How it works: 1) Measure sparsity (total vs. activated parameters). 2) Add n-grams after MoE hits diminishing returns. 3) Keep n-gram share at or below ~50%. 4) Tune n-gram order (N) and sub-tables (K). Why it matters: Overspending on any part can reduce overall performance. 🍞 Anchor: Set a rule like “no more than half the trip money goes to snacks.”

🍞 Hook: Think of giving each floor in a building its own mini library so they don’t all crowd one basement.

🥬 The Concept (Per-Layer Embedding, PLE): Each layer can get its own embedding table, or even per-layer n-gram embeddings (PLNE) to inject information deeper. How it works: 1) Replace a gate in the MLP with an embedding lookup. 2) Optionally use n-gram outputs per layer. 3) Compare against a simple top-of-model n-gram approach. Why it matters: Without careful design, per-layer methods can add activated compute and not beat simpler n-gram scaling. 🍞 Anchor: The paper finds plain n-gram embeddings generally outperform PLE and PLNE for the same parameter budgets.

🍞 Hook: Imagine the librarian whispering so quietly that no one hears the helpful hint.

🥬 The Concept (Embedding Amplification): Scale or normalize embedding outputs so they aren’t drowned out by attention and MLP signals. How it works: 1) Multiply embeddings by a scaling factor (often √D). 2) Or apply LayerNorm before merging. 3) Monitor residual stream norms. 4) Keep signal strength balanced during training. Why it matters: Without amplification, n-gram signals can get lost, reducing gains. 🍞 Anchor: Turning up the microphone so the announcements are audible.

The world before: MoE was the default way to scale sparsity in large models, letting you have many parameters without paying full compute each step. But as models got sparser and bigger, gains shrank, and system costs (especially I/O and communication) became painful. The embedding layer—cheap O(1) lookups—wasn’t fully used as a scaling route. The problem: When should you scale embeddings instead of experts? How many parameters should you budget? Which n-gram settings reduce collisions? How does width vs. depth change the benefits? And can these extra embedding parameters actually speed up inference end-to-end? Failed attempts: Simply adding more experts past the sweet spot gave little improvement and more communication. Early n-gram variants without amplification underperformed because their signal got swamped. Some per-layer embedding schemes added activated compute without consistent wins. The gap: A principled, measured recipe for when and how to scale embeddings, how to tune vocabulary sizes to avoid collisions, how to amplify signals, and how to translate sparsity into real speed. Real stakes: Faster and smarter assistants, better code agents, and more efficient long-context reading—all with lower latency and better throughput on the same hardware.

Aha! Moment in one sentence: Once Mixture-of-Experts passes its sweet spot, putting the next chunk of parameters into n-gram embeddings—carefully budgeted, collision-aware, and signal-amplified—beats adding more experts, and can be turned into real speedups with smart system design.

Three analogies:

1. Library vs. Labels: When hallways get crowded (MoE scaling saturates), better book labels (n-gram embeddings) help everyone find things faster.
2. Backpack Pockets: Instead of cramming more gear (experts) in the main compartment, add well-placed side pockets (n-gram tables) that are quick to reach.
3. Lego Baseplate: A stronger baseplate (embeddings capturing local context) makes every tower (layer) more stable so you don’t keep reinforcing higher floors (more experts).

Before vs. After:

• Before: The go-to move was to add experts. Performance kept improving but with diminishing returns and rising I/O costs.
• After: Add n-gram embeddings once MoE benefits taper off; keep embedding parameters ≤~50% of total; pick n-gram vocab sizes to avoid collision spikes; amplify embeddings so they’re heard; and use speculative decoding to convert parameter sparsity into throughput.

Why it works (intuition, no equations):

• Embeddings are O(1) lookups, so parameter counts can grow without proportional compute. N-grams inject rich local co-occurrence info directly at the input, reducing burden downstream. Multiple sub-tables and small projections act like independent views that reduce collision harm. Amplification prevents the residual stream from drowning out the embedding signal. Wider models can consume the added information better (more width to mix signals), while very deep models dilute early signals unless amplified.

Building blocks (bite-sized):

• N-gram Branch: For each token, collect recent n-grams, hash into sub-tables, look up small vectors, project, sum.
• Sub-Table Decomposition (K): Several differently sized sub-tables per n-gram reduce collision risk and improve coverage.
• Vocabulary Sizing: Avoid n-gram vocab sizes near integer multiples of base vocab to prevent sharp collision spikes.
• Hyperparameters: N≈3–5 and K≥2 are robust; extreme values give little added benefit.
• Embedding Amplification: Scale or LayerNorm the embedding outputs to keep their contribution strong.
• Width vs. Depth: Wider models extend the advantage range; much deeper models need more care (amplification) or see reduced gains.
• System Side: N-gram Cache, kernel fusion, and speculative decoding grow effective batch and turn theoretical sparsity into real latency and throughput wins.

🍞 Hook: Think of improving the labels on every book so all later steps are easier. 🥬 The Concept (Core Idea): Scale embeddings—especially n-gram embeddings—after MoE’s sweet spot, up to about half of total parameters, with collision-aware vocab sizes and amplified signals; then apply system optimizations so the speed benefits show up in practice. How it works: 1) Detect MoE saturation. 2) Allocate parameters to n-gram embeddings (≤~50%). 3) Choose N and K in a robust range (e.g., N=3–5, K≥2). 4) Avoid collision-prone vocab sizes. 5) Amplify embeddings. 6) Deploy with N-gram Cache, fused kernels, and multi-step speculative decoding. Why it matters: This route produces a better accuracy-speed Pareto frontier than simply scaling experts. 🍞 Anchor: Their 68.5B model with ~31B in n-gram embeddings outperforms a parameter-matched MoE baseline and runs fast with optimized inference.

High-level recipe: Text tokens → Base embedding + N-gram embedding branch → Transformer with MoE (fewer active parameters) → Speculative decoding and optimized kernels → Output tokens.

Step A: Build the N-gram Embedding Branch

• What happens: For each token i, form recent n-grams (2-gram, 3-gram, …, up to N). Hash each n-gram into K sub-tables with different vocab sizes, look up small vectors, apply small linear projections, sum them all with the base embedding.
• Why it exists: Captures local context early, packs knowledge into O(1) lookups, and reduces the load on downstream experts.
• Example: For the token “C” in “… A B C …”, include (B,C) for 2-gram and (A,B,C) for 3-gram, look them up in several sub-tables, project, and add to C’s base embedding.

Step B: Choose N and K (Hyperparameters)

• What happens: Pick N (n-gram order) and K (number of sub-tables) to balance coverage and collisions.
• Why it exists: Too small (N=2, K=1) underfits; too large sparsifies training signals with little gain.
• Example: They find N≈3–5 and K≥2 work consistently well, with small variation beyond that sweet range.

Step C: Budget Parameters Across Experts vs. Embeddings

• What happens: Monitor sparsity (total vs. activated parameters). Once MoE hits diminishing returns (“sweet spot”), allocate the next parameters to n-gram embeddings. Keep n-gram parameters ≤~50% of total.
• Why it exists: Prevents overspending on embeddings (U-shaped performance vs. proportion). Ensures win over increasing experts.
• Example: Their curves cross near a total/active ratio ≈20; at that point n-gram embeddings are ~50% of total.

Step D: Reduce Hash Collisions via Vocabulary Sizing

• What happens: Choose n-gram vocab sizes that significantly deviate from integer multiples of the base vocab size; keep multiple sub-tables for robustness.
• Why it exists: Collision counts spike near integer multiples, especially for 2-grams, which blurs meaning.
• Example: If base vocab is 128k, avoid n-gram vocab sizes too close to simple multiples like 30× or 32×; pick sizes that steer clear of those spikes.

Step E: Amplify the Embedding Signal

• What happens: Multiply embedding outputs by a scaling factor (often √D) or apply LayerNorm before merging into the residual stream.
• Why it exists: Prevents attention outputs from drowning out the embedding signal, which otherwise weakens n-gram benefits.
• Example: With amplification, they consistently reduce training and validation loss by ~0.02 compared to the same model without it.

Step F: Shape the Backbone (Width vs. Depth)

• What happens: Keep sufficient width (hidden size and module dimensions) to exploit the extra embedding information; be cautious with very deep stacks.
• Why it exists: Wider models maintain an advantage over MoE at higher parameter ratios; very deep models dilute early signals unless amplified.
• Example: At 1.3B activated parameters, n-gram embeddings retain advantage even at ratios up to ~50; at 280M, the advantage fades earlier.

Step G: Deploy Efficient Inference

• What happens: Use an N-gram Cache (device-side ID management) to speed lookups, fuse kernels to reduce overhead, and apply multi-step speculative decoding to expand effective batch size.
• Why it exists: Turning theoretical parameter sparsity into wall-clock speed requires high hardware utilization and low launch/I/O overhead.
• Example: They separate a fast draft model (which can use standard embeddings) and cache n-gram results during drafting to avoid recomputation in verification.

Secret Sauce:

• The combination—not just one piece. Specifically: (1) Add n-grams after MoE’s gains flatten; (2) keep embedding share ≤~50%; (3) avoid collision-prone vocab sizes; (4) amplify embeddings; and (5) use N-gram Cache + fused kernels + speculative decoding to realize real-world speedups.

End-to-end with real data flow:

1. Tokenize input (sequence length can be long; their model reaches up to 256k context via YARN during training).
2. For each token, compute base embedding and N-gram branch: hash 2–N-gram windows into K sub-tables, look up vectors, project, sum with base.
3. Apply amplification (scaling or LayerNorm) so the residual stream carries a strong embedding signal.
4. Pass through transformer layers with MoE; thanks to embedding scaling, fewer parameters need to be actively read/used per token in MoE layers, reducing I/O.
5. During decoding, use speculative decoding to expand effective batch, the N-gram Cache to avoid repeated ID work, and fused kernels to keep GPUs busy.
6. Output next tokens; repeat.

🍞 Hook: Imagine labeling phrases well, then building highways so trucks don’t idle at toll booths. 🥬 The Concept (Method): A careful recipe that adjusts what to learn in embeddings, when to add them, how to keep them audible, and how to make the hardware go fast. How it works: Follow Steps A–G above. Why it matters: Doing only one or two steps won’t yield the full accuracy-speed improvement; the gains come from the full pipeline. 🍞 Anchor: In practice, this yields LongCat-Flash-Lite, which beats a same-size MoE baseline and runs fast with the provided kernels.

The Test: They measured training and validation loss vs. parameter allocation strategy, studied collision behavior and hyperparameter sensitivity, and evaluated downstream benchmarks across general knowledge, reasoning, coding, and agentic tool use. They also profiled inference throughput on multi-GPU servers with speculative decoding.

The Competition: Parameter-equivalent MoE baselines that spend the same total parameters by adding more experts, plus a direct baseline model (LongCat-Flash-Lite-Vanilla) where all n-gram embedding parameters were converted into experts.

Scoreboard with context:

• Scaling Curves: When sparsity is low, extra experts still help a lot. But at higher sparsity, adding n-gram embeddings beats adding experts. The curves cross around a total/activated ratio ~20 in one setting. Past that, embedding-heavy models are better—until you over-allocate embeddings (>~50%), after which performance dips (a U-shape trend echoed in other work).
• Hyperparameters: N=2, K=1 underperforms; N≥3, K≥2 perform similarly well, with small variance. This makes configuration robust and practical.
• Collisions: 2-gram collision spikes appear when the n-gram vocab size nears integer multiples of base vocab; steering vocab sizes away from those points substantially reduces collisions.
• Amplification: Adding scaling or LayerNorm to the embedding outputs improved all losses by ~0.02 versus a plain setup, indicating the embedding signal was previously being drowned.
• Width vs. Depth: Wider models show a bigger and longer-lasting advantage from n-gram embeddings, even up to ratios ~50 at 1.3B activated parameters. Much deeper models (20–40 layers) show reduced advantage unless amplified, consistent with early signals getting diluted.

Downstream Benchmarks (base models at ~1.3T tokens mid-train point):

• General: MMLU 85.52 (comparable to strong baselines), with big gains on Chinese-centric CEval 86.55 and CMMLU 82.48 over some peers.
• Reasoning: Gains across BBH and GPQA vs. the vanilla MoE baseline; solid on DROP and GSM8K.
• Coding: HumanEval+ 31.10 and BigCodeBench 36.05 beat the matched MoE baseline.

Chat Model (agentic + coding + general + math):

• Agentic Tool Use: Tau2-Bench Telecom 72.8 and Retail 73.1, beating reported baselines, and VitaBench 7.00 also ahead—like scoring a solid A when others hover around B.
• Agentic Coding: SWE-Bench 54.4 and TerminalBench 33.75, large margins over counterparts—like not just passing, but fixing more real GitHub issues reliably under constraints.
• General/Math: Competitive MMLU/MMLU-Pro, strong MATH500 and solid AIME24/25, indicating capable multi-step reasoning.

System Performance:

• Active Experts Reduced: N-gram scaling shifts capacity away from MoE, lowering activated parameters and I/O per step.
• Inference Throughput: With Eagle3-style deployment, wide expert parallelism, and SBO, plus N-gram Cache and fused kernels, they achieve strong tokens-per-second per device and per user. Multi-step speculative decoding grows effective batch and turns the sparse-parameter design into real latency and throughput gains.

Surprising Findings:

• Collision Spikes near integer multiples of the base vocab size (especially for 2-grams) significantly impact performance—this is a sharp, non-linear effect.
• The advantage of n-gram embeddings gets bigger with width but shrinks with depth, suggesting a clear architectural lever for planning future models.
• Simple amplification (scale or LayerNorm) delivers a consistent, measurable improvement, highlighting how easy it is for early signals to be overpowered in pre-norm stacks.

🍞 Hook: Imagine racing two cars: one adds more passengers (experts) and the other adds better directions (embeddings). Past a point, better directions win. 🥬 The Concept (Results): Across curves, ablations, collisions, and large benchmarks, embedding scaling—done right—beats adding experts and turns into real throughput gains with system support. How it works: Embed after MoE’s sweet spot, cap at ~50%, avoid collision vocab sizes, amplify signals, and deploy with cache + fused kernels + speculative decoding. Why it matters: It moves the Pareto frontier: better accuracy for similar or lower latency, especially in agentic and coding domains. 🍞 Anchor: LongCat-Flash-Lite (68.5B total, ~3–4.5B activated, ~31.4B in embeddings) consistently outperforms the parameter-matched MoE baseline and is highly competitive with peers.

Limitations:

• Collision Sensitivity: N-gram hashing still risks collisions; careful vocabulary sizing and multiple sub-tables help, but there’s no zero-collision guarantee.
• Depth Dilution: Benefits shrink in very deep stacks unless you amplify embeddings; designing for depth requires more tuning.
• Activated Compute in Variants: Per-layer (PLNE) adds activated parameters (due to projections) and didn’t consistently beat simpler n-gram scaling.
• Engineering Overhead: Real speedups need an N-gram Cache, fused kernels, and speculative decoding; without them, the wall-clock gains are smaller.
• Regime Dependence: If your MoE is still below its sweet spot, adding experts may still be the better next step.

Required Resources:

• Large-scale training data (they used 11T + 1.5T tokens across phases) and strong multi-GPU inference stacks to realize speed gains.
• Engineering to implement the cache, fused kernels, and speculative decoding.

When NOT to Use:

• Small models with tight memory where extra embedding tables are hard to store.
• Early in MoE scaling (below the sweet spot) when more experts still yield big gains.
• Ultra-deep pre-norm stacks without amplification, where embedding signals are likely to be drowned.

Open Questions:

• Can n-gram embeddings directly power an ultra-fast draft model for speculative decoding (draft-by-embedding)?
• How to do early rejection using embedding semantics safely without hurting acceptance rates?
• What’s the best per-layer allocation strategy (concentrated in a few layers vs. spread out)?
• Can learned, collision-aware hashing further reduce collisions versus fixed polynomial hashes?
• How do these findings generalize to non-text modalities or multilingual, byte-level tokenizations?

🍞 Hook: Think of this as a new trail on the mountain—well-marked, but with some rocky spots ahead. 🥬 The Concept (Discussion): Embedding scaling opens a powerful path, but it’s not universal; it works best after MoE’s sweet spot, in wide models, with attention to collisions and signal strength. How it works: Combine modeling and systems know-how; keep tuning vocab sizes, amplification, and deployment. Why it matters: The approach lifts both accuracy and efficiency, yet invites deeper research on drafting, rejection, and allocation. 🍞 Anchor: If your current model is MoE-heavy and plateauing, this paper’s recipe provides a practical, tested next move.

Three-sentence summary: This paper shows that after MoE reaches diminishing returns, scaling n-gram embeddings—kept within about half of total parameters, with collision-aware vocab sizes and amplified signals—beats adding more experts. With an N-gram Cache, fused kernels, and multi-step speculative decoding, the theoretical sparsity becomes real-world speed. The resulting 68.5B LongCat-Flash-Lite outperforms a parameter-matched MoE baseline and excels at agentic and coding tasks.

Main achievement: A principled, end-to-end recipe—model, training, and systems—for when and how to scale embeddings instead of experts, validated by a strong open model.

Future directions: Turn n-gram embeddings into a drafting engine, develop reliable early-rejection gates using embedding semantics, refine per-layer allocation strategies, and explore collision-aware or learned hashing schemes. Investigate multilingual and byte-level regimes, and study interactions with other compression and inference tricks.

Why remember this: It reframes scaling sparsity—showing that the embedding layer is not just a small front door but a powerful axis for adding knowledge cheaply and quickly. With the right timing, budgeting, and engineering, scaling embeddings moves the accuracy–latency frontier forward for real applications.

🍞 Hook: When more librarians stop helping, better labels start winning. 🥬 The Concept (Takeaway): Scale embeddings after the MoE sweet spot, keep them balanced and audible, and wire your system to be fast. Why it matters: That’s how you get a smarter, faster model without just throwing more compute at the problem. 🍞 Anchor: LongCat-Flash-Lite proves it in practice.