SLA2: Sparse-Linear Attention with Learnable Routing and QAT

🍞 Top Bread (Hook): You know how when you watch a long movie, you don’t stare at every pixel on the screen—you focus on the parts that matter, like faces and moving objects? That’s how our brains save effort while still understanding the story.

🥬 Filling (The Actual Concept):

• What it is: An attention mechanism in AI helps models focus on the most important parts of their input, like your eyes and brain do in a movie.
• How it works: (1) It compares each piece of information (a “token”) to others. (2) It scores what matters most. (3) It uses those scores to blend the useful pieces and make a decision. (4) Repeat across layers to build meaning.
• Why it matters: Without attention, the model wastes time treating everything as equally important, like trying to read a book while giving each word the same focus.

🍞 Bottom Bread (Anchor): When you ask a question like “What’s the capital of France?”, attention helps the model zoom in on “capital” and “France,” making “Paris” pop out.

🍞 Top Bread (Hook): Imagine you’re cleaning your room. You put the super-important items (like your homework) in clear spots and shove the less important stuff into a box to handle quickly later.

🥬 Filling (The Actual Concept – Sparse Attention):

• What it is: Sparse attention does the full, precise math only for the few most important pairs of tokens, and skips most others.
• How it works: (1) Pick top connections to keep. (2) Do exact attention math on those. (3) Ignore or downplay the rest. (4) Normalize so each row of attention still makes sense.
• Why it matters: It saves tons of time and memory on long sequences because we avoid computing everything about everything.

🍞 Bottom Bread (Anchor): When generating a video of a cat running, the model can precisely connect frames where the cat’s legs move, and skip long-range links to the background grass.

🍞 Top Bread (Hook): Picture a fast shortcut that gives a good-enough answer most of the time, like using a map’s overview to guess your route instead of checking every tiny street.

🥬 Filling (The Actual Concept – Linear Attention):

• What it is: Linear attention is a faster, approximate way to compute attention so cost grows roughly with sequence length, not with its square.
• How it works: (1) Change how queries and keys are represented using a special function. (2) Pre-compute helpful summaries. (3) Combine them quickly with queries. (4) Normalize to keep scales stable.
• Why it matters: It keeps attention speedy on very long sequences but is less precise than full softmax attention.

🍞 Bottom Bread (Anchor): For distant frames in a video that mostly carry general scene info (like overall brightness), linear attention can summarize them quickly.

🍞 Top Bread (Hook): Suppose you split chores with a friend: you wash the delicate glasses carefully, and your friend speed-cleans the sturdy plates. You need a rule for who does what.

🥬 Filling (The Actual Concept – Sparse-Linear Attention, SLA):

• What it is: SLA mixes two branches: sparse (precise on a few) and linear (fast on the rest) to balance quality and speed.
• How it works: (1) Choose positions to compute precisely (sparse branch). (2) Let the linear branch handle the remaining positions. (3) Combine their results to imitate full attention.
• Why it matters: Sparse alone may miss global info; linear alone may blur details. Together, they aim for the best of both.

🍞 Bottom Bread (Anchor): In a video, the sparse branch keeps the cat’s sharp motion consistent, while the linear branch spreads general scene info like lighting and sky color.

🍞 Top Bread (Hook): Imagine you split chores by eyeballing which dishes “look” dirtier—sometimes you guess wrong.

🥬 Filling (The Actual Concept – The Problem with Heuristic Routing):

• What it is: Older SLA used a fixed rule (a heuristic) to decide which pairs go sparse or linear, usually by picking the biggest attention weights for sparse.
• How it works: (1) Pool tokens into blocks. (2) Score block pairs. (3) Take Top-k as sparse. (4) Send the rest to linear.
• Why it matters: This guess can be suboptimal—some items sent to sparse don’t help much, and some sent to linear break important patterns.

🍞 Bottom Bread (Anchor): It’s like always hand-washing the tallest cups because they look dirtier, even if some short cups are actually grimey.

🍞 Top Bread (Hook): Think of measuring with a ruler that’s slightly off. Even if you split tasks well, your results can still be scaled wrong.

🥬 Filling (The Actual Concept – Mismatch in Decomposition):

• What it is: Sparse attention renormalizes rows, causing a scaling mismatch with what full attention actually contributes in those spots.
• How it works: (1) Full attention splits into “kept” and “not kept” parts. (2) Sparse attention recomputes probabilities only over “kept,” making each row sum to 1. (3) But full attention’s kept part has a smaller total mass. (4) So outputs differ by a per-row scale (alpha).
• Why it matters: If you don’t fix the scale, the linear branch must repair both “its own job” and “sparse’s scaling error,” which makes learning harder and quality worse.

🍞 Bottom Bread (Anchor): If you portion a pizza slice as 100% of your meal, you’ll think you ate a full meal when it was only a slice—the numbers won’t match what you truly ate.

🍞 Top Bread (Hook): Now imagine giving the team a coach who actually learns who should do which chore, and a measuring cup that fixes that ruler problem.

🥬 Filling (The Actual Concept – Why SLA2):

• What it is: SLA2 adds a learnable router and a learnable mixing ratio (alpha) that blend sparse and linear attention faithfully, plus low-bit attention trained the smart way (QAT) for extra speed.
• How it works: (1) A router learns which positions deserve sparse vs. linear. (2) A per-row alpha blends the two correctly. (3) Low-bit attention is used in forward training so the model adapts to quantization. (4) At inference, everything runs fast with minimal quality loss.
• Why it matters: This combination keeps details sharp, keeps global info stable, and makes attention both accurate and quick—especially for long video sequences.

🍞 Bottom Bread (Anchor): The result is videos that look as good or better than full attention but render much faster—even at 97% sparsity.

Real Stakes: Faster, cheaper, and greener video generation matters for creators, classrooms, streaming apps, and phones with limited compute. With SLA2, you can get smooth, high-quality results without waiting forever or needing giant GPUs.

🍞 Top Bread (Hook): Imagine a school orchestra that plays faster when the conductor assigns tricky solos to experts and background parts to the rest—plus a volume knob that balances them perfectly.

🥬 Filling (The Actual Concept – The Aha! in One Sentence):

• What it is: SLA2 learns both how to route each attention piece to the precise or fast branch and how to blend their outputs with a learned per-row ratio (alpha), while training to handle low-bit math.
• How it works: (1) Router learns which entries go sparse vs. linear. (2) Compute each branch efficiently. (3) Alpha blends them as alpha*sparse + (1−alpha)*linear, removing scaling mismatch. (4) Low-bit forward with QAT makes sparse even faster. (5) End-to-end fine-tuning locks in quality.
• Why it matters: It fixes both the suboptimal split and the scaling mismatch that made older SLA harder to learn and less accurate.

🍞 Bottom Bread (Anchor): The model ends up like a band that knows who should play the solo, who should keep the beat, and how loud each should be to make the best music.

Multiple Analogies:

1. Traffic Control: The router is a smart GPS sending heavy traffic (important pairs) onto high-speed exact lanes and the rest onto express summaries; alpha is the traffic light timing that blends flows smoothly.
2. Cooking: Sparse attention is chopping delicate herbs precisely; linear attention is bulk-prepping veggies fast; alpha is the recipe’s ratio that keeps flavor balanced.
3. Sports: Stars take clutch plays (sparse), role players keep the flow (linear), and alpha is the coach’s minute allocation that keeps the team efficient and fresh.

Before vs. After:

• Before: Heuristic split (often Top-k of pooled scores) and a projection trying to patch scale errors; linear branch had to cover both its job and sparse’s mismatch.
• After: Learnable routing trained to minimize approximation error, faithful alpha mixing that aligns with the true decomposition, and QAT so low-bit math works smoothly at inference.
• What changes: Higher sparsity (up to 97%), big attention speedups (around 18.6×), and equal or better video quality—even surpassing full attention after fine-tuning.

Why It Works (Intuition, no equations):

• The router learns patterns of where exact attention matters most (like motion edges or subject interactions) and where summaries suffice (backgrounds or repetitive textures).
• Alpha corrects the per-row scale so the sparse branch doesn’t overstate its piece; this lets the linear branch focus on global information instead of fixing someone else’s error.
• QAT makes the model “practice” with low-bit math during training, so it becomes robust to quantization at test time.

Building Blocks (each with a sandwich):

🍞 Hook: You know how a teacher learns which students need more help vs. who can work independently? 🥬 Learnable Router:

• What it is: A small model that decides, for each block pair, whether to use sparse or linear attention.
• How it works: (1) Pool nearby tokens in Q and K to reduce cost. (2) Project them into a router space. (3) Score pairs and pick top-k per row. (4) Use a soft, differentiable Top-k during training; switch to hard Top-k at inference.
• Why it matters: It finds a smarter split than simple “largest weights,” increasing sparsity without losing quality. 🍞 Anchor: The router learns that the cat’s paws and nearby pixels need precise links, while distant sky patches can use global summaries.

🍞 Hook: Imagine a mixing knob that balances vocals and instruments just right. 🥬 Alpha Mixing (Faithful Sparse–Linear Decomposition):

• What it is: A learned per-row ratio alpha that blends the sparse and linear outputs so their probabilities sum correctly.
• How it works: (1) Compute sparse output on kept entries. (2) Compute linear output on the rest. (3) Blend them as alpha*sparse + (1−alpha)*linear. (4) Normalize by design so rows stay stable.
• Why it matters: It fixes the scaling mismatch that made older SLA lean on a projection to patch errors. 🍞 Anchor: The “volume” of sparse vs. linear is set per row, so fine details stay crisp and global tone stays smooth.

🍞 Hook: Think of practicing a piano piece while wearing gloves so that when you take them off, you play even better. 🥬 Quantization-Aware Training (QAT) for Low-Bit Attention:

• What it is: Train while simulating low-bit arithmetic in the forward pass, but keep backward precise, so the model learns to handle low-bit at test time.
• How it works: (1) Quantize Q, K (and later P, V) during forward. (2) Dequantize to combine results. (3) Backpropagate using full-precision tensors. (4) Fine-tune so errors shrink.
• Why it matters: You get the speed of low-bit math with much less accuracy loss. 🍞 Anchor: After QAT, the model runs with INT8/FP8-like speed yet keeps quality high, even at very high sparsity.

High-Level Overview: Input (Q, K, V) → Learnable Router (build sparse mask M) → Compute Sparse Output on M and Linear Output on 1−M → Blend with alpha row-wise → Final attention output O.

Step-by-step (like a recipe), with sandwiches for key pieces:

1. Inputs and Pooling 🍞 Hook: Imagine summarizing each chapter of a book into a short paragraph before deciding which pages to study carefully. 🥬 What it is: We pool (average) neighboring tokens in Q and K into blocks so the router is cheaper to run.

• How it works: (1) Split Q into blocks of size bq, K into blocks of size bk. (2) Replace each block with its mean. (3) Use these summaries for routing.
• Why it matters: Without pooling, the router would be too slow (it would need to look at every single token pair). 🍞 Anchor: For a 1,024-token sequence with bq=128 and bk=64, the router only considers 8×16 block pairs, not 1,024×1,024 pairs.

1. Learnable Router (SoftTop-k during training, Hard Top-k at inference) 🍞 Hook: Like picking the top students per row in a seating chart to answer hard questions. 🥬 What it is: A trainable module that projects pooled Q and K, scores block pairs, and selects top-k per row.

• How it works: (1) Two learnable projections map pooled Q and K into a router-friendly space. (2) Compute scores for each row of pooled-Q vs pooled-K blocks. (3) During training, use SoftTop-k (a smooth, differentiable version) that enforces the same number of selected entries per row. (4) During inference, switch to hard Top-k for maximum speed.
• Why it matters: A smart split boosts sparsity and quality. Without it, you either compute too much (slow) or cut the wrong parts (blurry results). 🍞 Anchor: The router might select 3% of block pairs for sparse attention (the most critical links) and send the other 97% to the linear branch.

1. Sparse Attention on Selected Entries (with QAT in forward) 🍞 Hook: Think of carefully polishing only the most visible parts of a display case. 🥬 What it is: Exact softmax attention computed only where the mask M=1, with quantized forward math to go faster.

• How it works: (1) For each selected block pair, compute local scores and softmax safely (streaming style). (2) Multiply probabilities by V to get outputs. (3) Use low-bit quantization for Q, K, P, and V in forward. (4) Keep backward gradients in full precision for stability.
• Why it matters: Exact attention preserves fine details. Without it, small but important patterns (like motion edges) vanish. 🍞 Anchor: The model exactly links the cat’s paw in frame t to the paw in frame t+1, maintaining motion sharpness.

1. Linear Attention on the Complement (1−M) 🍞 Hook: Like using a big paint roller to quickly cover the background wall. 🥬 What it is: A fast approximation that summarizes the unselected pairs.

• How it works: (1) Pre-compute key–value summaries only for blocks where M=0. (2) Combine them with transformed queries. (3) Normalize each row to keep scales consistent. (4) Skip any unneeded score computations entirely.
• Why it matters: Without linear attention, you’d either ignore the rest (lose context) or compute everything (too slow). 🍞 Anchor: It quickly carries over general scene lighting and colors across frames.

1. Alpha Mixing (Faithful Sparse+Linear Blend) 🍞 Hook: Like using a slider to blend a sharp close-up with a wide background shot until it looks just right. 🥬 What it is: A learned per-row weight alpha that blends sparse output and linear output so they match the true decomposition.

• How it works: (1) Compute both outputs. (2) Blend as alpha*sparse + (1−alpha)*linear. (3) Ensure rows are properly normalized by design. (4) Learn alpha during training.
• Why it matters: Without alpha, sparse outputs are mis-scaled; the linear branch must fix errors it didn’t cause. 🍞 Anchor: If alpha=0.7 for a row, that row trusts the sparse details more; if alpha=0.2, it leans on linear’s global context.

1. Training Strategy (Two Stages) 🍞 Hook: Like learning drills first with safety cones (soft rules), then playing the real game with official rules. 🥬 What it is: A two-stage process: (Stage 1) pretrain the router and alpha with a differentiable SoftTop-k; (Stage 2) fine-tune the whole diffusion model with hard Top-k.

• How it works: (Stage 1) Use saved Q, K, V samples from different layers and timesteps; minimize the difference between full attention and SLA2 outputs under different sparsity levels (like 5%, 4%, 3%). (Stage 2) Replace attention with SLA2 inside the model; fine-tune end-to-end on the diffusion loss using hard Top-k so training matches inference behavior.
• Why it matters: Without Stage 1, the router may start bad and destabilize fine-tuning. Without hard Top-k in Stage 2, training/inference mismatch harms quality. 🍞 Anchor: After Stage 1, the router already picks sensible pairs; Stage 2 then polishes the whole system for the final task.

1. Quantization-Aware Training Details 🍞 Hook: Practicing with ankle weights so game day feels easier. 🥬 What it is: Low-bit math in the forward pass, full precision in the backward pass.

• How it works: (1) Quantize inputs (Q, K) before computing masked scores; later quantize P and V for the product. (2) Dequantize carefully to keep scales correct. (3) Keep gradients FP16 to avoid training instability. (4) End-to-end fine-tuning adapts parameters to low-bit noise.
• Why it matters: Post-training quantization alone can hurt quality; QAT keeps quality high. 🍞 Anchor: In ablations, skipping QAT lowered video quality; with QAT, quality stays high while speed increases.

1. Efficiency Tricks (Flash-style, Blocked Implementation) 🍞 Hook: Packing your suitcase by sections so you never unpack everything at once. 🥬 What it is: A FlashAttention-style, blockwise kernel that only computes what’s needed.

• How it works: (1) Stream softmax per block to avoid full score matrices. (2) Only compute for M=1 in sparse branch. (3) In linear branch, precompute K^T V only where M=0. (4) Fuse steps to reduce memory movement.
• Why it matters: Without these, you’d lose most of the speed savings to memory overheads. 🍞 Anchor: SLA2 shows up to ~18.7× kernel speedups over FlashAttn2 at 97% sparsity.

Concrete Mini-Example:

• Suppose a 12-token sequence is split into Q-blocks of 3 (4 rows) and K-blocks of 4 (3 columns). The router selects top-1 per row (≈33% block sparsity): say row 1→col 2, row 2→col 1, row 3→col 2, row 4→col 3. Sparse computes exact attention only on those block pairs; linear computes summaries for others. Alpha near 0.8 on row 1 emphasizes precise links; alpha near 0.3 on row 4 trusts global summaries more. Blending yields an output close to full attention at a fraction of the cost.

Secret Sauce:

• Learnable routing focuses sparse compute exactly where it matters most.
• Alpha mixing fixes the longstanding scaling mismatch so each branch does its own job.
• QAT unlocks low-bit speed without paying a big quality tax.

🍞 Top Bread (Hook): Think of a race where runners are judged not only by how fast they finish but also by how gracefully they run and how well they follow the course.

🥬 Filling (The Actual Concept – The Test):

• What it is: The team measured video quality and speed on strong video diffusion models while pushing sparsity very high.
• How it works: (1) Fine-tune SLA2 and baselines on text-to-video models (Wan2.1-1.3B-480P and Wan2.1-14B-720P). (2) Evaluate visual quality on many dimensions (like sharpness, consistency, and how much people prefer it). (3) Measure speed via kernel throughput and end-to-end latency.
• Why it matters: It shows whether SLA2 can keep videos looking great while cutting attention cost dramatically.

🍞 Bottom Bread (Anchor): It’s like testing a new bicycle that’s both lighter (faster) and steadier (better handling) than older bikes on the same track.

🍞 Top Bread (Hook): You know how we sometimes compare your typing speed and accuracy to your classmates’? We need fair comparisons here too.

🥬 Filling (The Actual Concept – The Competition):

• What it is: SLA2 was compared to Full Attention, SLA (older sparse–linear), VSA, and VMoBA.
• How it works: (1) Run each method with typical sparsities (90%, 95%, and for SLA2 also 97%). (2) Use the same training budget and datasets. (3) Track both quality and speed.
• Why it matters: Beating top baselines at lower cost shows real progress.

🍞 Bottom Bread (Anchor): It’s like a spelling bee where SLA2 spells more words correctly than others while doing it faster.

🍞 Top Bread (Hook): If I say “you got 87%,” that doesn’t mean much unless you know others got 75%.

🥬 Filling (The Actual Concept – The Scoreboard, with Context):

• What it is: SLA2 reaches about 97% attention sparsity and around an 18.6× attention speedup, while maintaining or improving video quality.
• How it works: (1) At 90% and 95% sparsity, SLA2 beats all baselines on multiple VBench-like quality metrics. (2) Even at 97% sparsity, SLA2 still tops others that use 90% sparsity, and can even surpass full attention after fine-tuning. (3) Kernel throughput shows up to ~18.7× over FlashAttn2; end-to-end latency drops by 2.3× on 1.3B and 4.35× on 14B models.
• Why it matters: It’s like getting an A+ in quality while running the race way faster when others at best get B’s at slower speeds.

🍞 Bottom Bread (Anchor): With SLA2, a long, high-res cat video renders much quicker, yet the fur, motion, and lighting stay crisp and natural.

🍞 Top Bread (Hook): Sometimes the experiment surprises you, like discovering you can run faster with better form, not just stronger legs.

🥬 Filling (The Actual Concept – Surprising Findings):

• What it is: Sparse methods, after fine-tuning, can even beat full attention on some metrics.
• How it works: (1) The fine-tuning dataset quality matters—a good dataset can make the sparse+linear combo generalize better. (2) The router learns to emphasize truly useful links. (3) Alpha prevents magnitude drift.
• Why it matters: It challenges the instinct that “more compute always means better.”

🍞 Bottom Bread (Anchor): Like cleaning a room by focusing on the mess that matters most; you can end with a tidier space than if you tried lazily dusting every corner.

Ablations:

• QAT: Removing QAT and then quantizing dropped quality; with QAT, low-bit attention kept quality while improving speed ~1.3× at the kernel level.
• Router: The learnable router clearly outperformed a plain Top-k router (used by older SLA). Lower sparsity gives best quality, but even 97% sparsity remains very strong.

🍞 Top Bread (Hook): Even the best tool has situations where it’s not perfect—like a sports car that’s amazing on highways but not ideal on rocky trails.

🥬 Filling (The Actual Concept – Honest Assessment):

• Limitations:
  1. Data and tuning: To get the best of SLA2 (especially the router and alpha), you benefit from a decent amount of fine-tuning data and careful hyperparameters.
  2. Router cost: Although pooled and blockwise, the router still adds overhead; on ultra-short sequences the relative gain may be smaller.
  3. Extreme approximation: Pushing sparsity even beyond 97% or using very aggressive low-bit settings can harm edge cases (tiny, fast details).
  4. Engineering complexity: QAT and custom kernels require careful implementation to realize the full speedups.
• Required Resources:
  • A GPU setup that supports mixed precision and efficient attention kernels; enough memory to fine-tune; and data for training router+alpha in Stage 1 and full model in Stage 2.
• When NOT to Use:
  • Tiny models or very short sequences where full attention is already cheap.
  • Tasks where every token pair is equally crucial (rare), making sparsity less beneficial.
  • Settings where training/fine-tuning isn’t possible and PTQ alone causes unacceptable quality loss.
• Open Questions:
  • Can routing be further improved with richer context (e.g., multi-head aware signals) without losing speed?
  • How does alpha generalize across domains (e.g., long-form storytelling vs. dynamic sports)?
  • What is the best co-design of sparsity + quantization across different hardware (GPU vs. specialized accelerators)?
  • Can we learn dynamic sparsity targets (instead of fixed k%) that adapt by layer, head, or timestep?

🍞 Bottom Bread (Anchor): Think of SLA2 as a tuned race bike: amazing for long rides and speed, but you still choose a mountain bike for rocky trails.

🍞 Top Bread (Hook): Picture a team where one player is super precise, another is super fast, and a coach perfectly decides who handles each play and how to blend their contributions.

🥬 Filling (The Actual Concept – 3-Sentence Summary):

• SLA2 learns how to route attention entries to a precise sparse branch or a fast linear branch and then blends them with a learned per-row ratio (alpha) that fixes a known scaling mismatch.
• It uses a differentiable router during training (with SoftTop-k), hard Top-k at inference, and quantization-aware training so low-bit attention runs fast without hurting quality.
• On strong video diffusion models, SLA2 reaches about 97% sparsity and around an 18.6× attention speedup, while maintaining or improving video quality and cutting end-to-end latency.

Main Achievement: A faithful sparse–linear attention design—learnable routing plus alpha mixing—paired with QAT that delivers both high speed and high quality, even at very high sparsity.

Future Directions: Smarter routers (multi-head, multi-scale), adaptive sparsity targets, deeper co-design with hardware, and broader testing across domains like long-form video, audio-visual tasks, and streaming generation.

Why Remember This: SLA2 shows that with the right split, the right blend, and training that anticipates low-bit math, you can have speed and quality together—making high-end video generation more practical for everyone.