Qwen3-TTS Technical Report

🍞 Top Bread (Hook): You know how an audiobook narrator reads words on a page and brings them to life with the right tone and rhythm?

🥬 Filling (The Actual Concept): Text-to-Speech (TTS) is technology that turns written words into spoken speech.

• What it is: A system that reads text out loud like a digital narrator.
• How it works: 1) Read the text, 2) Plan how it should sound (pronunciation, pauses, emotion), 3) Turn that plan into audio.
• Why it matters: Without TTS, computers can’t speak to us naturally—making assistants, accessibility tools, and language learning much harder.

🍞 Bottom Bread (Anchor): When your map app says “Turn left in 200 meters,” that’s TTS doing the talking.

The World Before: Early TTS could be robotic and monotone. Newer neural TTS sounded better but still struggled with three big realities: real-time streaming (starting audio instantly, not after a long wait), fine-grained control (e.g., “speak like a calm teacher with a warm smile”), and multilingual consistency (keeping the same voice across different languages). Voice cloning existed, but often needed longer samples, and changing style or emotion precisely was tricky.

🍞 Top Bread (Hook): Imagine building with LEGO bricks—you need the right-sized blocks to make strong models fast.

🥬 Filling (The Actual Concept): Discrete Speech Tokenization breaks speech into small building-block tokens for models to handle.

• What it is: A way to turn continuous audio into sequences of symbols (tokens), like letters for sound.
• How it works: 1) Analyze audio, 2) Compress it into tokens that keep meaning and sound details, 3) Let a model predict these tokens, then 4) Turn tokens back to audio.
• Why it matters: Without tokens, the model has to juggle raw waveforms—too big and too detailed for fast, stable generation.

🍞 Bottom Bread (Anchor): It’s like zipping a huge video into a smaller file so you can stream it smoothly and then unzip it to watch clearly.

The Problem: Make TTS that is at once multilingual, controllable, robust for long passages, and ultra-low-latency for streaming—plus clone a voice from just 3 seconds. Teams found trade-offs: fast systems could sound less expressive; expressive systems could be slow; multilingual systems might change voice timbre or accent too much.

🍞 Top Bread (Hook): Picture trying to pour water as someone keeps handing you new cups—you want the water to start flowing immediately, not after you finish stacking all the cups.

🥬 Filling (The Actual Concept): Streaming Synthesis means the model starts speaking while it’s still reading the text.

• What it is: Audio begins almost right away and keeps flowing as more text arrives.
• How it works: 1) Read a bit of text, 2) Predict the next chunk of speech tokens, 3) Decode and play them, 4) Repeat.
• Why it matters: Without streaming, users wait for the whole sentence or paragraph before hearing anything.

🍞 Bottom Bread (Anchor): Voice assistants that answer mid-sentence instead of making you wait are using streaming.

Failed Attempts: Purely semantic tokenizers (great for the “what” being said) often lacked expressive detail. Purely acoustic tokenizers (great for the “how” it’s said) could overload the language model with fine-grained bits, causing drift and errors over long speech. Diffusion decoders produced high quality but added delay; some autoregressive systems were quick but fragile for long or multilingual speech.

🍞 Top Bread (Hook): Think of a friend who can copy voices after hearing a short clip from a cartoon—their brain grabs what makes that voice unique.

🥬 Filling (The Actual Concept): Voice Cloning copies a person’s voice from a short sample.

• What it is: Making a new digital speaker that sounds like the target.
• How it works: 1) Listen to a reference clip, 2) Extract speaker traits (timbre, pitch habits), 3) Use those traits to guide new speech.
• Why it matters: Without cloning, every voice would sound generic or need hours of training.

🍞 Bottom Bread (Anchor): Giving your game character your own voice after the model hears a 3-second clip is voice cloning.

The Gap: We needed a design that kept meaning stable, carried expressive detail, started speaking in under ~100 ms, and held the same voice across languages and minutes of speech. We also needed natural language control (“brighter tone, slightly slower, hint of excitement”) that the model truly follows.

🍞 Top Bread (Hook): Imagine the same actor performing in English, Chinese, and French—but always sounding like themselves.

🥬 Filling (The Actual Concept): Multilingual Capability lets the same digital voice speak many languages consistently.

• What it is: One model that reads multiple languages without changing identity.
• How it works: 1) Train on many languages, 2) Learn shared patterns of speech and pronunciation, 3) Keep speaker traits steady across languages.
• Why it matters: Without it, your “voice” becomes a different person when switching languages.

🍞 Bottom Bread (Anchor): A cloned “Aiden” voice speaking both English and Korean while sounding like the same person is multilingual capability at work.

Real Stakes: Faster, clearer TTS improves accessibility (screen readers), customer support, education, entertainment, and voice assistants. Low latency makes conversations feel natural. Controllability and cloning let creators and companies craft distinct, brand-safe voices without big recording sessions. Multilingual consistency bridges global audiences with the same trusted voice.

🍞 Top Bread (Hook): Imagine a factory with two conveyor belts: one brings in words, the other instantly sends out the matching sounds.

🥬 Filling (The Actual Concept): The key insight is to pair a dual-track language model that reads text and immediately predicts speech tokens with two complementary speech tokenizers—one rich and one ultra-fast—so you get high quality and ultra-low latency together.

• What it is: A unified system where text tokens and speech tokens flow side-by-side, decoded by tokenizers tuned for either expressivity (25 Hz) or speed (12.5 Hz).
• How it works: 1) Convert text to tokens, 2) Dual-track model predicts the next speech tokens as soon as text arrives, 3) Decode tokens to audio with the appropriate tokenizer’s decoder, 4) Stream it out, instantly.
• Why it matters: Without this pairing, you usually pick either fast or expressive; here you get both, plus strong control.

🍞 Bottom Bread (Anchor): You type “Read in a friendly, calm tone,” and within about a tenth of a second the voice starts speaking that way.

Multiple Analogies:

1. Camera lenses: The 25 Hz tokenizer is like a portrait lens (expressive detail), while the 12.5 Hz tokenizer is a fast-action lens (quick capture). The body (dual-track LM) swaps lenses depending on the shot.
2. Highways: One lane carries “meaning” traffic; another carries “sound style” traffic. Smart merging keeps flow smooth without jams.
3. Orchestra: Text is the sheet music; tokenizers are instrument sections (strings for warmth, percussion for timing); the dual-track conductor coordinates them in real time for a seamless performance.

Before vs After:

• Before: You waited longer for first audio, had weaker control, or lost consistency in long/multilingual passages.
• After: First audio in ~97–101 ms (12 Hz variants), precise style control via instructions, strong multilingual identity, and stable long-form speech.

🍞 Top Bread (Hook): Think of solving a puzzle by placing several pieces at once instead of one-by-one.

🥬 Filling (The Actual Concept): Multi-Token Prediction (MTP) predicts multiple codebook tokens per time step.

• What it is: A module that outputs several speech tokens together (semantic layer first, then residual acoustic layers).
• How it works: 1) Predict the main semantic token, 2) Predict all residual acoustic tokens in parallel, 3) Combine them to form a rich frame.
• Why it matters: Without MTP, generation is slower and may miss fine details or add delay.

🍞 Bottom Bread (Anchor): It’s like guessing not just the next letter, but the next letter plus the accents and flourishes in one go—faster and prettier.

🍞 Top Bread (Hook): You know how some backpacks have one big compartment (simple but cramped), while others have many pockets (organized and fast to access)?

🥬 Filling (The Actual Concept): Two Tokenizers—25 Hz single-codebook and 12.5 Hz multi-codebook—cover different needs.

• What it is: 25 Hz blends semantic and acoustic cues with a diffusion transformer for high-fidelity; 12.5 Hz separates them across many small codebooks and uses a lightweight causal ConvNet for decoding.
• How it works: 25 Hz: single track → mel via flow matching (DiT) → BigVGAN vocoder. 12.5 Hz: semantic head + 15-layer RVQ for acoustics → causal ConvNet decodes directly.
• Why it matters: Without both, you’d be stuck either with higher latency (25 Hz only) or less general expressivity/control (12.5 Hz only). Together, you choose the best tool per use case.

🍞 Bottom Bread (Anchor): A live chatbot picks the 12 Hz path to talk instantly; a studio-quality narration can pick 25 Hz for lush expressivity.

Why It Works (intuition):

• Discrete tokens simplify what the LM must learn, avoiding long-horizon waveform errors.
• Separating semantic and acoustic layers (especially at 12.5 Hz) keeps meaning stable while letting style be refined.
• The dual-track arrangement reduces “thinking delay,” mapping text to sound continuously.
• Post-training with human preferences (DPO/GSPO) tunes outputs toward what listeners like.

Building Blocks (with sandwiches as introduced):

• Dual-track LM Architecture 🍞 Hook: Imagine two runners passing a baton in sync—text leads, audio follows immediately. 🥬 Filling: A model that ingests text tokens and emits aligned speech tokens right away; keeps speaker identity via a learned speaker encoder. Why it matters: Without it, you’d wait for whole sentences and lose real-time feel. 🍞 Anchor: While you type a question, the voice already begins answering.

• Qwen-TTS-Tokenizer-25Hz 🍞 Hook: Like a high-res camera that captures both face and background texture. 🥬 Filling: Single-codebook at 25 Hz; tokens converted to mel with a Flow-Matching DiT, then BigVGAN to waveform; supports streaming with block-wise attention. Why it matters: Without it, you lose some expressive richness. 🍞 Anchor: Rich, human-like storytelling with smooth prosody.

• Qwen-TTS-Tokenizer-12Hz 🍞 Hook: A sprint shoe—built for fast starts and efficient strides. 🥬 Filling: 12.5 Hz, multi-codebook (semantic + 15-layer RVQ acoustics), fully causal, decoded by a lightweight ConvNet; emits audio immediately. Why it matters: Without it, you can’t hit ~100 ms first audio in streaming. 🍞 Anchor: A customer support bot that answers with near-zero delay.

• Multi-Token Prediction (MTP) (Sandwich above)

• Instruction Following and Voice Cloning 🍞 Hook: Like giving a director’s note—“more cheerful, a bit slower”—and the actor adjusts. 🥬 Filling: Prompts include style controls; cloning from 3 s reference or from a text–speech example; post-training aligns behavior to human preferences. Why it matters: Without this, you can’t reliably steer tone or identity. 🍞 Anchor: “Use Aiden’s voice, slower pace, soft enthusiasm.”

At a high level: Input (text + optional voice reference + style instructions) → Tokenization (text tokens + speech tokens for references) → Dual-track LM predicts next speech tokens as text arrives → Code2Wav decodes speech tokens to audio → Streamed Output.

Step 1: Prepare Inputs (ChatML formatting)

• What happens: The system receives text plus optional controls: a 3-second voice clip for cloning, a chosen preset voice, or a descriptive instruction like “warm, slower, news anchor style.” All are wrapped in a consistent ChatML format so the model reads them like a conversation.
• Why it exists: Without a unified format, the model would struggle to parse instructions and references consistently.
• Example: User: “Please read this paragraph in Aiden’s voice, 10% slower, gentle and friendly.” + a 3-second Aiden clip.

Step 2: Tokenize Text and (if any) Reference Audio

• What happens: Text → standard Qwen tokenizer. If cloning from a reference clip, the learnable speaker encoder extracts a speaker embedding. If using in-context learning (text–speech pair), the speech is also encoded as tokens for the model to imitate prosody.
• Why it exists: Models need compact, structured inputs (tokens/embeddings) to reason efficiently.
• Example: “Bonjour à tous !” becomes text tokens; a 3-second sample becomes a speaker vector.

Step 3: Choose a Tokenizer Path (25 Hz vs 12.5 Hz)

• What happens: The system can deploy either: a) 25 Hz Single-Codebook Path (expressivity): Tokens later pass through a Flow-Matching Diffusion Transformer (DiT) to mel-spectrograms, then BigVGAN to waveform. Streaming is supported by chunking with a sliding-window attention mask (current block + 3-lookback + 1-lookahead), preserving context while keeping latency manageable. b) 12.5 Hz Multi-Codebook Path (ultra-low latency): A semantic codebook + 15-layer RVQ acoustic codebooks run fully causally. A lightweight causal ConvNet reconstructs waveforms incrementally with no look-ahead.
• Why it exists: Different applications need different trade-offs: lush narration vs instant replies.
• Example: Live voice chat uses 12.5 Hz; audiobook recording might prefer 25 Hz.

Step 4: Dual-Track Autoregressive Prediction

• What happens: The LM processes incoming text tokens and immediately predicts the aligned speech tokens frame-by-frame. For 12.5 Hz, it first predicts the semantic codebook, then the Multi-Token Prediction (MTP) module produces all residual acoustic codebooks at once.
• Why it exists: Without dual-track prediction, you’d wait for larger text chunks, increasing latency and risking instability.
• Example: As the phrase “the capital of France” is read, the next few speech frames are already predicted and sent to the decoder.

Step 5: Streaming Decoding to Waveform (Code2Wav)

• 25 Hz path: DiT with Flow Matching converts code to mel in chunks using block-wise attention; BigVGAN turns mel into audio. Because DiT needs a small look-ahead, the first packet includes waiting for sufficient tokens (e.g., LM must output 16 tokens for the first 8-token mel chunk; at 25 Hz ≈ 320 ms mel per packet + ~130 ms vocoder right-context). Then steady packets roll every 8 tokens.
• 12.5 Hz path: Pure left-context decoding with a causal ConvNet; one token ≈ 80 ms. To reduce overhead, packets are 4 tokens (≈ 320 ms) but can start as soon as tokens arrive—no look-ahead required.
• Why it exists: Efficient decoding is crucial for first-audio latency and scalability with many users.
• Example: 12.5 Hz 0.6B variant hits ≈97 ms first packet end-to-end in tests.

Step 6: Training Strategy for Quality and Stability

• Pre-training S1 (General): Train on 5M+ hours multilingual audio to map text → speech robustly.
• Pre-training S2 (High-Quality CPT): Emphasize high-quality data to reduce hallucinations and improve naturalness.
• Pre-training S3 (Long-Context): Raise max tokens from 8,192 to 32,768 and upsample long clips for stability over 10+ minutes.
• Post-training: a) Direct Preference Optimization (DPO) aligns outputs to human likes via preference pairs. b) GSPO + rule-based rewards improve intelligibility, style following, and stability. c) Lightweight speaker fine-tuning enables strong target-voice adaptation while staying multilingual and stable.
• Why it exists: Without staged training and alignment, models either sound bland, wander off-topic, or break in long passages.
• Example: After S3, the model can narrate a 12-minute article without odd resets or repeated words.

Step 7: Controllability and Thinking Pattern

• What happens: Style/control instructions are prepended to inputs. A probabilistically activated “thinking pattern” during training helps the model interpret complex descriptions step-by-step before speaking.
• Why it exists: Without explicit instruction handling, the model may miss nuanced requests (e.g., “slower but brighter, slight smile”).
• Example: “Use a confident yet empathetic tone; keep pauses after commas; medium volume.”

Secret Sauce:

• Dual-track + MTP: Immediate alignment between text and rich speech tokens, with multiple codebooks predicted per step.
• Two tokenizers, two strengths: 25 Hz for expressivity and long-form smoothness; 12.5 Hz for near-instant streaming.
• Causal, batch-friendly decoders: Especially the 12.5 Hz ConvNet keeps latency low even with high concurrency.
• Human-feedback alignment: DPO/GSPO ensures the outputs feel natural to listeners, not just score well on ASR.

Concrete Data Walkthrough:

• Input: “Read: ‘The aurora danced across the polar sky.’ Voice: Aiden; slower 10%; calm, warm.”
• Tokenization: Text → tokens; Aiden 3s → speaker vector.
• Dual-track LM (12.5 Hz): Predict semantic token for frame t, MTP predicts 15 residual acoustic tokens; repeat.
• Decode: 4 tokens (≈320 ms) form first packet; causal ConvNet outputs audio; user hears speech ≈100 ms after start.
• Outcome: Natural, warm, slightly slower voice consistent with Aiden across an entire paragraph.

The Test: Researchers measured how accurate, natural, controllable, and fast Qwen3-TTS is.

• Accuracy: Word Error Rate (WER) on Seed-TTS and multilingual sets; speech reconstruction quality (PESQ/STOI/UTMOS) and speaker similarity (SIM) for tokenizers.
• Control: InstructTTSEval for following detailed style/voice instructions.
• Cross-lingual: Keeping the same voice across languages on CV3-Eval.
• Long-form: Stability over 10+ minutes.
• Speed: First-packet latency and real-time factor (RTF) under concurrency.

The Competition: Compared with open and commercial systems like CosyVoice 3, MiniMax, ElevenLabs Multilingual v2, GPT-4o-mini-tts, seed baselines, and others (Higgs-Audio-v2, VibeVoice, VoxCPM).

Scoreboard (with context):

• Zero-shot cloning (Seed-TTS): Qwen3-TTS-12Hz-1.7B reached WER 0.77 (zh) | 1.24 (en), which is like scoring an A+ when top classmates are at A or B (e.g., CosyVoice 3 at 0.71 | 1.45; Seed-TTS baseline at 1.12 | 2.25). The 12 Hz models consistently beat 25 Hz on WER, showing better long-range stability.
• Multilingual (10 languages): Qwen3-TTS gets best WER in 6/10 languages (Chinese, English, Italian, French, Korean, Russian) and best speaker similarity in all 10, topping MiniMax and ElevenLabs. That’s like winning most academic subjects and also taking gold in the voice “look-alike” contest.
• Cross-lingual (voice transfer): New SOTA in tough cases like zh→ko: error rate drops ≈66% vs CosyVoice 3 (4.82 vs 14.4). Voice identity stays consistent across languages while content stays clear.
• Controllability (InstructTTSEval): The 12Hz-1.7B Voice Design model sets a new open-source high score (better DSD/RP than specialized systems), and Qwen3-TTS beats GPT-4o-mini-tts by large margins in target-speaker editing while remaining competitive with Gemini models.
• Target-speaker SFT: After fine-tuning on one voice (monolingual), Qwen3-TTS generalizes that identity to 10 languages and beats GPT-4o-Audio-Preview in 7/10 languages on WER.
• Long-form: The 25 Hz 1.7B variant shines here (lowest WER among baselines), suggesting the semantic-rich path can be extra stable over 10+ minutes, whereas some chunk-based systems show boundaries or drift.

Speed and Scaling:

• First-Packet Latency: ≈97 ms (12Hz-0.6B), ≈101 ms (12Hz-1.7B) end-to-end. That’s faster than a blink.
• Under concurrency, the 12.5 Hz causal decoder keeps RTF low and scales well—practical for live services.

Surprising Findings:

• 12.5 Hz wins on content accuracy (WER) in many settings, likely because coarser frames simplify long-horizon prediction.
• 25 Hz can be more stable for very long passages—semantic emphasis may help resist drift.
• A single SFT voice transfers well to unseen languages—strong cross-lingual generalization with limited fine-tuning.

Tokenizer Benchmarks:

• 25 Hz Stage 1 matches or beats S3 tokenizers on ASR WER; Stage 2 trades slight ASR drop for richer acoustics—useful for TTS quality.
• 12 Hz tokenizer sets new SOTA on reconstruction (PESQ_WB/NB, STOI, UTMOS, SIM) at 12.5 Hz, balancing quality and bitrate exceptionally well.

Limitations:

• Language and dialect coverage: Strong across 10+ languages, but not exhaustive; rare dialects or code-switching nuances may still trip the model.
• Extreme styles: Singing, shouting, or highly theatrical prosody may require specialized training.
• Latency trade-offs: The 25 Hz path needs limited look-ahead (DiT + vocoder), so it won’t be as snappy as 12.5 Hz in first audio.
• Safety and identity: Voice cloning requires careful consent and watermarking policies beyond the model itself.

Required Resources:

• Training scale: 5M+ hours and multi-stage post-training, typically demanding large compute clusters.
• Serving: Low-latency inference stack (vLLM-like engine, CUDA Graphs, torch.compile), fast storage, and GPU(s) per concurrent users at scale.
• Data curation: High-quality pipelines for continual pretraining (CPT) and long-context upsampling.

When NOT to Use:

• Ultra-precise lip-sync for dubbing without alignment tools—needs dedicated timing alignment.
• Singing synthesis or music generation—different modeling targets.
• Very low-resource or unseen languages/dialects without fine-tuning—quality may vary.

Open Questions:

• Fairness and accent bias: How evenly does quality hold across accents and dialects not seen often in training?
• Finer control knobs: Can we expose interpretable sliders (speed, brightness, breathiness) that are guaranteed and disentangled?
• Robustness: How to further reduce rare long-form drift and improve code-switching?
• Safety: Built-in watermarking or speaker-consent checks at the model level.
• Efficiency: Can we keep 25 Hz expressivity but shave down look-ahead even more?

Three-Sentence Summary: Qwen3-TTS unifies high-quality, controllable, multilingual, and streaming text-to-speech by pairing a dual-track language model with two complementary tokenizers (25 Hz expressive and 12.5 Hz ultra-fast). It achieves state-of-the-art results in zero-shot cloning, cross-lingual transfer, instruction following, and long-form stability, while delivering ≈100 ms first-audio latency. Open-sourced under Apache 2.0, it’s practical for real-time assistants, content creation, and accessibility.

Main Achievement: Showing that you don’t have to choose between speed, control, and quality—dual-track prediction plus carefully designed tokenizers and post-training alignment make them work together at scale.

Future Directions: Expand language coverage and dialect depth, add more granular and disentangled style controls, further cut 25 Hz latency, explore watermarking/consent flows, and generalize to broader audio generation tasks beyond TTS.

Why Remember This: Qwen3-TTS demonstrates that instant, natural, multilingual speech with fine control is now feasible and open, setting a blueprint for next-generation, voice-first human–computer interaction.