MOSS Transcribe Diarize Technical Report

🍞 Top Bread (Hook): Imagine you’re the class note-taker. It’s not enough to write the words; your friends also need to know who said each sentence and exactly when they said it so they can jump back to that part in the recording.

🥬 Filling (The Actual Concept — Automatic Speech Recognition, ASR)

• What it is: ASR is a tool that listens to talking and turns it into written words.
• How it works:
  1. The microphone records sound waves.
  2. A model turns those waves into speech features.
  3. Another model guesses the letters and words.
  4. It cleans up the guesses to form readable text.
• Why it matters: Without ASR, we can’t get a transcript at all. 🍞 Bottom Bread (Anchor): When you say, “Hello everyone,” ASR writes “Hello everyone.”

🍞 Top Bread (Hook): Picture two friends with similar voices chatting behind a door; you try to tell who is speaking just by listening.

🥬 Filling (The Actual Concept — Speaker Diarization)

• What it is: Diarization decides which speaker is talking at each moment.
• How it works:
  1. It learns voice “fingerprints” (embeddings) from the audio.
  2. It clusters similar voice chunks together.
  3. It draws boundaries between speakers over time.
• Why it matters: Without diarization, we’d have words but no idea who said them. 🍞 Bottom Bread (Anchor): The transcript says: [S1] “Hi,” [S2] “Hello,” so you know who spoke.

🍞 Top Bread (Hook): Think of a movie script that shows not only lines and characters, but also exact timecodes to edit scenes perfectly.

🥬 Filling (The Actual Concept — Speaker-Attributed, Time-Stamped Transcription, SATS)

• What it is: SATS is a transcript that says what was said, who said it, and when.
• How it works:
  1. Turn speech into words (ASR).
  2. Assign each stretch of words to the right speaker (diarization).
  3. Add precise timestamps around each speaker’s turns.
• Why it matters: Without timestamps and speaker tags, it’s hard to search, skim, or trust long meeting notes. 🍞 Bottom Bread (Anchor): “[0:11] [S01] Good morning!” tells you the time and the speaker.

The World Before: For years, people built SATS by stitching together separate parts: one system for ASR (like Whisper), another for diarization (like x-vector clustering or Pyannote), and sometimes an extra aligner for timestamps. This ‘glue-it-together’ plan worked okay for short clips with few speakers, but struggled in long meetings with many voices, accents, and overlaps.

The Problem: In these stitched pipelines, mistakes pile up. If the ASR mishears a word, the diarization step might mislabel the speaker next. And because each tool is trained separately, they don’t share the same global view of the whole conversation. Also, many models can only look at short chunks at a time, so they forget who people were earlier and get confused about long-range references like “As I said 20 minutes ago…”. Timestamping often needs yet another tool, creating more chances for mismatch.

Failed Attempts: Some researchers tried a semi-cascaded fix: keep ASR and diarization, then use a language model at the end to clean up and make speaker labels consistent (like DiarizationLM). Better, but still not truly end-to-end, so errors still sneak through. Others tried bringing speech and speakers closer together, like Sortformer (which trained a speaker model first) and SpeakerLM (which merged ideas inside a larger model). These improved parts of the problem but usually handled only short audio (about a minute) or few speakers, and often didn’t output turn-level timestamps by themselves.

The Gap: What was missing was a single model that could:

• Listen once to long audio (tens of minutes),
• Write the words, tag the speakers, and produce timestamps directly,
• Keep speaker memory steady even if the same person returns after a long break,
• Avoid chunk boundaries that cause timing and identity hiccups.

Real Stakes (Why You Should Care):

• Meetings: You need trustworthy records that let you jump to “the part where Alex gave the timeline,” not just read a wall of text.
• Call centers: Accurate who-said-what-when helps training, audits, and quality checks.
• Accessibility: People who are deaf or hard of hearing need clear, speaker-labeled captions with timing to follow multi-person talks.
• Legal/education: Depositions, lectures, and group discussions need precise timelines and speakers to cite or review evidence.
• Search and analytics: You can ask, “Show me when Speaker 3 first mentioned budget,” and jump there.

🍞 Top Bread (Hook): Imagine if one super note-taker could listen to the whole discussion at once and write a perfect, time-coded script with names for each line.

🥬 Filling (The Actual Concept — Why end-to-end SATS now)

• What it is: A single multimodal large language model (MLLM) that does transcription, who-spoke-when, and timestamps together.
• How it works:
  1. An audio encoder turns speech into sound features.
  2. A projection maps those features into the language model’s space.
  3. The language model generates a combined script: timestamps, speaker tags, and words.
• Why it matters: Shared context reduces error chains and keeps speakers consistent across long stretches. 🍞 Bottom Bread (Anchor): One pass produces: [0.11] [S01] “Good morning!” [1.11] [S02] “Morning, guys!”

That is the scene MOSS Transcribe Diarize walks into: a world that needs unified, long-context, end-to-end SATS to make multi-speaker transcripts truly reliable and useful.

🍞 Top Bread (Hook): You know how a great orchestra conductor hears every instrument together and keeps the whole piece in rhythm? That’s better than having separate mini-conductors for each section who try to sync later.

🥬 Filling (The Actual Concept — The “Aha!” Moment)

• What it is: The key insight is to train one unified model that, in a single pass, writes the words, tags the speakers, and prints timestamps—over very long audio—so nothing gets lost between separate tools.
• How it works:
  1. Feed the long audio into an audio encoder to get multi-speaker acoustic features.
  2. Project those features into the text LLM’s space so sound and language live together.
  3. Autoregressively generate a transcript that interleaves timestamp tokens and speaker labels with words.
  4. Use a very large context window (128k tokens) so the model can remember who’s who for up to ~90 minutes.
• Why it matters: This ends the back-and-forth handoffs that cause error chains, and it preserves long-range speaker memory. 🍞 Bottom Bread (Anchor): Output looks like: [0.11] [S01] Good morning! [1.11] [S02] Morning, guys!

Multiple Analogies (three ways to see it):

• Conductor: One leader hears everything at once and keeps time/speakers synchronized.
• Movie Script Supervisor: Tracks who speaks each line and records exact timecodes for editing.
• Class Note-Taker: Writes a clean, time-coded, who-said-what summary while listening to the full lecture.

Before vs. After:

• Before: Separate ASR + diarization + aligner tools, often chunking audio into pieces, causing speaker drift and boundary glitches.
• After: Single end-to-end model that handles full-length audio in one go, maintaining consistent speaker identities and printing timestamps directly.

🍞 Top Bread (Hook): Imagine a Swiss Army knife that can listen and write at the same time.

🥬 Filling (The Actual Concept — Multimodal Large Language Model, MLLM)

• What it is: An MLLM understands multiple kinds of input (like audio and text) under one roof.
• How it works:
  1. Audio becomes features via an encoder.
  2. A projection aligns those features with the text model’s representations.
  3. The LLM reasons jointly about sounds, words, and speakers.
• Why it matters: One brain for both listening and writing reduces mismatch. 🍞 Bottom Bread (Anchor): The model hears two voices, knows which is which, and writes clear, labeled lines with times.

🍞 Top Bread (Hook): Think of cooking a sandwich all at once instead of preparing each part in different kitchens and hoping they fit later.

🥬 Filling (The Actual Concept — End-to-End Modeling)

• What it is: Training one model to do the whole task from input to final output.
• How it works:
  1. Show the model audio and the desired speaker-tagged, time-stamped text.
  2. Let it learn to map directly from audio to that output.
  3. Optimize it so the whole chain improves together.
• Why it matters: No error cascades between disconnected parts. 🍞 Bottom Bread (Anchor): The model directly prints [time][speaker] words, without passing results to another tool.

🍞 Top Bread (Hook): Picture a giant whiteboard where you can keep notes from the entire class period instead of erasing every few minutes.

🥬 Filling (The Actual Concept — Context Window)

• What it is: The amount of information the model can look at and remember at once.
• How it works:
  1. Increase the token limit (128k) so long audio fits in memory as tokens.
  2. Let attention span the whole meeting so earlier speakers stay recognizable.
  3. Maintain global consistency across an hour or more.
• Why it matters: Small windows force chunking and cause speaker drift; big windows preserve continuity. 🍞 Bottom Bread (Anchor): A person who speaks at minute 3 and again at minute 70 is still recognized as the same speaker.

🍞 Top Bread (Hook): Imagine you write the time next to each diary entry line so you can jump back later.

🥬 Filling (The Actual Concept — Timestamp Generation as Text Tokens)

• What it is: The model prints timestamps (like [1.23]) right in the transcript, as if they were words.
• How it works:
  1. Insert formatted time tokens between segments during training.
  2. Teach the model to output them at boundaries and turns.
  3. Keep timing stable over long durations without special aligners.
• Why it matters: Precise, inline timestamps make searching, skimming, and syncing easy. 🍞 Bottom Bread (Anchor): “[12.07] [S03] Let’s move on.” lets you jump to minute 12.07 instantly.

🍞 Top Bread (Hook): Think of remembering the voices of your classmates the whole school year.

🥬 Filling (The Actual Concept — Speaker Memory)

• What it is: The model’s ability to remember and keep each speaker consistent over time.
• How it works:
  1. Jointly learn voice and words under one model.
  2. Use the long context to carry identity clues across far-apart turns.
  3. Reduce confusion when voices are similar or noisy.
• Why it matters: Without good memory, labels drift and transcripts become unreliable. 🍞 Bottom Bread (Anchor): Even after a long silence, the model still tags the returning voice as [S02], not a new speaker.

Why It Works (Intuition):

• Shared context: One model sees everything together, so it aligns words, voices, and times naturally.
• Global memory: A huge context window anchors identities and long-range references.
• Textual timestamps: Printing times as tokens avoids fragile position tricks and works over hour-long audio.
• Unified training: Optimizing the whole task at once removes friction between separate parts.

Building Blocks:

• Audio encoder + projection to text space.
• Autoregressive generator that emits [time][speaker] tokens and words.
• Training data: large, real conversations plus simulated multi-speaker mixes with overlaps.
• Evaluation that checks words (CER), words+speakers (cpCER), and the extra difficulty from speakers (Δcp).

High-Level Overview: Input audio → Audio Encoder → Projection to LLM space → Autoregressive Generation of [timestamp][speaker] words → Speaker-attributed, time-stamped transcript

Step-by-Step (like a recipe):

1. Audio Ingest and Encoding

• What happens: The raw waveform is fed into an audio encoder that turns sound into compact, informative features for multiple voices at once.
• Why it exists: Raw sound is too big and messy; features make patterns clearer for the language model.
• Example: A 40-minute meeting becomes a sequence of acoustic embeddings that capture who’s talking and how words sound.

1. Projection into the Text LLM Space

• What happens: A learned projection maps audio features into the same kind of vectors the language model uses for text.
• Why it exists: Sound and text need to speak the same “language” inside the model so they can align smoothly.
• Example: The voice pattern for “budget” becomes a representation close to the text concept “budget.”

1. Long-Context Modeling (128k Tokens)

• What happens: The model is configured to handle very long sequences so an entire meeting can be processed in one pass.
• Why it exists: Chunking breaks continuity and causes speaker confusion at boundaries; a long window keeps the story straight.
• Example: A question at minute 5 and an answer at minute 55 are related correctly because both fit inside the same context.

1. Timestamp-as-Text Insertion and Generation

• What happens: Temporal information is represented as formatted tokens (e.g., [0.11]) that the model learns to output between segments and before speaker tags.
• Why it exists: Turning time into text tokens keeps timing accurate over long spans without relying on fragile absolute positions.
• Example: The model emits: [0.11] [S01] Good morning! [1.11] [S02] Morning, guys!

1. Joint Speaker Attribution and Word Generation

• What happens: The model autoregressively prints time tokens, then speaker tags like [S01], then the words spoken.
• Why it exists: Generating everything together lets the model tie voices to words and times consistently.
• Example: “[12.07] [S03] The deadline moved to Friday.” is produced in one fluent sequence.

1. Training Data: Real + Simulated Conversations

• What happens: The model learns from in-the-wild audio (meetings, podcasts, films) and from simulated multi-speaker mixes.
• Why it exists: Real data teaches natural variety; simulation fills gaps and provides controlled overlaps, turn-taking, and noise levels.
• Example: The simulator chooses 2–12 speakers, interleaves short segments, allows up to 80% overlap of the shorter segment, snaps boundaries to low-energy points, adds cross-fades, and mixes in real noise at 0–15 dB SNR.

1. Unified Objective and Output Normalization

• What happens: The model is trained to minimize errors on the final combined output (timestamps, speaker tags, words). For evaluation, a consistent normalization removes extra tags and matches speaker IDs by best permutation.
• Why it exists: Unified training improves the whole chain at once; normalization makes fair comparisons across systems.
• Example: Two systems that use [S1] vs. [S01] are compared fairly after normalization.

1. Inference: Single-Pass Transcript

• What happens: At test time, you feed the full audio; the model emits a clean, time-stamped, speaker-labeled transcript in one go.
• Why it exists: Single-pass inference avoids chunk boundaries that cause timing artifacts and identity drift.
• Example: A 90-minute discussion is processed without splitting, preserving who-said-what-when throughout.

🍞 Top Bread (Hook): Think of a magic pen that writes the time, the speaker’s name, and the sentence all together.

🥬 Filling (The Actual Concept — Audio Encoder)

• What it is: A network that turns sound waves into features that capture phonetics and speaker traits.
• How it works:
  1. Slice audio into frames.
  2. Extract patterns (like frequencies) that signal speech sounds and voices.
  3. Pack them into vectors.
• Why it matters: Better features mean clearer recognition and speaker separation. 🍞 Bottom Bread (Anchor): The encoder hears two overlapping voices and preserves clues that help tag them later.

🍞 Top Bread (Hook): Imagine plugging a guitar into an amp with the right adapter so the sound comes through clearly.

🥬 Filling (The Actual Concept — Projection Module)

• What it is: A learned adapter that maps audio features into the text model’s space.
• How it works:
  1. Take encoder outputs.
  2. Transform them so they align with the LLM’s token embeddings.
  3. Enable joint reasoning about sound and words.
• Why it matters: Without this adapter, the LLM wouldn’t understand the audio’s structure. 🍞 Bottom Bread (Anchor): The word “schedule” in audio lands near the text concept “schedule” inside the model.

🍞 Top Bread (Hook): Think of jotting down the time next to every sentence in your notes.

🥬 Filling (The Actual Concept — Timestamps as Tokens)

• What it is: The model treats times like special words (e.g., [1.44]).
• How it works:
  1. Insert and train on formatted time tokens.
  2. Generate them at boundaries and turns.
  3. Keep alignment stable across long audio.
• Why it matters: You can search and jump precisely without a separate aligner. 🍞 Bottom Bread (Anchor): “[23.50] [S02] Let’s recap.” points you to the exact moment.

🍞 Top Bread (Hook): Picture remembering a classmate’s voice the whole semester.

🥬 Filling (The Actual Concept — Long-Range Speaker Memory)

• What it is: The model’s ability to keep each speaker’s identity steady over time.
• How it works:
  1. Train jointly on words and voices.
  2. Use a large context so earlier clues stay visible.
  3. Reduce confusion under similar voices or noise.
• Why it matters: Keeps labels from drifting in long meetings. 🍞 Bottom Bread (Anchor): Speaker 4 sounds the same at minute 3 and minute 63, and is labeled [S04] both times.

The Secret Sauce:

• Single-pass end-to-end generation avoids cross-module mismatches and boundary artifacts.
• Textual timestamps scale to hour-long audio without fragile positional tricks.
• The 128k-token window lets the model hold the whole meeting in mind.
• Real-plus-simulated training teaches the model to survive overlaps, noise, and speaker re-entries.

Putting It All Together:

• Input: Multi-speaker audio.
• Steps: Encode audio → Project to text space → Generate [time][speaker] words with a long context.
• Output: A readable, searchable transcript with who-said-what-when, ready for meetings, podcasts, and films.

🍞 Top Bread (Hook): Think of a talent show where contestants must sing, dance, and keep perfect timing all at once. You don’t just grade their singing—you also check if their dance matches the beat and who performed each part.

🥬 Filling (The Actual Concept — What was tested and why)

• What it is: The model was tested on recognizing words, keeping speaker labels correct, and marking times across different kinds of audio.
• How it works:
  1. Use three datasets: AISHELL-4 (long, real meetings), Podcasts (long, multi-guest talks), Movies (short, overlap-heavy clips).
  2. Compare against strong commercial systems: Doubao, ElevenLabs Scribe v1, GPT-4o (when possible), Gemini 2.5 Pro, and Gemini 3 Pro (when stable).
  3. Measure pure transcription errors (CER), joint word+speaker errors (cpCER), and the extra errors caused by speaker attribution (Δcp = cpCER − CER).
• Why it matters: Great transcripts need accurate words and correct speakers with precise timing. 🍞 Bottom Bread (Anchor): If CER is low but Δcp is high, the words are right but the who-said-what is confused.

🍞 Top Bread (Hook): Counting red marks on a spelling test tells you how many letters were wrong.

🥬 Filling (The Actual Concept — Character Error Rate, CER)

• What it is: The percentage of character mistakes in the transcript (ignoring which speaker said it).
• How it works:
  1. Compare predicted text to the true text.
  2. Count insertions, deletions, and substitutions.
  3. Divide by the total number of characters.
• Why it matters: Shows raw transcription accuracy. 🍞 Bottom Bread (Anchor): If the true line is “hello” and you wrote “hallo,” that’s one substitution error.

🍞 Top Bread (Hook): Imagine rearranging name tags to match who actually said each line in a group skit.

🥬 Filling (The Actual Concept — cpCER)

• What it is: Like CER, but it also includes speaker labels and chooses the best matching of predicted speaker IDs to the true ones.
• How it works:
  1. Align predicted [speaker + text] against ground truth.
  2. Try all permutations of speaker label matching.
  3. Keep the assignment with the fewest character errors.
• Why it matters: Fairly scores the joint task even if you called Speaker 1 “S01” and the reference called them “S2.” 🍞 Bottom Bread (Anchor): If you swapped S1 and S2 everywhere, cpCER still finds the best matching before scoring.

🍞 Top Bread (Hook): Think of Δcp as the “extra trouble” caused by mixing up who spoke, beyond plain spelling mistakes.

🥬 Filling (The Actual Concept — Δcp)

• What it is: The difference cpCER − CER; it isolates how much speaker mix-ups hurt performance.
• How it works:
  1. Compute CER (just words).
  2. Compute cpCER (words + speakers).
  3. Subtract to find the speaker-attribution penalty.
• Why it matters: Smaller Δcp means steadier speaker labeling. 🍞 Bottom Bread (Anchor): If CER is like a B+ and Δcp makes it drop to a C, speaker labeling needs work.

The Scoreboard (with context):

• AISHELL-4 (long real meetings, ~40 minutes): • MOSS Transcribe Diarize achieved CER 15.43% and cpCER 20.04%, with Δcp 4.61%. • Competing systems showed higher CER and cpCER; for example, a baseline had CER 18.18% and cpCER 27.86% (Δcp 9.68%). • What it means: MOSS not only heard words better, it kept speakers straight over long spans, cutting the speaker penalty roughly in half versus that baseline.

• Podcasts (long multi-guest interviews): • MOSS reached CER 4.46% and cpCER 6.97%, with Δcp 2.50%. • Baselines had higher errors; e.g., one showed CER 7.93% and cpCER 10.54% (Δcp 2.61%). • What it means: Even when word recognition is strong for others, MOSS trims the extra diarization pain further, showing robust speaker memory.

• Movies (short clips, fast alternation, overlaps): • MOSS delivered CER 7.50% and cpCER 13.36%, with Δcp 5.86%, leading the pack. • Some competitors had decent CER but much larger Δcp, meaning they struggled to keep speakers correct under dense overlaps. • What it means: The unified, timestamp-as-text approach helps even in fast, choppy scenes.

Important Practical Notes:

• GPT-4o couldn’t process some long-form audio under this protocol, and Gemini 3 Pro often failed to stick to the required output format at long durations, so they were omitted where applicable.
• This highlights a real-world gap: being “multimodal” in name isn’t enough; reliable, hour-scale SATS requires specialized design and formatting robustness.

Surprising Findings:

• Long-context modeling (128k tokens) helped not just in long meetings but also in short, tricky clips by keeping identities steady.
• Emitting timestamps as text tokens worked stably across hour-long audio without extra alignment tools.
• Mixing real data with controlled simulations taught the model to handle overlaps up to 80% and noisy conditions, improving short and long-form performance alike.

Limitations:

• Compute and memory: A 128k-token context with a large model is resource-hungry, especially for training and very long inputs.
• Real-time streaming: The paper focuses on single-pass, long-context processing; low-latency, streaming SATS remains future work.
• Fine-grained timestamp scoring: While the model emits timestamps, standardized, segment-level timestamp metrics across diverse datasets are still developing.
• Multilingual breadth: The model covers mainly Chinese and English (with some others in Movies). Wider language and dialect coverage needs more data and tests.
• Identity naming: It outputs anonymous speaker IDs (e.g., [S01]) rather than real names; mapping to known identities (enrollment) is not the focus here.

Required Resources:

• Strong GPUs/TPUs with enough memory to handle 128k-token sequences and long audio feature streams.
• Storage and bandwidth for large training corpora (real + simulated) and long evaluation recordings.
• Careful prompt/output formatting to ensure stable inference over long durations.

When NOT to Use:

• Ultra-low-latency live captioning where partial results must stream every few hundred milliseconds; a dedicated streaming design may be better today.
• Extremely noisy, music-dominated, or non-speech audio where transcription is not the main goal.
• Single-speaker, short clips where a lightweight ASR might be cheaper and fast enough.
• Strict privacy settings where sending long raw audio to a large model is not allowed.

Open Questions:

• Streaming SATS: How to keep the end-to-end benefits while delivering low-latency partial results?
• Timestamp evaluation: What standard, fine-grained metrics best capture segment-level timing quality across languages and domains?
• Multilingual expansion: How to robustly scale to many languages, dialects, and code-switching?
• Speaker enrollment: How to let users pre-register voices and keep consistent IDs across meetings without hurting zero-shot generalization?
• Efficiency: Can we shrink compute needs (distillation, sparse attention, caching) without losing long-range consistency?

Three-Sentence Summary: MOSS Transcribe Diarize is a unified audio–text model that performs transcription, speaker attribution, and timestamping together in one end-to-end pass. By using a 128k-token context and printing timestamps as text tokens, it keeps speaker identities steady and timing accurate across up to ~90 minutes of audio. On meetings, podcasts, and movie clips, it beats strong commercial systems on joint accuracy, especially in keeping Δcp low.

Main Achievement: It shows that fully end-to-end, long-context SATS—with timestamps emitted directly as tokens—outperforms modular or semi-cascaded systems and scales to hour-long, multi-speaker audio.

Future Directions: Build a true streaming version that preserves long-range speaker memory; develop finer timestamp metrics and benchmarks; broaden multilingual coverage; explore speaker enrollment for stable IDs across sessions; and improve efficiency for everyday deployment.

Why Remember This: It’s a blueprint for the next generation of meeting and conversation tools: a single, reliable model that knows who said what and when over long spans. That means clearer notes, better search, fairer analytics, and easier accessibility—turning messy, multi-voice audio into structured knowledge you can trust.