Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

🍞 Hook: You know how your school year isn’t just one long class? There are weekends, holidays, and lots of moments you didn’t see—but you still need to remember what matters to keep learning. AI faces the same thing when trying to understand videos that span days or even months.

🥬 The Concept: Multimodal Lifelong Understanding is about teaching AI to make sense of long, messy streams of video and audio over real time, not just tiny clips.

• How it works (why we need it):
  1. Most old datasets used short clips where everything is right next to each other.
  2. Real life is different: important events may be hours or days apart.
  3. An AI needs to remember key facts over long gaps and connect clues spread out over time.
• Why it matters: Without this, AI can ace quick quizzes but fail at real-life stories—like forgetting who lost their wallet yesterday or when a rule changed last week.

🍞 Anchor: Imagine watching a 2-month travel vlog and being asked, “How many times did the streamer sing that song on subways in three cities?” You can’t just skim one clip—you need to track scattered moments and stitch them together.

🍞 Hook: Think of two different timers when you watch a show: one is how long you’re actually watching (screen time), and the other is how many days go by in the story.

🥬 The Concept: Observational Duration vs. Physical Temporal Span. Observational Duration is how much video you actually process; Physical Temporal Span is how much real-world time the video covers.

• How it works:
  • We define Observational Duration by $T_{dur} = \sum_{i=1}^{N} l_i$. Example: if a dataset has 3 clips with lengths 60 s, 120 s, and 30 s, then $T_{dur} = 60 + 120 + 30 = 210$ seconds.
  • We define Physical Temporal Span by $T_{span} = (\tau_N + l_N) - \tau_1$. Example: if the first clip starts at 100 s, the last clip starts at 500 s and lasts 50 s, then $T_{span} = (500 + 50) - 100 = 450$ seconds.
• Why it matters: In real life $T_{span} \gg T_{dur}$. Example: $T_{span} = 30$ days (2,592,000 s) while $T_{dur} = 10$ hours (36,000 s); here $T_{span}$ is about 72 times larger. If AI ignores this, it will miss the big empty gaps where life still changes.

🍞 Anchor: You watched 10 hours total of a streamer’s month-long trip. Even though you missed most days, you still must answer questions that depend on events that are hours apart.

🍞 Hook: Picture stuffing all your homework for the whole semester into one tiny folder—eventually papers fall out.

🥬 The Concept: Working Memory Bottleneck means the model’s short-term workspace gets too full when we add more and more frames, so performance can even go down.

• How it works:
  1. End-to-end models try to swallow many frames at once.
  2. As frames pile up, noise and irrelevant details flood their attention.
  3. The model’s internal cache/memory grows and becomes hard to search effectively.
• Why it matters: When the memory is jammed, the AI can’t reliably find the right clue, so adding more video can hurt instead of help.

🍞 Anchor: The paper shows that some strong models score okay on answers but almost fail to point to the exact time where the evidence appears—like guessing an answer without showing your work.

🍞 Hook: Imagine using a map to find a hidden treasure—but the map covers the whole Earth. Where do you even start?

🥬 The Concept: Global Localization Collapse happens when an agent tries to find specific moments directly in a massive, month-long video and loses track of where to look.

• How it works:
  1. The agent depends on global search over the raw video.
  2. The timeline is huge and very sparse.
  3. The agent keeps scanning the wrong regions and never stabilizes on the right intervals.
• Why it matters: Without a smarter plan, agents wander endlessly across time and can’t reliably ground answers.

🍞 Anchor: It’s like trying to find a 5-minute concert clip inside 100 hours of livestream by fast-forwarding randomly—you’ll likely miss it.

🍞 Hook: Think of a smart notebook that only keeps the best highlights and updates itself when you learn more.

🥬 The Concept: Dynamic Memory Management teaches an AI to store, merge, and update only useful memories so its “brain” doesn’t overflow.

• How it works:
  1. Summarize each segment into compact notes.
  2. Merge overlapping or repeated notes.
  3. Retrieve the right notes when a question arrives; refine them if needed.
• Why it matters: Without it, the model drowns in details and can’t scale to month-long understanding.

🍞 Anchor: Instead of saving every video frame, the AI saves, “At 8:15 pm in Chongqing: took a river cruise.” When asked later, it can jump back to check just that part.

🍞 Hook: You know how you solve a big puzzle by first sorting pieces, then trying a spot, checking, and trying again?

🥬 The Concept: ReMA (Recursive Multimodal Agent) is an AI that builds a language-based memory bank from video and recursively calls tools to re-check details until it’s confident.

• How it works:
  1. Perception Phase: chop the long video into chunks; summarize each chunk into memory.
  2. Control Phase: when asked a question, search memory, choose to re-inspect precise time ranges, and then update memory.
  3. Repeat a few rounds until ready to answer and point to the right clips.
• Why it matters: This avoids overstuffing a single giant context and turns video into an organized knowledge base.

🍞 Anchor: Asked “How many times did the streamer sing that song on subways across three cities?”, ReMA searches its notes, replays only promising spots, and tallies exact moments with timestamps.

🍞 Hook: When teachers grade your answer, they want to see where in the textbook you found it.

🥬 The Concept: Clue-Grounded Annotation labels the exact video intervals that contain the proof for each answer.

• How it works:
  1. Every QA pair includes the “clue” time ranges.
  2. Models must both answer and show the matching time windows.
  3. Quality checks remove questions solvable by common sense alone.
• Why it matters: This prevents guessing and forces real video-based reasoning.

🍞 Anchor: If you answer “He sang it four times,” you also need to point to the four time ranges in the livestream where it actually happens.

🍞 Hook: Scoring long videos is tricky—being off by a few minutes in a 100-hour video shouldn’t be treated the same as being off by a whole day.

🥬 The Concept: Ref@N is a grounding score that bins the timeline into N-second chunks and measures overlap between predicted and true bins.

• How it works:
  1. Split the whole video into equal bins of size N seconds.
  2. Mark bins touched by your predicted intervals and the ground-truth intervals.
  3. Compute $\text{Ref@N} = \frac{|P \cap G|}{|P \cup G|} \times 100$. Example: Suppose N = 300 s (5 min), total video = 3600 s (12 bins). If truth covers bins {2,3} and prediction covers {3,4}, then $|P\cap G|=1$, $|P\cup G|=3$, so $\text{Ref@N} = \frac{1}{3}\times 100 \approx 33.33$.
• Why it matters: It’s fair across long timelines and focuses on whether you found the right neighborhood in time.

🍞 Anchor: If you’re off by one 5-minute bin but otherwise right, you still get partial credit that makes sense for huge videos.

The world before: AI was good at quick clips and single images, but real life has gaps and evolving stories. The problem: How can an AI remember and reason over days-to-months while staying grounded in actual frames? Failed attempts: Just feeding more frames to big models caused memory overload; global search agents got lost. The gap: We needed a dataset that truly tests lifelong gaps and a method that turns endless video into well-managed memory. Real stakes: From assistive wearables to home robots and safety cams, AIs must remember what happened yesterday to act wisely today.

🍞 Hook: Imagine you’re making a scrapbook of a long trip—you don’t paste every photo. You pick highlights, label them, and add notes so you can quickly find what you need later.

🥬 The Concept in One Sentence: The key insight is to convert long, noisy video into a compact, updatable, language-based memory and use recursive tool calls to refine just the parts that matter for each question.

• Multiple Analogies:

  1. Librarian analogy: Instead of shelving every page of every book, the librarian writes smart summaries and a great index, then pulls the right books when asked.
  2. Detective analogy: The detective builds a case file (memory), then re-checks specific camera times (video tool) only when a clue demands it.
  3. Backpack analogy: On a long hike, you don’t carry your whole house—you pack essentials (summaries) and fetch from basecamp (full video) only if needed.

• Before vs After:

  • Before: End-to-end models stretched context windows and choked on too many frames; agents tried global searches and got lost.
  • After: ReMA summarizes first, plans next, then inspects precisely; it scales by thinking in loops and keeping memory tidy.

• Why It Works (intuition behind the math):

  • Lifelong streams have $T_{span} \gg T_{dur}$ (e.g., 2 months vs 10 hours; ratio ~72). Example: $T_{span}=2{\text{ months}}\approx 5{,}184{,}000\text{ s}$, $T_{dur}=72{,}000\text{ s}$; $T_{span}/T_{dur}\approx 72$.
  • If you process everything equally, irrelevant tokens swamp attention. Summarization compresses low-value redundancy; retrieval narrows focus; recursive inspection corrects mistakes.
  • Ref@N encourages finding the right time neighborhoods rather than pixel-perfect borders in giant videos, which stabilizes training and evaluation.

• Building Blocks (each with a Sandwich):

  1. Multimodal Lifelong Understanding 🍞 Hook: Think of following a friend’s adventures all month, not just one afternoon. 🥬 What it is: Understanding audio+video over long spans with big gaps and changing contexts. How it works: Track evolving state, connect far-apart clues, and keep identity consistent. Why it matters: Real life isn’t a continuous recording; you must reason across what you didn’t see. 🍞 Anchor: Remember what the streamer promised on Monday and check if it happened on Thursday.
  2. Observational Duration ($T_{dur}$) 🍞 Hook: Screen time vs. story time. 🥬 What it is: Total length of video you actually process, $T_{dur} = \sum l_i$. Example: $l=[10,20,30]$ s → $T_{dur}=60$ s. How it works: Add up each clip’s playback time. Why it matters: It tells you how much data the model truly sees. 🍞 Anchor: You watched 3 short clips totaling 1 minute.
  3. Physical Temporal Span ($T_{span}$) 🍞 Hook: Calendar time between first and last event. 🥬 What it is: Real-world coverage of the dataset, $T_{span} = (\tau_N + l_N) - \tau_1$. Example: first at 0 s, last at 300 s with 30 s length → $T_{span}=330$ s. How it works: From earliest start to latest end. Why it matters: Shows how many real gaps you must bridge. 🍞 Anchor: Clips are spread across a day even if you only watched minutes.
  4. Working Memory Bottleneck 🍞 Hook: Too many tabs open slows your computer. 🥬 What it is: Overfilling the model’s short-term space harms reasoning. How it works: Extra frames add noise, blow up caches, and dilute attention. Why it matters: More input can paradoxically reduce accuracy. 🍞 Anchor: Adding frames made some models worse at finding evidence.
  5. Global Localization Collapse 🍞 Hook: Looking for one paragraph in a library without an index. 🥬 What it is: Agents fail when searching raw video globally over huge timelines. How it works: Sparse signals + giant space = unstable, drifting search. Why it matters: Prevents reliable grounding in month-scale tasks. 🍞 Anchor: The agent keeps scanning irrelevant hours and misses the 5-minute clue.
  6. Dynamic Memory Management 🍞 Hook: Keep highlights, toss clutter. 🥬 What it is: Summarize, merge, and update memory to stay compact. How it works: Create notes per chunk, merge overlaps, retrieve and refine. Why it matters: Enables scaling to weeks and months. 🍞 Anchor: A clean index of events beats thousands of raw frames.
  7. ReMA (Recursive Multimodal Agent) 🍞 Hook: Plan, check, update, repeat. 🥬 What it is: An agent that builds a memory bank and iteratively inspects only what matters. How it works: Two phases—Perception (summarize) and Control (search memory, inspect video, update, answer). Why it matters: Focuses compute on useful moments and stays grounded. 🍞 Anchor: It answers and shows exact timestamps where the proof lives.
  8. Clue-Grounded Annotation 🍞 Hook: Show your work policy. 🥬 What it is: Labels include proof intervals for each QA. How it works: Human-verified clues; filters remove questions answerable by common sense alone. Why it matters: Forces genuine video reasoning over guessing. 🍞 Anchor: You can’t just say “four times”; you must point to all four spots.
  9. Ref@N 🍞 Hook: Grade by neighborhoods of time. 🥬 What it is: Bin-based overlap score: $\text{Ref@N} = \frac{|P \cap G|}{|P \cup G|} \times 100$. Example: Truth bins {5,6,7}, prediction {6,7} → $\frac{2}{3}\times 100 \approx 66.67$. How it works: Discretize time; compare sets of bins. Why it matters: Robust to tiny boundary errors in huge videos. 🍞 Anchor: Being off by one 5-minute bin isn’t a total fail.

In short, the aha is: turn endless video into a tidy, living memory you can query and refine—then show your receipts (timestamps).

At a high level: Input (ultra-long video + question) → Perception Phase (summarize into memory) → Control Phase (reason, retrieve, re-inspect) → Output (answer + grounded time intervals).

Step-by-step with Sandwich explanations for key steps and tools:

1. Perception Phase: Building the Memory Bank

• 🍞 Hook: Before a big exam, you don’t reread the entire textbook—you make summary cards.
• 🥬 What happens:
  • Segment the video into chunks of length ∆t (e.g., 5 minutes) and run MMInspect to create concise captions/notes.
  • Use MemManage to merge overlapping or redundant notes so the memory stays small but informative.
  • Store the results in a searchable Memory Bank.
  • Why this step exists: Without early summarization, the agent would drown in raw frames and hit the Working Memory Bottleneck.
• 🍞 Anchor: For a 3-hour livestream, the system creates about 36 five-minute notes, such as “00:35–00:40: streamer sings on subway in City A.”

1. Core Tools

• MMInspect (Visual Observation)
  • 🍞 Hook: Sometimes you rewatch just the exact scene you need.
  • 🥬 What it is: A tool that samples frames in a time range and asks a vision-language model to describe what happens.
    • How it works:
      1. Sample frames in [start, end].
      2. Generate local descriptions (with optional timestamps).
      3. Align times to the global timeline.
    • Why it matters: It turns pixels into text the agent can reason about and re-checks fine details when needed.
  • 🍞 Anchor: “At 01:12:10, the streamer boards a river cruise” becomes an indexed memory snippet.
• MemManage (State Consolidation)
  • 🍞 Hook: Combine duplicate sticky notes so your notebook doesn’t explode.
  • 🥬 What it is: A merger that replaces overlapping notes with a single, clearer summary.
    • How it works: Find overlaps, concatenate contents, write a compact combined version, and store that instead.
    • Why it matters: Keeps memory size stable and focused on high-entropy updates.
  • 🍞 Anchor: Two notes about the same subway singing get merged into one clean note with all timestamps.
• MemSearch (Retrieval & Aggregation)
  • 🍞 Hook: Use a great index to find the right chapter fast.
  • 🥬 What it is: A two-stage retrieval system that finds top-k relevant notes and groups them by time to summarize across events.
    • How it works: Vector search → rerank with an LLM → group by intervals → summarize per group → summarize across groups.
    • Why it matters: Answers often require multiple clips hours apart; this stitches them together.
  • 🍞 Anchor: To count subway songs across cities, the system pulls all “singing” notes and sums verified occurrences.

1. Control Phase: Recursive Reasoning Loop

• 🍞 Hook: Like solving a mystery—hypothesize, check evidence, update, repeat—until confident.
• 🥬 What happens:
  • The controller (an MLLM) reads the question and the current memory.
  • It chooses one action per step: Answer, MemSearch, or MMInspect.
  • After each action, it updates the memory, notes what changed, and decides the next move.
  • Why this step exists: Without iterative checks, the agent either guesses or scans blindly.
• 🍞 Anchor: For “Which transport modes at night in Chongqing?” the agent retrieves car/ship candidates, inspects precise times, discards day scenes, and confirms night car + ship.

1. Evaluation Metrics

• Answer Recall Accuracy (LLM-as-a-judge)
  • 🍞 Hook: A teacher gives partial credit when your idea is close but incomplete.
  • 🥬 What it is: A scoring scheme using an external LLM to grade answers.
    • How it works: The judge assigns $s \in \{0, 0.5, 1\}$ based on match quality. Example: If your answer is partly correct, $s=0.5$; a perfect match gets $s=1$; a wrong answer gets $s=0$.
    • Why it matters: Smooths out small wording differences while punishing hallucinations.
  • 🍞 Anchor: “Paris” vs “the capital is Paris” both score 1; “Lyon” scores 0.
• Ref@N (Temporal Grounding)
  • 🍞 Hook: Hitting the right neighborhood counts more in giant timelines.
  • 🥬 What it is: Bin-based intersection-over-union: $\text{Ref@N} = \frac{|P \cap G|}{|P \cup G|} \times 100$. Example: If truth bins are {10,11,12} and your prediction is {11,12,13}, then $\text{Ref@N} = \frac{2}{4}\times 100 = 50$.
    • Why it matters: Gives a fair, size-aware grounding score for ultra-long videos.
  • 🍞 Anchor: Off by one 5-minute bin still gets credit instead of zero.

1. Dataset Design and Splits

• 🍞 Hook: Practice on early chapters; test on new chapters and even a different book.
• 🥬 What it is: MM-Lifelong spans Day, Week, and Month scales with clue-grounded QA and careful splits.
  • How it works:
    1. Day-scale gaming (dense), Week-scale egocentric life (continuous days), Month-scale livestreams (sparse, across 51 days).
    2. Train on early month segments; validate on later segments; test on unseen Day/Week domains.
    3. Questions include Needle-in-a-Lifestream (find rare moments) and Multi-Hop Reasoning (combine disjoint clues).
  • Why it matters: Prevents time leakage and forces generalization across domain and time.
• 🍞 Anchor: You can’t memorize one room and pass the test; Week/Day tests are different subjects entirely.

1. Secret Sauce

• 🍞 Hook: Don’t carry the whole library—carry a superb index and a good plan.
• 🥬 What makes it clever:
  • Language-first memory turns pixel floods into searchable, compressible notes.
  • Recursive tool use focuses compute where it matters, correcting early biases.
  • Grounding with Ref@N and clue intervals forces honest, evidence-based answers.
• 🍞 Anchor: ReMA doesn’t just say “four”—it shows all four intervals and scales better as problems get longer.

Concrete mini-example (data flow):

• Input: 10-hour livestream, question: “How many times did the streamer sing that song on subways across three cities?”
• Perception: Segment into 5-minute chunks; MMInspect summarizes each; MemManage merges overlaps.
• Control: MemSearch finds all “song + subway” memories; MMInspect re-checks uncertain clips; memory updates count per city; Answer returns the number and timestamps.
• Output: “4 times; intervals: [01:23:10–01:24:05], [03:15:40–03:16:02], [06:49:30–06:51:44], [07:36:54–08:21:09].”

🍞 Hook: Imagine a marathon with checkpoints. It’s not enough to say you finished—you need to show which checkpoints you passed and when.

1. The Test: What did they measure and why?

• Answer Recall Accuracy: Can the model give the right final answer? The judge maps scores to $s \in \{0, 0.5, 1\}$. Example: If a model is partially right, $s=0.5$; fully right gives $s=1$; wrong is $s=0$.
• Ref@N: Can the model point to the correct time intervals? $\text{Ref@N} = \frac{|P \cap G|}{|P \cup G|} \times 100$. Example: If $|P\cap G|=2$ and $|P\cup G|=5$, then $\text{Ref@N}=40$.
• Why: Together, they test both brain (answer) and receipts (grounding) on Day, Week, and Month splits.

1. The Competition: Who did they compare against?

• End-to-end MLLMs: GPT-5, Qwen3-VL variants, Video-XL-2-8B, Eagle-2.5-8B, Nemotron.
• Agentic baselines: VideoMind-7B, LongVT-7B, DeepVideoDiscovery (DVD).
• Human reference scores for context.

1. The Scoreboard (with context):

• Month-scale validation (hardest due to sparsity):
  • ReMA: 18.62% accuracy; Ref@300 = 16.37%. This is like getting a solid B when many others are stuck at D for grounding.
  • GPT-5 (end-to-end): peaked near 15.25% accuracy but very low grounding (near 0–1%), meaning it can guess right sometimes but can’t show where in the video.
  • DVD: decent grounding (e.g., Ref@300 = 4.48% on val) compared to other agents but still far behind ReMA.
• Week and Day tests (OOD domains):
  • ReMA remains best overall with 16.75% accuracy and 11.51% Ref@300 on Day, and 18.82% accuracy and 16.37% Ref@300 on Week.
  • Other agents and end-to-end MLLMs trail, especially on grounding.

1. Surprising Findings:

• More frames can hurt end-to-end MLLMs: performance oscillates and may drop as context grows—classic Working Memory Bottleneck.
• Agent recursion rounds help: ReMA improves with more reasoning rounds up to a sweet spot (~3–5), then saturates. That’s like studying in short, focused sprints.
• Finer perception granularity (smaller ∆t) yields better results: using 2-minute chunks beats feeding a “full video” chunk, which collapses accuracy and grounding. Smart chunking > Big chunking.
• Multimodal controllers (e.g., GPT-5) plan and tool-call better than text-only controllers, even for text-space decisions.
• LLM-as-a-judge aligned closely with humans (F1 ~99% for GPT-5 as judge), showing reliable automated grading.

1. What it means:

• End-to-end alone isn’t enough at lifelong scales; without dynamic memory, bigger context just adds noise.
• Building a language-centric, persistent memory and reasoning recursively is a strong path forward.
• ReMA’s gains on grounding prove it doesn’t just guess—it finds the right places in time.

🍞 Anchor: Think of ReMA as the student who not only answers the history question but also flips the textbook to the exact pages that prove it—while classmates either guess or look in the wrong chapters.

1. Limitations (honest assessment)

• Single primary subject per scale: one game character, one camera wearer, one streamer. Great depth but limited subject diversity.
• Unobserved periods: The dataset tests cross-gap reasoning but doesn’t yet model complex interactions between off-camera events and on-camera changes.
• Tool availability: ReMA assumes access to a capable MLLM, embedding store, and retrieval pipeline, which may be heavy for some deployments.
• Judge dependency: Accuracy uses LLM-as-a-judge; while validated against humans, it remains an approximation.

1. Required Resources

• Storage and compute for pre-processing long videos into memory (vector DB like FAISS, embeddings, and an MLLM for inspection).
• A controller model with stable tool-use to avoid premature termination.
• Time to tune perception granularity (∆t) and recursion depth for a given domain.

1. When NOT to Use

• If the task is short, dense, and fully visible (e.g., a 30-second clip), end-to-end MLLMs may be simpler and faster.
• If precise, continuous pixel-level tracking is needed without text summaries (e.g., micro-level motion studies), direct video models might be preferable.
• If external world knowledge dominates (e.g., famous sports outcomes), results can be contaminated by parametric memory—closed-book visual tests are safer.

1. Open Questions

• How to scale to year-level timelines without contamination from public knowledge?
• How to explicitly model the impact of off-camera intervals on future states?
• Can we learn when to store, merge, or forget memories end-to-end (learned memory policies)?
• How to intertwine visual grounding with symbolic knowledge graphs to fight concept drift gracefully?
• Can on-device agents maintain privacy while still building strong persistent memory?

Bottom line: ReMA shows a promising direction, but broader subjects, cleaner disentangling of off-camera effects, and even more scalable memory policies remain exciting frontiers.

1. Three-sentence summary

• This paper introduces MM-Lifelong, a long-horizon multimodal dataset that separates what you watch from how much real time passes, forcing models to reason across big temporal gaps.
• It identifies two core failure modes in today’s systems (Working Memory Bottleneck and Global Localization Collapse) and proposes ReMA, a recursive agent with dynamic memory management.
• ReMA substantially improves both answering and grounding across day, week, and month scales, showing that memory-centric agents can break through the long-context ceiling.

1. Main Achievement

• Turning raw, endless video into a living, language-based memory and using recursive tool calls to verify just the right moments—resulting in significantly better grounded reasoning at lifelong scale.

1. Future Directions

• Expand subjects and domains while preserving non-Googleable, clue-grounded tasks.
• Learn memory policies end-to-end (what to store, merge, and forget) and integrate structured knowledge graphs.
• Develop richer evaluations for off-camera effects and finer intra-clip dynamics.

1. Why Remember This

• It reframes long-video understanding: don’t just extend context—engineer persistent, active memory and prove answers with timestamps. That mindset is key to building AI that can live, remember, and reason alongside us over months and beyond.