Animate Any Character in Any World

🍞 Hook: Imagine you’re directing a school play. You already have a stage (the world) and actors (the characters). You want to tell each actor what to do, where to stand, and where the audience should look. But your actors and stage keep changing shape and positions by themselves—chaos!

🥬 The World Before: In AI video generation, we had two main paths. One created beautiful, explorable 3D stages but with no active actors (static world models). The other controlled a single moving thing—like a car or character—but the rest of the world was mostly a painted backdrop you couldn’t really direct (controllable-entity models). That meant you could either explore a world without life, or you could move a character with limited actions in a world that didn’t listen to you.

🍞 Anchor: Think of video tools that can fly you through a 3D home tour (world but no actors), versus a game replay tool that only lets you move a car along a track (actor but limited control). Neither lets you stage a full play with many kinds of actions inside your chosen set.

🍞 Hook: You know how when filming a skit with your phone, you want your friend to look the same in every shot and the room to stay the same too? That’s called consistency.

🥬 The Problem: People wanted to pick their own character and their own 3D space, then say commands like “wave,” “pick up the phone,” or “run left,” and also control the camera—while keeping the character’s look and the room layout stable across many clips. Past methods often changed the character’s face/clothes, bent the room’s geometry, or forgot what happened a few moments ago.

🍞 Anchor: If the AI forgets, it’s like filming a superhero who suddenly changes costumes and teleports between shots.

🍞 Hook: Imagine making a stop-motion movie with LEGO. If every new frame jiggles the background or swaps a minifigure’s head, the story breaks.

🥬 Failed Attempts: Many tried to fix this by: (1) baking camera motions into special math codes and injecting them into models, which can be brittle and less intuitive; (2) using text or single images to “remind” the model of who the character is, which often fails when turning around or moving far; and (3) trying to remember the past with lightweight memory tricks, which still drift over long scenes.

🍞 Anchor: It’s like telling a camera operator, “Please move in a circle,” by giving them a complicated formula rather than simply walking the path in the actual room.

🍞 Hook: Imagine you have a perfect map of your stage and clear photos of your actor from all sides. Now it’s much harder to get lost.

🥬 The Gap: We needed a method that (a) locks down the environment in real 3D so camera moves are precise, (b) locks down the character’s identity from all angles, (c) accepts plain-language actions, and (d) stitches clips smoothly over time.

🍞 Anchor: That’s the space AniX fills—using the 3D world as a grounded memory, the character’s multi-view photos as an identity badge, and a strong video model as the animator.

🍞 Hook: Think of texting a friendly robot director: “Follow the runner,” “Now wave,” “Now orbit the camera,” and it just does it—without forgetting what the room or actor looks like.

🥬 Real Stakes: This matters to creators (film, game prototyping, education videos), to designers previewing characters in virtual sets, and to anyone who needs consistent, controllable long videos. It reduces reshoots, keeps brand characters on-model, and lets non-experts direct complex shots with simple language.

🍞 Anchor: A teacher could place a historical figure into a virtual museum and say, “Walk to the dinosaur exhibit and point at the T-Rex,” getting a clean, consistent tutorial video in minutes.

— New Concepts Introduced —

🍞 Hook: You know how Google Maps gives you a solid sense of where buildings and roads are, no matter how you rotate the view? 🥬 3D Gaussian Splatting (3DGS): It’s a way to represent a 3D scene using lots of soft blobs (Gaussians) that capture color and shape. How: (1) Place many colorful “blobs” in 3D, (2) For any camera view, project blobs to the screen, (3) Blend them to render the image fast. Why it matters: It gives a stable, editable 3D world so camera moves are accurate and the room doesn’t wobble. 🍞 Anchor: Like building a room from thousands of tiny balloons; walk around it with your camera and it still looks like the same room.

🍞 Hook: Imagine making a flipbook; each new page continues the story from the last page. 🥬 Temporal Coherence: It means frames stay smooth and consistent over time. How: (1) Use past frames as guidance, (2) Keep identity and layout stable, (3) Avoid sudden jumps. Why it matters: Without it, videos jitter and characters morph. 🍞 Anchor: A running character shouldn’t teleport or swap shirts between frames.

🍞 Hook: Think of cooking with both pictures and a written recipe. 🥬 Multi-Modal Inputs: The model uses several input types together—text (commands), images (character views), and 3D scene renders. How: (1) Encode each input into tokens, (2) Feed them jointly into the generator, (3) The model learns how they connect. Why it matters: Depending on text alone can be vague; visual and 3D cues remove confusion. 🍞 Anchor: Saying “orbit the camera” plus showing the 3D map makes the camera path unambiguous.

🍞 Hook: You know how voice assistants follow spoken commands? 🥬 Natural Language Control: Users write plain instructions like “run forward” or “play guitar.” How: (1) An encoder turns text (and character refs) into meaning vectors, (2) The generator uses them to guide motion, (3) Actions map to movement or gestures. Why it matters: No sliders or code needed—just words. 🍞 Anchor: Type “turn left and wave,” and the character does it in the given room.

🍞 Hook: Telling a story page by page keeps the plot straight. 🥬 Conditional Autoregressive Video Generation: The model makes the next video clip conditioned on previous clips, the scene, the character, and the text. How: (1) Take past clip tokens, (2) Add scene and character tokens, (3) Add instruction tokens, (4) Generate the next clip. Why it matters: This keeps long stories consistent over many steps. 🍞 Anchor: First clip: “run forward,” next: “now stop and salute,” and the same actor stays in the same room.

🍞 Hook: Imagine you are a movie director with a real 3D set and a reliable actor photo sheet (front, left, right, back). You whisper, “Run forward.” The camera glides behind, and everything looks right—every time.

🥬 The Aha! Moment (one sentence): Use the user’s 3D scene as a grounded spatial memory and multi-view character images as an identity lock, then guide a strong pre-trained video generator with these visual conditions and natural-language commands, while rendering a 3D-guided camera video so the viewpoint is exact.

🍞 Anchor: You load a 3DGS city, pick a hero (front/left/right/back photos), type “run to the right,” and AniX outputs a smooth clip with the right outfit, right city, and right camera path.

— Three Analogies —

🍞 Hook: Think of a puppet show where the stage is fixed, the puppet’s costume sheet is pinned backstage, and your script tells the puppeteer what to do. 🥬 Analogy 1: The 3DGS is the stage, the multi-view character is the costume guide, text is the script, and the video model is the skilled puppeteer. Why it works: The puppeteer always checks the stage map and costume sheet before moving. 🍞 Anchor: When you say “bow,” the puppet bows center-stage without changing outfits.

🍞 Hook: Consider GPS for filming: you mark a camera route on a 3D map and follow it precisely. 🥬 Analogy 2: Instead of hoping the model understands camera math, AniX renders a short “guide video” from the 3D world along your path. The generator then mirrors that motion. Why it works: Concrete visual guidance beats abstract codes. 🍞 Anchor: You choose “circle around the character,” and the output really circles—no drift.

🍞 Hook: Imagine a sports replay system that keeps the same player identity as you switch angles. 🥬 Analogy 3: Multi-view images pin down the player’s appearance from all sides, so turning or sprinting doesn’t distort identity. Why it works: The model has references for every angle. 🍞 Anchor: Whether facing front or back, the jersey and hairstyle stay correct.

— Before vs After —

• Before: Worlds with no actors, or actors with only simple moves, and wobbly camera control; identity and scene drifted across time.
• After: Any character in any user-chosen 3D scene, open-ended actions via text, precise camera paths by rendering from the 3D scene, and consistent long-horizon clips.

— Why It Works (intuition) —

• Grounding: The 3DGS scene acts like a solid map; the model doesn’t guess geometry—it reads it.
• Identity Lock: Multi-view character images provide a stable, angle-aware identity anchor.
• Memory: Generating clip-by-clip while conditioning on the previous one keeps stories coherent.
• Gentle Fine-Tuning: A small set of locomotion and camera samples “teaches better movement” without erasing the model’s broad knowledge, like a dance class for a well-read actor.
• Visual Camera Guide: Rendering from the 3DGS along the exact path removes ambiguity in camera control.

— Building Blocks (with Sandwich explanations) —

🍞 Hook: Like shrinking a movie into a tiny comic strip you can process fast. 🥬 Video VAE Tokens: The model compresses videos and images into tokens (small latent codes). How: (1) Encode frames into compact tokens, (2) Do generation/denoising on tokens, (3) Decode back to frames. Why it matters: It’s faster and more stable than working on full pixels. 🍞 Anchor: A 93-frame clip becomes a handful of tokens the model can handle efficiently.

🍞 Hook: Think of cleaning a foggy picture until it becomes sharp. 🥬 Diffusion with Flow Matching: The model starts from noisy tokens and learns how to “flow” them toward clean video tokens. How: (1) Mix noise with targets, (2) Predict the velocity to denoise, (3) Repeat a few steps to reach a clean clip. Why it matters: It produces sharp, controllable videos. 🍞 Anchor: From “static on a TV” to a clear shot of your running character.

🍞 Hook: Imagine a big team meeting where everyone shares notes. 🥬 Multi-Modal Diffusion Transformer: A Transformer reads text, scene, character, and prior-clip tokens together to plan the next frames. How: (1) Concatenate sequences, (2) Let attention share info across all inputs, (3) Output denoised targets. Why it matters: It fuses commands with visuals reliably. 🍞 Anchor: The model hears “turn left,” sees the 3D map, checks the character views, and turns left in the right spot.

🍞 Hook: Tape an “X” on the floor so actors know where to stand. 🥬 Character Anchor (Mask): A simple bounding box tells the model where the character belongs. How: (1) Provide a per-frame (train) or fixed (inference) mask, (2) Fuse it with tokens, (3) Keep the person separate from the background. Why it matters: Prevents the character from melting into scenery or drifting. 🍞 Anchor: The hero stays in the marked zone as they move.

🍞 Hook: A photo sheet of an actor’s front, left, right, and back. 🥬 Multi-View Character Inputs: Four views lock appearance from all sides. How: (1) Encode each view, (2) Add special position tags so the model knows which is which, (3) Use all of them during generation. Why it matters: Keeps identity consistent when turning. 🍞 Anchor: The jacket logo and hair match from any angle.

🍞 Hook: Practice sprints to run better in any sport. 🥬 Light Post-Training (LoRA): The giant video model learns better motion on a small locomotion set using LoRA adapters. How: (1) Freeze most weights, (2) Add small trainable layers, (3) Fine-tune on basic runs/follows. Why it matters: Motion improves without forgetting general knowledge. 🍞 Anchor: After a short workout, the model runs more naturally while still knowing many other actions.

🍞 Hook: Make a trailer for the camera to follow. 🥬 Camera Control via 3DGS Render: Render a short scene-only video along your chosen camera path and feed it as a condition. How: (1) Plan the path (orbit/follow/stationary), (2) Render from the 3D scene, (3) Condition the generator. Why it matters: Viewpoints match exactly. 🍞 Anchor: “Circle the character” results in a true circle around them.

🍞 Hook: Build a story in chapters. 🥬 Auto-Regressive Clips with Noisy Memory: Feed part of the previous clip (slightly noised during training) as context. How: (1) Split tokens into ‘past’ and ‘to-generate,’ (2) Add a little noise to close the train/test gap, (3) Generate the next chunk. Why it matters: Long videos flow without forgetting. 🍞 Anchor: Clip 2 continues seamlessly from Clip 1.

🍞 Hook: Boil soup faster without changing the recipe. 🥬 DMD2 Distillation (Fast 4-Step): Teach a small fast student to mimic a slower teacher model. How: (1) Freeze teacher, (2) Train student with guidance, (3) Keep quality while cutting steps. Why it matters: Much quicker generation with tiny quality loss. 🍞 Anchor: The 4-step model is ~7.5× faster yet stays sharp.

At a high level: Inputs (3DGS scene, multi-view character images, text instruction, prior clip) → Encode into tokens → Fuse tokens and add a rendered scene-only camera guide → Multi-Modal Diffusion Transformer denoises latent video tokens (Flow Matching) → Decode to video clip → Repeat for long-horizon stories.

Step-by-step (what, why, example):

1. User configuration (what): You choose a 3DGS scene, load a character’s front/left/right/back images, place a virtual camera, and set a character anchor (a bounding box). Why: This locks down the world, the actor’s identity, where they appear, and where the camera starts. Example: A 3DGS street, a soccer player’s 4 views, a following camera, and an anchor in the right lane.

2. Instruction parsing and camera path (what): You type a command. AniX classifies it as locomotion (move), gesture, object-centric action, or camera behavior. It then picks a camera path: follow for locomotion, stationary for gestures/object actions, or a special path (e.g., orbit) for camera behavior. Why: Different actions fit different camera strategies, improving clarity and control. Example: “Run forward” sets a forward-follow path; “wave” keeps the camera still; “orbit the character” sets a circle.

3. Render the scene-only camera guide (what): Given the path, AniX renders a short video from the 3DGS scene without the character. Why: This visual guide precisely encodes the camera motion (no guessing). Example: A 3-second orbit video of the empty plaza.

4. Tokenization with a Video VAE (what): Encode the target video space, the scene guide, the character anchor mask, the multi-view character images, and the text into compact tokens. Why: Tokens make large-scale generation efficient and stable. Example: 93 frames compress to manageable latent tokens.

5. Fuse conditions (what): Combine scene tokens and mask tokens directly with the noisy target tokens via a small projector; concatenate text tokens and character tokens to the sequence. Why: Scene+mask stabilize layout and placement; text and multi-view images steer actions and identity. Example: The model sees “run forward,” the 3D street layout, the anchor box, and the soccer player’s four views.

6. Denoising via Flow Matching in a Diffusion Transformer (what): Start from noise-laced target tokens and predict the clean “velocity” that moves them toward the true video. Why: Flow Matching provides a smooth training target and stable learning. Example: From static to crisp footage of the player sprinting down the street.

7. Positional treatment (what): Apply 3D rotary positional embeddings (3D-RoPE) to video tokens; apply temporal shifts to each character view so the model distinguishes front/left/right/back. Why: The model needs to know where and when tokens belong. Example: “Front” view gets one shift, “left” another, etc.

8. Auto-regressive mode (what): Split the target clip; feed the first quarter as ‘preceding tokens’ and generate the rest. Add a little noise to preceding tokens during training to match inference conditions. Why: Ensures long-horizon consistency even when the model conditions on its own past outputs. Example: Clip 2 uses the tail of Clip 1 as context and continues the run into a turn.

9. Training data pre-processing (what): Use GTA-V gameplay to collect short single-action clips; segment out the character; inpaint the background to get a clean scene video; label the action text; render 4 character views. Why: Clean supervision isolates the scene, the actor, and the action. Example: A “run left” clip with a clean, character-removed street scene and its mask.

10. Gentle fine-tuning (what): Start from a strong pre-trained video generator; freeze most parts; apply LoRA adapters to a few layers; train briefly on basic locomotion and camera behaviors (360p/720p). Why: Boost motion realism and camera following without breaking general knowledge. Example: After 5k steps, the base model moves more naturally.

11. Acceleration (what): Use DMD2 to distill 30 denoise steps down to 4 steps. Why: Big speedups with minor quality loss. Example: 121s down to ~21s for a 93-frame 360p clip on one H100 GPU.

What breaks without each step:

• No scene-only guide: Camera may drift or not follow the intended path.
• No multi-view character: Identity fails when turning; clothes/hair change.
• No anchor mask: The character blends into the background or drifts.
• No prior-clip conditioning: Long scenes forget the past and lose coherence.
• No gentle fine-tuning: Motions look stiffer; following actions are less accurate.

Worked example (actual data flow):

• Inputs: 3DGS city; 4 images of a runner; text: “The character is running forward.”
• Path: follow forward; render scene-only guide (empty street moving forward).
• Tokens: Encode text with character images (LLaVA-based), encode guide video, mask, and target tokens.
• Denoise: Transformer fuses all conditions and denoises via Flow Matching.
• Output: A clip of the same runner sprinting forward through the same city, camera smoothly following.

Secret sauce:

• Geometric grounding via 3DGS rendering for camera control (simple, explicit, robust).
• Multi-view identity locking + anchor masks for stable characters in stable worlds.
• Tiny, smart fine-tuning that sharpens motion without losing generalization.
• Auto-regressive conditioning with noise augmentation that keeps long stories coherent.

— Additional Sandwich Concepts —

🍞 Hook: Add shoulder patches without replacing the whole uniform. 🥬 LoRA Fine-Tuning: Small adapter layers tweak a big model. How: Insert low-rank modules into attention/MLP layers and train only them. Why it matters: Efficient improvements without catastrophic forgetting. 🍞 Anchor: The model keeps its broad skills but moves more naturally.

🍞 Hook: Teach a new sprinter by showing them a pro runner. 🥬 DMD2 Distillation: A fast student copies a slow teacher’s behavior. How: Freeze teacher, train student and a scoring helper, reduce steps to 4. Why it matters: Much faster generation with similar quality. 🍞 Anchor: The 4-step student runs almost as well as the 30-step teacher.

🍞 Hook: Think of a talent show. You don’t just look at a costume; you also judge the dance, how well they follow instructions, and whether they keep the rhythm for the whole song.

🥬 The Test: AniX is judged on (a) visual quality (does the world look stable and nice?), (b) character consistency (does the character stay the same person?), (c) action controllability (does the model follow the command?), and (d) long-horizon coherence (do clips connect smoothly over time?). Why: Real usefulness needs all four. 🍞 Anchor: A beautiful but disobedient dancer fails; a precise but jittery stage fails too.

1. Visual quality and camera control (WorldScore):

• Setup: Compare against large video generators (CogVideoX, HunyuanVideo, Wan family) and world models (DeepVerse, Hunyuan-GameCraft, Matrix-Game-2.0).
• Metrics: WorldScore groups: Controllability, Quality, Dynamics. Static and Dynamic overall scores summarize performance.
• Result: AniX hits ~84.6 on WorldScore—like scoring an A+ while many others score B-range. It excels in camera following (thanks to scene-rendered guides), subject/photometric/style consistency, and maintains strong motion metrics.
• Meaning: The scene stays faithful to the user’s 3D world, and the camera does what you asked.

1. Action control and generalization:

• Setup: Four seen locomotion actions (“run forward/left/right/backward”) plus 142 novel actions (gestures and object interactions) not in the fine-tuning set.
• Measures: Human success rate plus frame-wise CLIP text-image similarity.
• Result: 100% success on locomotion; ~80.7% on rich, unseen actions. CLIP scores improve over the base model.
• Meaning: Even with tiny fine-tuning focused on running, the big pre-trained model’s broad skills remain—and are easier to control now.

1. Character consistency on novel characters:

• Setup: 30 new characters, evaluate DINOv2 and CLIP similarity between generated character crops and reference views.
• Result: AniX leads. Multi-view inputs matter: front only < front+back < all four views (best). Anchors matter: per-frame masks (train) and a consistent anchor (test) give top scores.
• Meaning: The character stays on-model while turning and moving.

1. Long-horizon generation with visual conditions:

• Setup: Auto-regressive mode across clips; compare full-visual conditioning against variants with text-only character or text-only scene.
• Result: Full visual conditions keep quality and identity from fading over time; lighter variants drift and degrade.
• Meaning: The 3DGS and multi-view character act like strong memory aids.

1. Speed vs quality (distillation):

• Setup: 30-step original vs 4-step distilled vs 4-step original (no distill).
• Result: Distilled 4-step is ~7.5× faster with only small drops in DINOv2 and CLIP-Aesthetic; the undistilled 4-step degrades more.
• Meaning: Smart distillation keeps quality while slashing latency.

Surprising findings:

• Small, simple fine-tuning (mostly running and camera follow/orbit) significantly boosts motion realism and control without harming general knowledge—like a brief coaching session for a well-trained athlete.
• Rendering the camera path as a scene-only guide works better and more intuitively than encoding camera math into special vectors.
• Hybrid data (adding some real-world locomotion clips, tagged as “real” vs “rendered”) increases photorealism for real people (better DINOv2/CLIP), showing style disentanglement is possible.

— Sandwich notes for key metrics —

🍞 Hook: Like grading a music performance on pitch, rhythm, and stage presence. 🥬 WorldScore: A bundle of metrics for controllability, quality, and motion dynamics. How: (1) Compute multiple sub-scores, (2) Aggregate into static and dynamic grades, (3) Compare across models. Why it matters: Balanced judging prevents overfitting to one aspect. 🍞 Anchor: AniX doesn’t just look good—it listens and moves well, too.

🍞 Hook: Matching a sketch to a photo. 🥬 CLIP and DINOv2 Similarity: They measure how well images match text (CLIP) or identity/appearance (DINOv2). How: (1) Crop the character, (2) Compare features to references, (3) Average over frames. Why it matters: Quantifies identity faithfulness and instruction alignment. 🍞 Anchor: If the shirt logo and hairstyle match the references, the scores are high.

🍞 Hook: Even the best directors have limitations—sometimes the stage is small, the lights are dim, or the cameras are heavy.

🥬 Limitations:

• Data dependence: The output quality depends on the 3DGS scene and character references. Poor scenes (blurry, sparse points) or low-quality character views reduce fidelity.
• Action understanding: Natural language parsing covers many actions but can still misread unusual or ambiguous commands.
• Physics and interactions: Complex physical realism (soft-body cloth, precise hand–object grasping) and tricky occlusions may not be perfect.
• Single main actor: The method focuses on one controllable character; multi-agent choreography is not the primary target.
• Compute: High-quality, high-resolution clips still need strong GPUs, especially without the distilled model.

🍞 Anchor: If you give a fuzzy room scan or tiny character photos, it’s like shooting a movie with a smudged lens and unclear costume notes.

🥬 Required resources:

• Hardware: At least one modern GPU for inference; faster multi-GPU setups for higher resolutions or distillation.
• Assets: A decent 3DGS scene (can be auto-generated) and four clear character views.
• Software stack: Video VAE encoders, LLaVA for text+image encoding, and the AniX diffusion transformer.

🍞 Anchor: Think of it as needing a good stage, a complete costume sheet, and a sturdy camera.

🥬 When not to use:

• If you need exact physics or multi-character fights with contact-rich interactions.
• If your scene must be photogrammetry-grade accurate without any artifacts.
• If you have no way to provide usable scene or character references.

🍞 Anchor: For a scientific biomechanics demo with precise forces, this is not a physics simulator.

🥬 Open questions:

• Rich object interaction: How far can we push hand–object manipulation using only language and current visual guidance?
• Multi-actor scenes: What’s the cleanest way to add two or more controllable characters with collision awareness?
• Memory scaling: Can longer, movie-length sequences remain consistent with stronger, learned world memories?
• Camera semantics: Beyond paths, can users direct cinematography styles (rack focus, dolly zoom) via more expressive controls?
• Data diversity: How much real-world mixed data is ideal to boost photorealism without losing generalization?

🍞 Anchor: Next steps look like expanding the cast, deepening interaction, and teaching the camera more storytelling tricks.

Three-sentence summary: AniX lets you animate any character inside any 3D world with simple text commands while keeping both the character’s identity and the environment’s layout stable. It grounds camera motion by rendering a scene-only guide from the user’s 3DGS and locks identity using multi-view character images, then uses a strong pre-trained video model fine-tuned lightly for better motion. Across benchmarks, AniX shows higher visual quality, control accuracy, and long-horizon coherence than existing baselines, and it runs much faster after distillation.

Main achievement: Turning a powerful but general video generator into a precise, grounded animator by combining (1) 3DGS-based camera guidance, (2) multi-view identity conditioning, (3) a small but effective locomotion fine-tuning, and (4) auto-regressive memory.

Future directions:

• Multi-character interactions with collision and social behaviors.
• Richer object manipulation and tool use guided by text.
• Cinematography-aware control (focus pulls, zoom language, style presets).
• Larger, hybrid datasets that blend rendered and real footage for peak realism.
• Stronger long-term world memory for scene persistence across many clips.

Why remember this: AniX shows a practical recipe for “make a movie with words” that respects the real 3D layout and the actor’s identity. Its simple, grounded camera control and multi-view character anchoring are easy to apply elsewhere. And it proves that a small, smart fine-tuning can unlock better motion without sacrificing the breadth of a large foundation model.