Horizon-LM: A RAM-Centric Architecture for LLM Training

🍞 Hook: Imagine you’re trying to fit a giant, heavy encyclopedia into a tiny school backpack. It doesn’t matter how strong you are if the book simply can’t fit. That’s what training huge AI models often feels like today.

🥬 The World Before: Most large language models (LLMs) were trained in a GPU-centric world. GPUs (the fast calculators) were expected to hold the entire model, the training map (called an autograd graph), and all the extra notes (optimizer states and activations). As models grew from billions to hundreds of billions of parameters, single GPUs ran out of room. The common fix? Use many GPUs and spread the model across them with clever sharding and communication tricks.

🍞 Anchor: Think of spreading a long group project across a dozen classmates so each keeps some pages at their desk. It works—but needs a lot of desks and complicated passing of pages.

🍞 Hook: You know how moving boxes across the street is easier if you use a handcart and don’t try to carry them all at once? The AI world tried a version of that too.

🥬 Failed Attempts: Systems like ZeRO-Offload and ZeRO-Infinity tried to help by offloading (temporarily moving) some model pieces from GPU to CPU or disk when not in use. This extended capacity, but they still treated the GPU as the “home base.” The GPU still needed a long-lived model and the full training graph, and the CPU worked more like a messy storage closet than a tidy library. Memory grew in unpredictable ways: not just with model size, but with the tangle of runtime buffers, graphs, and communication metadata.

🍞 Anchor: It’s like borrowing your neighbor’s closet to store winter coats but still trying to change outfits in your tiny bedroom. You’re always shuffling clothes and tripping over hangers.

🍞 Hook: Picture two types of space: a small, super-fast desk (GPU) and a big, roomy bookshelf (CPU RAM). The trick is to stop insisting that everything must live on the small desk.

🥬 The Problem: Fine-tuning and adapting large models is increasingly a memory problem, not a raw-compute problem. You often just need all the parameters and optimizer states present—but not necessarily on the GPU at the same time. However, when training systems keep the GPU as the main owner and keep a giant autograd graph, you are forced into multi-GPU clusters and unpredictable CPU memory bloat.

🍞 Anchor: If you only need to look at one chapter at a time, why bring the entire bookshelf to your desk?

🍞 Hook: Imagine if the bookshelf (CPU RAM) were the real home for your books, and the desk (GPU) were just where you open one book at a time to read.

🥬 The Gap: We needed a system that treats host memory as the authoritative parameter store and makes the GPU a temporary, stateless worker. We also needed a clear, predictable memory recipe: host memory should scale like the model size, and GPU memory should depend only on the biggest chapter (the widest layer), not the whole book.

🍞 Anchor: You pull one chapter, read it on the desk, put it back, and pull the next. Your desk never gets crowded, and your bookshelf stays organized.

🍞 Hook: Think about a relay race where a runner hands off a baton without stopping. If everyone overlaps their moves perfectly, the race never slows.

🥬 Bandwidth and Scheduling: To keep training fast, you must constantly overlap moving weights to the GPU, computing, and sending gradients back. If you do these one at a time, the GPU waits and wastes time. A double-buffered, multi-stream pipeline turns training into a smooth, continuous flow so the GPU never idles.

🍞 Anchor: While you’re reading page 5, your friend is already flipping to page 6 and getting ready to hand it to you. No pauses.

🍞 Hook: Why should you care? Because being able to fine-tune giant models on a single GPU makes advanced AI tools more available to schools, labs, and startups—not just giant companies.

🥬 Real Stakes: Universities and small teams often can’t get many high-end GPUs. If we can train 70B–120B parameter models on one GPU with lots of RAM, many more people can customize and align powerful models for their specific tasks (teaching, medicine, law, science) without renting big clusters.

🍞 Anchor: It’s like giving every classroom a super microscope that used to be only in national labs.

— New Concepts (Sandwich Style) —

🍞 Hook: You know how your phone has fast but small memory and your computer might have bigger storage? GPUs and CPUs are like that too. 🥬 Concept: GPU-centric training means the GPU keeps the whole model and the training map (autograd) in its small, fast memory.

• How it works: 1) Copy model to GPU, 2) Build a big graph of all compute steps, 3) Run forward and backward, 4) Keep lots of activations in GPU memory.
• Why it matters: It’s fast when it fits, but breaks when models are too big for GPU memory. 🍞 Anchor: It’s like trying to store all your textbooks on a tiny desk—you run out of space fast.

🍞 Hook: Imagine a treasure map that shows every step from start to finish. 🥬 Concept: Autograd graphs are that map for training—recording operations so gradients can flow backward.

• How it works: 1) Track ops in forward pass, 2) Keep activations, 3) Use chain rule backward.
• Why it matters: The map is big. If you can’t store it, you can’t backtrack efficiently. 🍞 Anchor: If you can’t keep the clues, you have to re-walk parts of the path.

🍞 Hook: Sometimes you move stuff around the house to make room temporarily. 🥬 Concept: Offloading moves tensors between GPU, CPU, and disk to save GPU memory.

• How it works: 1) Detect pressure, 2) Ship some tensors out, 3) Bring them back when needed.
• Why it matters: Helps for a while, but still assumes the GPU is the boss, so complexity balloons and memory grows unpredictably. 🍞 Anchor: You stuffed coats in the hallway closet, but your bedroom is still cramped for changing.

🍞 Hook: You know how a library stores all books on shelves, and a study table shows just the one book you’re reading right now? That’s the whole trick here.

🥬 The “Aha!” in one sentence: Make CPU RAM the official home for all model parameters and optimizer states, and use the GPU only as a temporary, per-layer calculator with fast in-and-out streaming.

🍞 Anchor: Keep the bookshelf (RAM) neat and full; use the table (GPU) to open one chapter at a time.

— Multiple Analogies —

1. Kitchen: Pantry (CPU) stores ingredients; stove (GPU) only holds the pan you’re cooking with now. You fetch, cook, and put things back—never crowding the stove.
2. Theater: Director (CPU) keeps the script and calls scenes; actors on stage (GPU) perform one scene at a time; props are brought in just-in-time and whisked away.
3. Relay Race: One runner is sprinting (GPU compute) while the next baton (weights) is already on the way; finished batons (gradients) move back to the coach (CPU) without stopping the race.

— Before vs After —

• Before: GPU owns full models and big graphs; scaling needs many GPUs and creates messy, unpredictable host memory.
• After: CPU owns all parameters and optimizer states; GPU holds only the current layer’s data briefly; memory stays clean, predictable, and scales with model size, not runtime clutter.

— Why It Works (Intuition, no equations) —

• If you only bring one layer’s weights to the GPU at a time and send gradients back right away, the GPU memory needed depends on the largest single layer, not on the whole model. That frees you from depth limits.
• If you pack each layer’s weights contiguously and stream them with big, single transfers, the copy is fast and easy to overlap with computing.
• If you recompute small parts during backprop instead of storing everything, you save tons of memory for a tiny compute cost.
• If the CPU does the optimizer updates, the GPU never has to keep FP32 optimizer states; host RAM holds them predictably.

— Building Blocks (Sandwich Style) —

🍞 Hook: You know how you keep your important files in one folder so nothing gets lost? 🥬 Concept: Memory-centric training means organizing training around RAM first, not GPU memory.

• How it works: 1) All weights/optimizers live in RAM, 2) GPU streams one layer at a time, 3) Gradients go straight back, 4) CPU updates.
• Why it matters: Predictable, model-proportional memory so you can plan capacity. 🍞 Anchor: A tidy bookshelf beats a crowded desk.

🍞 Hook: Sometimes you use a rough sketch and a detailed diagram together. 🥬 Concept: Mixed BF16/FP32 precision stores weights/gradients in BF16 (2 bytes) and optimizer moments in FP32 (8 bytes) on the CPU.

• How it works: 1) Keep big stuff compact (BF16), 2) Keep precise state (FP32) for stable updates, 3) Convert as needed for compute.
• Why it matters: Saves memory while preserving training stability. 🍞 Anchor: Use a small notebook for notes (BF16) and a big sturdy binder for master records (FP32).

🍞 Hook: Think of the CPU as the coach and the GPU as the sprinter. 🥬 Concept: CPU-master, GPU-template model makes CPU the owner and GPU a reusable, stateless template runner.

• How it works: 1) CPU stores everything, 2) GPU loads per-layer weights into a template, 3) Computes, 4) Discards, repeat.
• Why it matters: Breaks the link between model size and GPU count. 🍞 Anchor: The coach brings the right baton at the right time; the runner never carries all batons.

🍞 Hook: Picture two buckets you swap so water pours continuously. 🥬 Concept: Pipelined double-buffered execution overlaps copy-in, compute, and copy-out using multiple GPU streams.

• How it works: 1) Buffer A computes while Buffer B loads next layer, 2) Gradients stream out on another lane, 3) Events coordinate timing.
• Why it matters: Keeps the GPU busy and hides transfer time. 🍞 Anchor: While you read page 10, page 11 is already in your other hand.

🍞 Hook: Sometimes you redo a quick step to avoid keeping a giant pile of notes. 🥬 Concept: Explicit recomputation rebuilds small activation pieces during backward instead of storing them all.

• How it works: 1) Checkpoint every K layers, 2) Recompute local activations for that block, 3) Do local backward, 4) Evacuate gradients.
• Why it matters: Slashes memory needs with minimal extra compute. 🍞 Anchor: Instead of storing every math scratch, you rework short bits when needed.

🍞 Hook: Imagine a travel bag with everything for one scene. 🥬 Concept: Layer-contiguous tiling (flat tensor layout) packs each layer’s weights/gradients/optimizer states into one big, tidy block.

• How it works: 1) Single large DMA per layer in/out, 2) Zero-copy views on GPU, 3) Fewer launches, less overhead.
• Why it matters: Faster transfers, simpler memory. 🍞 Anchor: One suitcase per scene instead of ten tiny pouches.

🍞 Hook: Road lanes help cars move at once. 🥬 Concept: Multi-stream scheduling runs compute, weight loads, and gradient offloads on separate GPU streams with events.

• How it works: 1) Weight-ready events, 2) Backward-done events, 3) Buffer-free events, 4) No global pauses.
• Why it matters: Smooth traffic prevents jams and idle time. 🍞 Anchor: Trucks bringing goods, workers making goods, and vans shipping goods all move in parallel.

At a high level: Input → CPU streams next layer’s weights → GPU computes with reusable layer template → GPU sends gradients back → CPU updates weights → Repeat.

— Step-by-step (with Sandwich Concepts where new) —

1. Organize the CPU as the Master Parameter Store 🍞 Hook: You know how it’s faster to find a book if the library shelves are sorted by chapter? 🥬 Concept: Structured parameter store with layer-contiguous tiles keeps each layer’s weights, gradients, and optimizer moments together.

• What happens: Horizon-LM allocates a big RAM area where each layer i is a single, contiguous block: [BF16 weights | BF16 gradients | FP32 moments].
• Why this exists: Without it, you’d scatter small tensors everywhere, causing hundreds of tiny, slow copies and wasted time.
• Example: For a transformer layer, all projections (q, k, v, o) and MLP weights live next to each other so one big copy moves them. 🍞 Anchor: One suitcase per scene—zip it once, move everything at once.

1. Use Pinned Slabs to Stage Transfers 🍞 Hook: Couriers use a loading dock to move big boxes quickly. 🥬 Concept: Pinned slab pools are small, fixed, page-locked buffers for high-speed DMA between CPU and GPU.

• What happens: Before moving to GPU, the next layer is copied JIT into a pinned slab; similarly, gradients return into pinned slabs.
• Why this exists: Pinning everything would exhaust RAM; pin just enough to keep the pipeline full.
• Example: Two weight slabs (double buffer) and K gradient slabs feed and drain the pipeline smoothly. 🍞 Anchor: A few sturdy loading docks beat turning your whole warehouse into a dock.

1. Double-Buffered, Multi-Stream GPU Ingestion 🍞 Hook: While you’re writing with one pen, your other pen is being refilled. 🥬 Concept: Double buffering with three GPU streams (compute, H2D weights, D2H gradients) overlaps copy and compute.

• What happens: Buffer 0 is used by compute while Buffer 1 loads the next layer’s weights. A gradient stream evacuates gradients from the last computed layer.
• Why this exists: To keep the GPU busy and hide transfer time behind computation.
• Example: While layer i runs, layer i+1 weights arrive, and layer i−1 gradients leave—all at once. 🍞 Anchor: Like a kitchen with one pan cooking, another pan being prepped, and a waiter carrying finished plates out.

1. Stateless Layer Templates and Zero-Copy Views 🍞 Hook: Reusable molds can make many cookies without rebuilding the mold. 🥬 Concept: GPU layer templates are operator shells that bind to weight views from the flat buffer without extra allocations.

• What happens: After the big weight copy lands, the system creates tensor views directly into the buffer and binds them to the template.
• Why this exists: Avoids tiny allocations and copy_ calls; reduces overhead and fragmentation.
• Example: Attention and MLP templates reuse the same kernels, just pointed at new weights each layer. 🍞 Anchor: New batch of cookies, same tray; just pour in new batter.

1. Forward Pass with Sparse Checkpoints 🍞 Hook: You place sticky notes every few pages so you can find your place quickly. 🥬 Concept: Checkpoint every K layers to bound activation memory.

• What happens: Run layer-by-layer forward; keep only every K-th activation; immediately release weights and intermediate activations.
• Why this exists: Storing all activations would explode memory; sparse checkpoints plus recompute keeps it small.
• Example: For K=12, you store layers 12, 24, 36, etc. 🍞 Anchor: Instead of saving the whole story, keep a few bookmarks and re-skim short parts later.

1. Block-wise Recomputation and Local Backward 🍞 Hook: To solve a puzzle, sometimes you quickly rebuild a small section to remember how pieces fit. 🥬 Concept: Explicit recomputation rebuilds just the needed activations for one K-layer block, then runs local backward.

• What happens: Load the nearest checkpoint, recompute the block forward, compute gradients for each layer, and immediately offload them.
• Why this exists: Avoids giant autograd graphs and deep activation stacks on the GPU.
• Example: Recompute layers 25–36, compute gradients layer-by-layer in reverse, and send each gradient home. 🍞 Anchor: Redo a tiny bit of math instead of hoarding every scratch note.

1. Immediate Gradient Evacuation and CPU-Side Optimization 🍞 Hook: As soon as a dish is cooked, the waiter takes it out so the kitchen doesn’t get crowded. 🥬 Concept: Gradient evacuation sends ∇θ back to RAM right after it’s produced; the CPU applies Adam updates there.

• What happens: Gradients are flattened and D2H-copied to gradient slabs; background CPU threads unflatten, accumulate, and update weights.
• Why this exists: Keeps GPU free of long-lived gradients and FP32 moments; overlaps compute with CPU optimization.
• Example: While computing layer i−1, CPU updates layer i moments. 🍞 Anchor: Hot plates go out immediately; the chef starts the next dish.

1. Bandwidth-Aware Scheduling and Events 🍞 Hook: Traffic lights keep different lanes moving smoothly without crashes. 🥬 Concept: Event-driven multi-stream orchestration coordinates when weights are ready, when gradients are done, and when buffers can be reused.

• What happens: Weight-ready events let compute proceed; backward-done events trigger gradient offload; buffer-free events let H2D reuse slabs.
• Why this exists: Prevents stalls and overwrites; ensures full overlap.
• Example: The next H2D waits only for the exact buffer it needs, not for a whole-device sync. 🍞 Anchor: Green lights for the right lane at the right time.

1. Predictable Memory Pools 🍞 Hook: Pre-setting table places avoids chaos during dinner rush. 🥬 Concept: Pre-allocated GPU workspaces (streaming buffers, activation stack, checkpoint anchors) and fixed host slabs eliminate runtime jitter.

• What happens: All temporary spaces are allocated once; lifetimes are explicit and stack-like.
• Why this exists: Avoids fragmentation and surprises; keeps GPU memory bounded by the biggest layer and K checkpoints.
• Example: The activation stack grows and shrinks predictably per block. 🍞 Anchor: No searching for clean plates mid-service; everything has a place.

1. Mixed BF16/FP32 on Host 🍞 Hook: Use a small notebook for drafts and a big ledger for the official record. 🥬 Concept: Store weights/gradients in BF16 and optimizer moments in FP32 in RAM to hit the 12 bytes/parameter target (approx).

• What happens: BF16 halves the bytes for weights/gradients; FP32 for moments preserves training stability; overall host memory scales linearly with parameters.
• Why this exists: Makes terabyte-scale models feasible in RAM.
• Example: 100B parameters need about 1.2 TB for core states. 🍞 Anchor: Compact notes plus a precise ledger keep memory modest and accuracy steady.

Secret Sauce:

• CPU is the sole source of truth; GPU holds nothing long-term.
• Layer-contiguous tiles and big, single DMA copies minimize overhead.
• Double-buffered, three-stream pipeline hides copy time behind compute.
• Explicit recompute replaces giant autograd graphs with small, local work.
• Deterministic pools keep memory bounded and predictable.

The Test: The team measured three things that really matter for single-node training:

• Training speed (TFLOPS): How busy and fast the GPU stays.
• Host (CPU) memory footprint: Does memory grow cleanly with model size or explode unpredictably?
• Accuracy: Does this method train just as correctly as standard baselines?

Competition: They compared Horizon-LM to strong baselines people use today:

• DeepSpeed ZeRO-3 Offload (CPU offloading)
• ZeRO-Infinity (GPU–CPU–SSD offloading)
• PyTorch Native (when the model fits fully on GPU)
• ColossalAI Gemini (on A100 PCIe verification)

Scoreboard with Context:

• Feasibility at Huge Scale: On a single H200 with 1.5 TB RAM, Horizon-LM trains models up to 120B parameters. Competing offloading systems typically fail much earlier on a single GPU because their host memory use grows with runtime clutter, not just model size.
• Speed: On a regular single A100 PCIe machine, Horizon-LM is up to 12.2× faster than ZeRO-3 offloading (e.g., 122 TFLOPS vs 10 TFLOPS at 14B). That’s like finishing a 12-lap race while the other runner completes only one.
• Sustained TFLOPS: On GH200/H200, Horizon-LM keeps high throughput even as models scale: e.g., around 284 TFLOPS (7B), ~264 TFLOPS (14B), and stays above ~250 TFLOPS for 32B and beyond—where offloading baselines degrade or fail.
• Depth Scaling: With fixed width and GPU allocation, Horizon-LM’s throughput dips only ~20% going from 28 to 180 layers (7.6B → 43.0B params). Baselines suffer severe slowdowns or run out of memory by 84 layers.
• Width Scaling: As width grows (harder per layer), all systems slow down, but Horizon-LM’s curve is flatter. At 3.5× width, it’s ~1.21× faster than ZeRO-3 and continues to work up to 5.0× width while others OOM earlier.
• Host Memory: Horizon-LM’s host memory grows close to the theoretical model footprint (about 12 bytes/parameter plus small, fixed slabs), while baselines balloon due to extra runtime buffers and duplication.
• Accuracy: On MetaMathQA at 7B and 14B, Horizon-LM matches or slightly exceeds ZeRO and native training, e.g., ~88.99% vs ~88.91–88.97% at 7B and ~92.52% vs ~92.36–92.41% at 14B. That’s like getting an A+ alongside the best students.

Surprising Findings:

• Host memory, not GPU memory, is the true boundary for single-node training of huge models—once you make the GPU stateless and stream by layer.
• Explicit recomputation plus per-layer streaming can outperform sophisticated offloading stacks by large margins, even on plain PCIe.
• Throughput stability across depth shows that bounding the GPU to the largest layer (not the whole model) really breaks the old scaling rules.

Anchored Examples:

• On a single A100 PCIe Gen4 x16: Horizon-LM hits ~128 TFLOPS on 7B vs Gemini ~53 TFLOPS and ZeRO-3 ~36 TFLOPS (2.42× and 3.56× faster). At 14B: 122 vs 15 and 10 TFLOPS (8.13× and 12.20× faster). At 32B: baselines OOM; Horizon-LM ~114 TFLOPS.
• Depth test at fixed GPU alloc (3.83 GB) and fixed width: Horizon-LM keeps running up to 180 layers; baselines OOM by 84 layers.

Takeaway: Make RAM the boss, stream layers, overlap copy and compute, and recompute locally—then single-GPU training of 100B+ models becomes both fast and feasible.

Limitations:

• You need lots of host RAM. For example, 100B parameters need about 1.2 TB just for core states. Commodity servers may not have that much.
• Bandwidth matters. If CPU–GPU links are slow or noisy, layer streaming can become the bottleneck unless overlapped well.
• Extra compute from recomputation. While small compared to storage savings, recomputation does add overhead.
• Not a perfect fit for compute-heavy pretraining. If your goal is maximum raw TFLOPS at trillion-scale pretraining with many GPUs, traditional multi-GPU parallelism may still shine.
• Engineering complexity. The scheduler, pinned slabs, and multi-stream events must be correct and tuned; misconfiguration can reduce gains.

Required Resources:

• One solid GPU (A100/H100/H200 class), ideally with fast interconnects, but PCIe Gen4 can work well.
• Very large host memory (hundreds of GB to 1–2 TB+) for 70B–120B models.
• Strong CPU cores and SIMD-optimized CPU Adam to keep updates from becoming a bottleneck.

When NOT to Use:

• If you only have small RAM and the model fits entirely on GPU, plain PyTorch native may be simplest and fastest.
• If your workload is dominated by huge batch pretraining on many nodes, established distributed strategies might be better.
• If you need every last percent of GPU-only peak performance and the model still fits, the streaming pipeline may not beat in-GPU training.

Open Questions:

• Can we compress CPU-side states (e.g., optimizer moments) further without losing stability, to reduce the 12 bytes/parameter bound?
• How far can GPUDirect Storage or CXL memory pooling push the capacity frontier for single-node training?
• Can adaptive checkpoint intervals (dynamic K) and smarter layer ordering reduce recompute overhead further?
• How do MoE and attention variants with very wide layers affect the width-bound feasibility and scheduling heuristics?
• What’s the best way to integrate differential privacy or low-rank adapters into the streaming pipeline for secure and efficient fine-tuning?

Three-Sentence Summary:

• Horizon-LM makes RAM the official home for all model states and treats the GPU as a temporary, per-layer calculator.
• By streaming one layer at a time, overlapping transfers with compute, and recomputing small blocks, it keeps GPU memory bounded by the widest layer and host memory cleanly proportional to model size.
• The result is fast, accurate single-GPU training and fine-tuning of 100B+ models where offloading baselines slow down or fail.

Main Achievement:

• Decoupling model scale from GPU count with a CPU-master, GPU-template design that delivers stable high throughput and predictable memory at hundred-billion-parameter scale on a single GPU.

Future Directions:

• Shrink host memory needs via smarter compression or optimizer redesign.
• Integrate GPUDirect Storage and emerging interconnects (e.g., CXL) for even larger models.
• Adaptive checkpointing and scheduling policies that learn the best pipeline overlap on the fly.
• Extending to multimodal and MoE architectures with width-aware tuning.

Why Remember This:

• It reframes the bottleneck: from “get more GPUs” to “organize memory and streaming right.”
• It opens node-scale fine-tuning of giant models to many more teams, labs, and classrooms.
• It shows that careful systems design—layer tiling, double buffering, explicit recompute—can beat heavyweight offloading by a wide margin even on plain PCIe.
• It points the way to a more accessible, RAM-first future for large-model training.