•Scaling laws are empirical rules that show how a model’s loss (error) drops as you grow model size, data, or compute. They take a power-law form: Loss = A × N^(-α), where N can be parameters, data tokens, or compute, and α is the scaling exponent. This lets us predict how bigger models might perform without training them.
•The 2020 Kaplan et al. study trained many GPT-2–style models and found neat power-law fits for three knobs: model size (parameters), data size (tokens), and compute (FLOPs). The exponents they reported were roughly α_params ≈ 0.076, α_data ≈ 0.074, and α_compute ≈ 0.069. These similar exponents suggest model size and data are about equally useful for improving loss.
•A key insight is that loss vs. scale looks like a straight line on a log-log plot, which is the signature of a power law. You can fit a line to log(loss) vs. log(N) and the slope gives you the exponent (up to a sign). This simple procedure turns messy training results into a clear predictive model.
•Hoffmann et al. (2022) studied the best way to spend a fixed compute budget, called compute-optimal scaling. They used the relation C ∝ N × D (compute ~ parameters × tokens) and found that you should grow model size and data together. With more compute, both the best N and D increase roughly in proportion to the square root of C in this simplified view.
•Compute-optimal scaling explains why just making a giant model without enough data wastes compute (overfitting), and using too much data for a tiny model underuses compute (underfitting capacity). There’s a sweet spot for each compute budget. Picking N and D well can save time and money while improving performance.
•Why do scaling laws appear? Learning curves often follow a power law with steps T: loss ∝ T^(−β). Combine this with compute C ∝ N × T, and you get loss ∝ C^(−β/(1+β)), another power law. Intuitively, this reflects diminishing returns: each extra batch helps a bit less than the last.
Why This Lecture Matters
Scaling laws turn large-model development from an expensive guessing game into a predictable engineering process. For ML engineers, researchers, and product teams, they provide a compact formula to forecast the returns from increasing parameters, data, or compute. This helps set realistic budgets, timelines, and goals, especially when training runs can cost millions of dollars. With compute-optimal scaling, you can choose the best mix of model size and data for a fixed compute budget, avoiding wasteful runs that either overfit or underutilize capacity.
In real projects, leaders need to justify investments and plan infra. Scaling laws quantify expected gains (e.g., around 5% loss reduction per doubling at today’s exponents), allowing teams to prioritize between collecting more clean data or engineering larger models. They also guide validation protocols—keeping tokenization and datasets consistent—so improvements are attributable to scale rather than setup changes. For applied scientists, scaling laws suggest when to stop: if marginal gains fall below a threshold, it may be time to improve data quality or algorithms instead of just scaling.
Career-wise, understanding scaling laws is now core literacy for professionals building or deploying LLMs. It shows you can reason about trade-offs, avoid costly mistakes, and communicate with stakeholders in concrete, quantitative terms. In an industry where models and datasets grow rapidly, scaling literacy helps you stay efficient and competitive. It’s a lens to make smarter decisions in research exploration, infrastructure design, and product planning.
Lecture Summary
Tap terms for definitions
01Overview
This lecture explains scaling laws for large language models (LLMs): simple empirical rules that show how a model’s error (loss) shrinks as you increase model size (number of parameters), data size (number of training tokens), or compute (amount of floating-point operations). Scaling laws are valuable because they let you predict the performance of models you haven’t trained yet. In 2020, before these laws were widely known, training a much bigger model was a costly guess. Now it is more scientific: by fitting a power law to smaller models, you can project to larger ones with surprising accuracy.
The lecture focuses on three pillars. First, the general form: loss = A × N^(−α), where N stands for the scale you’re changing (parameters, data, or compute), A is a constant, and α is the scaling exponent. Second, major empirical results: Kaplan et al. (2020) showed that loss follows power laws for parameters, data, and compute with similar exponents (around 0.07–0.08), and Hoffmann et al. (2022) introduced compute-optimal scaling to choose the best mix of model size and data for a fixed compute budget using C ∝ N × D. Third, basic theory: learning curves often follow a power law in training steps and, when combined with compute relations, naturally produce scaling laws. The lecture also outlines important limitations: emergence at large scales, data quality, and architecture dependence can bend or break simple power-law predictions.
This content is pitched at learners who know the basics of neural networks and Transformers but may be new to systematic scaling. If you understand what parameters, tokens, loss, and FLOPs are, you can follow the arguments. No deep math beyond logarithms and linear regression is required; the main technique is fitting a straight line on a log-log plot. If you have never seen cross-entropy loss or perplexity, this lecture gently explains their roles in measuring performance.
After this lecture, you will be able to: describe the standard scaling-law form and what the constants mean; read and interpret loss-vs-scale plots; fit your own scaling exponents by training multiple models and running a log-log linear regression; estimate how much you gain by doubling model size or data; and choose model size and data size intelligently for a fixed compute budget using the compute-optimal idea. You will also be able to list common caveats so you avoid overconfident extrapolations when data quality shifts or architectures change.
The lecture is structured in four parts. It starts with a definition and intuition for scaling laws, including why log-log straight lines indicate power laws. Next, it reviews landmark papers: Kaplan et al. (2020) for basic parameter/data/compute scaling and Hoffmann et al. (2022) for compute-optimal trade-offs, illustrating with conceptual plots and the relation C ∝ N × D. Then it provides a simple theoretical justification using learning curves and the law of diminishing returns to derive loss ∝ C^(−β/(1+β)). Finally, it covers other related scaling laws (e.g., transfer scaling) and important limitations: breakdown at extreme scales, emergent behaviors, data quality issues, and architecture dependence. The session ends with practical advice on how to estimate the scaling exponent α in your own projects by training several points, logging losses, and fitting a line in log space.
Key Takeaways
✓Fit in log space, plan in real space: Always log-transform scale and loss, then fit a line; use the resulting α to plan real-world runs. This keeps your math simple and your predictions easy to apply. Visualize the fit to catch outliers early. A clean fit is your best planning tool.
✓Balance model size and data: Treat parameters and data as co-equal levers since their exponents are similar. If memory is tight but storage is cheap, lean into more data; if data is scarce, scale parameters. Avoid extreme imbalances that cause overfitting or underutilization. Balance yields better returns at fixed compute.
✓Use compute as the budget frame: Think in terms of C ∝ N × D to plan feasible runs. For a fixed C, increasing one dimension forces you to decrease the other. This prevents overpromising and underdelivering on timelines and costs. It aligns engineering constraints with achievable gains.
✓Train to convergence parity: Ensure larger models are trained long enough; undertraining makes them look worse and distorts your fit. Watch validation curves to confirm plateaus. Equal effort across scales gives a fair comparison. It protects your exponent estimate.
✓Keep validation and tokenization constant: Use the same validation set and tokenizer across runs. Changing either can shift loss values unrelated to true improvements. This keeps your scaling curve honest. Consistency beats noisy metrics.
✓Estimate gains per doubling: Translate α into a simple rule of thumb, like ~5% loss drop per doubling. Use it to set stakeholder expectations about returns on added compute. This prevents unrealistic hopes about instant large gains. It supports steady, compounding progress.
Glossary
Scaling law
A rule that shows how a model’s error changes when you make the model bigger, give it more data, or use more compute. It usually looks like a power law, which is a simple math formula with an exponent. This makes it easy to draw as a straight line on a log-log plot. It helps predict results for bigger models without training them all. It turns many experiments into one clear pattern.
Power law
A math relationship where one quantity equals a constant times another quantity raised to a power. In scaling laws, loss changes as a power of model size, data, or compute. On a log-log plot, a power law becomes a straight line. The line’s slope equals the negative exponent. This makes finding the exponent easy with linear regression.
Exponent (α)
The exponent tells how fast loss falls when you scale up. A larger α means loss drops faster for each doubling. A smaller α means gains are smaller per doubling but still steady. In language models, α is often around 0.07. Even small exponents matter when you scale many times.
Constant (A)
The constant A sets the overall level of loss in the power law. It depends on the task, data quality, and training setup. Two teams might have the same α but different A because their data or recipe differs. A shifts the line up or down on a log-log plot. It captures difficulty and setup effects.
Version: 1
•Plots from scaling-law papers show families of curves: loss vs. parameters for several fixed data sizes, and loss vs. data for several fixed model sizes. On both axes, more of one helps you use more of the other. Bigger models benefit from more data, and more data pays off more with bigger models.
•Scaling laws are powerful but have limits. At very large scales, new behaviors can emerge that weren’t visible earlier. Data quality, architecture changes, and training tricks can also bend or break simple power-law predictions.
•Emergent behavior means a model suddenly gains skills (like coding or reasoning) that were not explicitly programmed. Simple power-law fits may not forecast exactly when these jumps appear. So, extrapolations should be done with care, especially far beyond observed scales.
•Data quality matters: noisy or biased datasets can spoil the expected gains. If training text contains lots of profanity or incorrect facts, the model may learn them. Scaling laws usually assume reasonably clean, representative data.
•Architecture matters too: scaling exponents are measured under fairly fixed architecture families (e.g., GPT-2–like Transformers). Changing architecture or training recipes (regularization, tokenization, optimization) can change constants and even exponents. Treat scaling laws as guides for similar setups, not universal laws of nature.
•Practically, to estimate α, train several models at different sizes or data amounts, record loss, log-transform both axes, and run linear regression. The line’s slope gives the exponent and the intercept gives A. With that fit, you can forecast what happens if you double parameters or data, or quadruple compute.
02Key Concepts
01
What scaling laws are: Scaling laws are empirical rules showing how model error (loss) falls as you increase model size, data, or compute. They usually take a power-law form: loss = A × N^(−α), where A is a constant and α is the scaling exponent. A power law means a straight line on a log-log plot, making it easy to fit and extrapolate. These laws let you predict the loss of larger models without training them, saving time and money. In practice, they guide decisions about how big to make models and how much data to gather.
02
Power laws and log-log plots: A power law relates two quantities with an exponent. If loss = A × N^(−α), then log(loss) = log(A) − α log(N), a straight line in log space. The slope of that line is −α, and the intercept is log(A). Seeing a straight line when plotting log(loss) vs. log(N) signals a power-law fit. This simple geometry turns noisy training results into a simple, predictive formula.
03
Loss as the performance yardstick: Loss (often cross-entropy for language models) measures how wrong a model’s predictions are. Lower loss means better predictions and often lower perplexity (perplexity is exp(loss) when loss uses natural logs). Because loss is smooth and consistent across datasets, it is commonly used in scaling-law studies. Tracking loss across different model sizes or data amounts reveals how efficiency changes with scale. It’s the main quantity that scaling laws try to predict.
04
Kaplan et al. (2020) results: Kaplan and colleagues trained GPT-2–style models and found clear power laws for three knobs: parameters (N), data tokens (D), and compute (C). They reported exponents of roughly α_params ≈ 0.076, α_data ≈ 0.074, and α_compute ≈ 0.069. These exponents were similar for parameters and data, suggesting both are equally helpful for reducing loss. The compute relation offered a way to think about gains as you spend more FLOPs. Their plots showed straight lines in log space, validating the power-law view.
05
Equal importance of model size and data: Because α for parameters and α for data were close, increasing either one brought similar marginal benefits. This means you can often improve by making the model bigger or by feeding it more tokens. In practice, the choice may depend on available data, engineering constraints, and memory limits. Balanced growth tends to unlock the best gains. This parity of exponents is one reason compute-optimal rules advise increasing both together.
06
Compute as a key driver: Compute connects parameters and data through the relation C ∝ N × D. Spending more compute lets you move to larger models and larger datasets together, typically improving loss along an optimal frontier. From a planning perspective, compute is the budget that unlocks how far you can scale both N and D. This framing helps teams allocate resources for the biggest return. It motivates compute-optimal strategies rather than scaling one dimension alone.
07
Hoffmann et al. (2022) and compute-optimal scaling: Hoffmann and colleagues studied how to choose N and D for a fixed compute budget. With C ∝ N × D, they found that optimal model size and data size grow together as you raise C. In simplified form, both N and D scale roughly with the square root of C. This prevents overfitting (too big N, too little D) and underutilization (too small N, too much D). The idea guides you to the best mix for the compute you can afford.
08
Learning curves and diminishing returns: Learning curves plot loss versus training steps and often look like a power law: loss ∝ T^(−β). The exponent β captures how fast loss drops as training progresses. Because each new batch teaches less than the last, gains slow down—a phenomenon called diminishing returns. When combined with compute C ∝ N × T, you get loss ∝ C^(−β/(1+β)), again a power law. This provides a simple theoretical reason why scaling laws emerge.
09
Interpreting exponents: The exponent α tells you how sensitive loss is to scaling. If α is 0.076 for parameters, doubling parameters multiplies loss by 2^(−0.076), a modest but steady gain. Small exponents mean improvements are real but gradual, requiring large scale-ups for big changes. This explains why very large models and datasets are needed for substantial loss reductions. Understanding α helps budget expectations and timelines.
10
Reading scaling-law plots: Typical plots show loss on the y-axis and parameters or data on the x-axis, with multiple lines for different fixed settings of the other variable. Lines slope downward in log-log space, indicating power-law behavior. The spacing between lines shows how extra data helps a fixed model size and vice versa. The best line at a given compute marks the efficient frontier. These visuals make trade-offs tangible.
11
Extrapolating to larger models: Once you fit A and α, you can forecast what loss you might achieve at, say, 10× parameters or 4× data. This lets you plan whether the expected gain is worth the cost. Extrapolation is most reliable when not too far beyond your training range. Still, it provides a baseline expectation before spending on huge runs. Teams use this to set budgets and milestones.
12
Fitting α in practice: To find α, train multiple models at different sizes or data amounts and record the final losses. Take logarithms of both the x-axis (scale) and y-axis (loss), then run linear regression. The slope in the log-log plot gives −α and the intercept gives log(A). More points and careful measurement reduce noise and improve confidence. This straightforward process turns experiments into an actionable formula.
13
Limits: emergence and breakdown: At very large scales, models can display emergent abilities like coding help or reasoning that were not visible before. These jumps can bend the simple power-law curve. As a result, extrapolations far beyond the data may mispredict. Treat scaling laws as strong trends, not absolute guarantees. Monitor outcomes as you scale to detect departures early.
14
Data quality and bias: Scaling laws often assume data is high quality and representative. If data is noisy, biased, or contains undesired content, the model can learn those patterns. In such cases, the expected gain from scaling may be lower or qualitatively different. Cleaning data or curating better sources can shift the curve. Quality often matters as much as quantity.
15
Architecture dependence: The measured exponents come from specific architectures and training setups. Changing model architecture, tokenization, optimization, or regularization can change exponents and constants. Therefore, scaling laws are most trustworthy when applied to similar model families. For different setups, you should refit the exponents. This ensures predictions stay relevant to your actual system.
16
Transfer scaling example: Research on transfer scaling shows that pretraining on more source data can improve a target task as a power law. For example, training on many animal images helps when later fine-tuning on cat images. Performance on the target task improves with the size of the source dataset, often following a predictable curve. This extends scaling ideas beyond pure language modeling. It highlights the broad reach of power laws in machine learning.
17
Overfitting vs. underfitting at scale: Too large a model with too little data risks overfitting, memorizing training details instead of generalizing. Too small a model with very large data may underfit, unable to capture all patterns. Compute-optimal rules help avoid both by balancing N and D. The result is a better use of every FLOP you spend. This balance is essential for efficient large-scale training.
18
Compute budgeting with C ∝ N × D: The relation C ∝ N × D gives a simple mental model for planning training runs. If you double N at fixed D, compute doubles, and vice versa. For a fixed budget, increasing one forces you to decrease the other. Finding the sweet spot yields the lowest loss within your compute limit. This is a practical lever for teams with constrained resources.
19
Perplexity and bits-per-token: In language modeling, loss is often measured in nats or bits-per-token; perplexity is a related metric. Lower loss means lower perplexity, which usually corresponds to better text prediction. Scaling laws typically focus on loss because it’s directly optimized. Converting between loss and perplexity helps interpret improvements. It also bridges academic reporting and practical benchmarks.
20
Practical workflow for scaling studies: Start by choosing a family of models and a clean dataset. Train multiple points across parameters or data sizes, keeping other settings fixed. Log the final loss for each and fit a power law in log space. Use the fit to make forecasts and choose future training runs. Refit if you change architectures or data quality.
03Technical Details
Overall Architecture/Structure of the Ideas
The power-law form: The central relationship is loss = A × N^(−α). Here, N stands for what you are scaling: parameters (model size), data tokens (dataset size), or compute (FLOPs). A is a positive constant capturing how hard the problem and setup are, and α is the scaling exponent describing how quickly loss falls as N grows. Taking logs gives log(loss) = log(A) − α log(N), a straight line in log space.
Three knobs and three exponents: Kaplan et al. measured exponents for parameters (α_params ≈ 0.076), data (α_data ≈ 0.074), and compute (α_compute ≈ 0.069). These numbers are small, meaning doubling scale gives steady but modest improvements: loss is multiplied by 2^(−α), which is close to but less than 1. For example, 2^(−0.076) ≈ 0.95, a ~5% reduction in loss per doubling of parameters. Across many doublings, these small gains compound into large improvements, explaining the massive resource investment in modern LLMs.
Compute as the unifying budget: The relation C ∝ N × D links parameters (N) and data size (D) to compute (C). Doubling parameters at fixed data doubles compute; doubling data at fixed parameters also doubles compute. For a fixed compute budget, you can’t increase both N and D freely—if one rises, the other must fall to keep C constant. This trade-off is the heart of compute-optimal scaling.
Compute-optimal choice of N and D: Hoffmann et al. studied how to choose N and D to get the lowest loss for a given C. In simplified terms, they found that as you increase C, you should increase both N and D together, each roughly scaling like the square root of the compute (N ∝ √C and D ∝ √C, in this simplified explanation). Intuitively, this balances model capacity and data coverage so the model neither memorizes too-small data nor leaves capacity unused. The result is a curve of best-achievable loss as compute grows.
Theoretical sketch via learning curves: Learning curves often show loss ∝ T^(−β), where T is training steps and β > 0. If you assume compute is proportional to N × T, you can rearrange to express T in terms of C and N, and then substitute into the loss formula. Under reasonable simplifications, this yields loss ∝ C^(−β/(1+β)), a power law in compute. This connects the micro (training steps) to the macro (compute budget) and explains why simple, robust scaling trends arise.
Data Flow and Measurement
Inputs: The main inputs are model size (parameters), dataset size (tokens), and compute budget (FLOPs). You also need a consistent training recipe (optimizer, learning rate schedule, tokenization) so results are comparable.
Process: Train several models, each at a different N (or D), keeping other variables as controlled as possible. For data scaling, you might fix N and vary D. For parameter scaling, fix D and vary N. If fitting compute scaling, vary C through combinations of N and D or length of training.
Outputs: Measure the final training or validation loss (often cross-entropy) for each run. Record pairs (N, loss), (D, loss), or (C, loss). These pairs become points on a log-log plot for fitting the power law.
Even without providing executable code, the steps are clear enough to implement in any ML stack (PyTorch, JAX, TensorFlow):
Preparing data and models:
Choose an architecture family (e.g., GPT-2–style Transformer) and keep it fixed except for size. Parameterize model size by depth (layers), width (hidden size), and attention heads so you can create versions at, say, 50M, 100M, 200M, 500M, and 1B parameters.
Prepare a clean dataset and a consistent tokenization. Ensure that when you vary D (data tokens), you do so in clean increments (e.g., 5B, 10B, 20B tokens) drawn from the same distribution.
Training runs:
For parameter-scaling experiments, pick a fixed D and train each model to convergence under a similar training schedule (same optimizer type, similar learning-rate schedule adapted to scale when needed). Record the final validation loss.
For data-scaling experiments, fix N and vary D, again recording losses. When varying data, either train longer or sample more fresh tokens; avoid reusing tokens excessively when claiming larger D.
For compute-scaling experiments, manipulate N, D, or training steps to produce distinct compute budgets C. Estimating C is often done by counting forward/backward FLOPs per token per parameter and multiplying by tokens processed; a simplified assumption is C ∝ N × D.
Fitting the power law:
Take logs: x = log(N) (or log(D), or log(C)), y = log(loss).
Fit a linear regression y = b0 + b1 x. The slope b1 should be approximately −α, so α ≈ −b1, and A ≈ exp(b0).
Check goodness-of-fit (e.g., R^2) and residuals in log space. If residuals show curvature, consider whether you are mixing regimes or whether the architecture/training recipe changed between points.
Using the fit to forecast:
To predict loss at a larger N*, compute log(loss*) = log(A) − α log(N*) and exponentiate.
To estimate how much gain you get from doubling N, compute the factor 2^(−α). For α ≈ 0.076, the gain is about 5% loss reduction per doubling of parameters.
Compute-optimal planning:
With C fixed, choose N and D to lie on or near the compute-optimal frontier. In the simplified square-root rule, N ∝ √C and D ∝ √C.
Practically, you pick a candidate N given your memory limits, then set D so that N × D ≈ C. If you have flexibility, sweep nearby N and D to verify which gives the lowest loss.
Tools/Libraries Used (conceptual)
Deep learning frameworks: PyTorch, JAX, or TensorFlow to define and train Transformer models of varying sizes.
Tokenizers: Byte-Pair Encoding (BPE) or unigram tokenizers to produce consistent tokens across runs.
Logging and plotting: NumPy/Pandas for data handling, Matplotlib/Seaborn for plotting log-log figures, and scikit-learn or built-in routines for linear regression in log space.
Compute estimators: Simple scripts to estimate C from N and D. While detailed FLOP counting depends on architecture, the proportionality C ∝ N × D is sufficient for fitting and planning.
Decide on 5–7 scale points (e.g., N ∈ {50M, 100M, 200M, 500M, 1B, 2B}). More points across a wide range improve fit quality.
Step 2: Prepare data slices
If scaling data, curate subsets with increasing token counts from the same source. Keep validation data constant and clean across all runs to compare losses fairly.
If scaling parameters, ensure the same training data is used across runs.
Step 3: Train and log
Train each run to comparable convergence criteria (e.g., validation loss plateau). Record final validation loss and any stability notes (e.g., if a run diverged).
Save metadata: run ID, N/D/C values, learning rate, batch size, steps, wall-clock time, and compute estimates.
Step 4: Fit the power law
Build a table with columns [scale_value, loss]. Compute [log_scale_value, log_loss].
Run linear regression: log_loss = b0 + b1 × log_scale_value. Extract α = −b1 and A = exp(b0).
Plot the data and the fitted line on log-log axes, and inspect residuals for patterns.
Step 5: Forecast and plan
Use the fit to estimate loss at larger scales you might try next. Compute expected gains from doubling or tripling scale.
For compute budgeting, apply C ∝ N × D and the simplified square-root guidance: as C grows, plan to grow both N and D together.
Tips and Warnings
Keep training recipes consistent: Changing architecture, optimizer, or tokenization across points can bend the curve, giving misleading exponents. If you must change, segment the data and fit each regime separately.
Beware of data contamination and quality: If validation data leaks into training or if data quality varies across scale points, your loss comparisons won’t be apples-to-apples. Curate carefully.
Convergence parity matters: Ensure larger models are trained sufficiently; undertraining bigger models can make them look worse than they would be if fully converged, biasing the slope upward (worse α).
Measure the right loss: Use the same validation loss definition (cross-entropy) and the same tokenization for all runs. Converting to perplexity is fine but fit on loss for simplicity.
Extrapolate cautiously: Power laws work well within observed ranges but can miss emergent jumps or slowdowns at extreme scales. Add safety margins to forecasts.
Compute estimates are approximate: While C ∝ N × D is the guiding relation, real costs include activation memory, optimizer state, and system overhead. Track empirical wall-clock and GPU-hours alongside theoretical FLOPs.
Overfitting signals: If loss keeps dropping on training data but stalls or worsens on validation, your N:D balance or regularization may be off. Adjust data size or regularization before concluding the scaling law failed.
Underfitting signals: If even training loss stalls high at small models, capacity is insufficient. Increase N or use more expressive architectures—but remeasure α after changes.
Log-space regression details: Using ordinary least squares on log-transformed data is standard. Consider robust regression if you suspect outliers (e.g., a run that diverged late). Report confidence intervals for α.
Interpret small α properly: Small exponents don’t mean scaling isn’t useful—gains compound across many doublings. Plan for the long game and combine scaling with data quality and optimization improvements.
Applying to Other Domains
Image classification: Similar power-law trends appear when scaling model size or data for vision models; loss or error rates often drop predictably. The exact α depends on architecture and datasets.
Reinforcement learning: While noisier, researchers have reported scaling-like patterns in RL when averaging across tasks and seeds. Care is needed due to variance and environment differences.
Transfer learning: Source-data scaling can improve target-task performance, often following power laws. This encourages pretraining on large, diverse corpora before specialization.
Putting It All Together
Scaling laws offer a compact, practical model: measure a handful of points carefully; fit a line in log space; use the fit to predict and plan; and adjust as you change setups. Compute binds model size and data size, so the best results come from balanced growth. Theoretical learning-curve intuition explains why the world often looks linear in log-log space. Finally, stay alert to limits: emergence, data quality, and architecture shifts can move you off the simple line—so measure, fit, and re-validate as you scale.
04Examples
💡
Doubling model size example: Suppose loss = A × N^(−0.076). If you double parameters from N to 2N, the predicted loss becomes A × (2N)^(−0.076) = A × 2^(−0.076) × N^(−0.076). Since 2^(−0.076) ≈ 0.95, loss drops about 5%. This shows gains are steady but modest per doubling.
💡
Doubling data size example: With α_data ≈ 0.074, doubling tokens from D to 2D multiplies loss by 2^(−0.074) ≈ 0.95. So more data helps similarly to more parameters. If your model memory is maxed out, gathering more clean tokens can achieve comparable gains. This equivalence guides practical choices.
💡
Compute scaling example: Using α_compute ≈ 0.069, doubling compute multiplies loss by 2^(−0.069) ≈ 0.95. Spending more compute (via larger N, larger D, or longer training) gives predictable benefits. This helps budgeting GPU-hours for expected returns. It frames loss reduction per extra compute dollar.
💡
Choosing N and D with a fixed compute budget: If your compute budget is C and C ∝ N × D, and you can afford C = 10^20 FLOPs, you could pick (N, D) pairs like (10^10, 10^10) or (10^11, 10^9). The compute-optimal view says grow both together—so stay near balanced pairs rather than extreme imbalances. This avoids overfitting (too big N, too little D) or underutilization (too small N, too much D). You get better loss for the same C.
💡
Forecasting at 10× parameters: Fit loss = A × N^(−0.076). If your current model with N0 has loss L0, then at 10N0 the predicted loss is L0 × 10^(−0.076) ≈ L0 × 0.84. That’s a 16% drop. Teams use such estimates to decide whether 10× parameters is worth the cost.
💡
Reading a log-log plot: Imagine a chart with model size on the x-axis (log scale) and loss on the y-axis (log scale). Straight, downward-sloping lines indicate power-law fits. Different colored lines for different data sizes show that more data shifts the line downward (better loss) at the same model size. The gap between lines shows the benefit of more data for fixed N.
💡
Learning-curve intuition: Plot loss over training steps T and see loss ∝ T^(−β). Early on, loss drops fast; later it flattens (diminishing returns). When you note that compute grows with both N and T, you can express loss as a function of compute, giving a power-law relation. This explains why power laws appear so robust across setups.
💡
Transfer scaling scenario: You want a cat-image classifier, but have limited cat photos. You pretrain on a huge dataset of other animals, then fine-tune on cats. As you increase source data (other animals), your cat classifier improves in a predictable, power-law-like way. This shows scaling benefits even when the target task data is scarce.
💡
Data quality counterexample: If your added tokens are noisy or biased (e.g., lots of profanity or contradictions), the expected 5% gain per doubling might not materialize. The model can learn unwanted patterns, reducing real-world performance. This illustrates why scaling laws assume reasonably clean, representative data. Curating data maintains the curve’s reliability.
💡
Emergent behavior caution: When scaling from 1B to 100B parameters, models may suddenly show new skills like tool use or code generation. These behaviors can cause bends in the loss curve not predicted by small-scale fits. Thus, extrapolations far beyond observed ranges carry uncertainty. Monitoring and re-fitting at intermediate scales reduces surprise.
💡
Undertraining bias example: If you train the largest model fewer steps than needed, its validation loss will be higher than it should be. Fitting a line through such a point can make α look smaller (or the fit noisier). Ensuring convergence parity across runs keeps the fit honest. Always check training curves before finalizing fits.
💡
Budget planning example: You have a budget to double compute every quarter. Using α_compute ≈ 0.069, you can expect about 5% loss reduction each quarter if you scale optimally. Setting these realistic expectations helps stakeholders plan milestones. Combining scaling with data cleaning may yield even better-than-predicted gains.
💡
Choosing between more data or more params: With similar exponents for data and parameters, if storage is cheap but memory is tight, choose more data; if data is scarce but you can afford larger models, choose more parameters. Either path gives similar marginal returns. This flexibility is a practical benefit of the findings. You tailor scaling to your constraints.
💡
Validation set consistency example: If you change the validation set between runs, loss changes might reflect dataset differences rather than true improvements. Keeping a fixed validation set makes comparisons fair. This practice strengthens the reliability of your scaling-law fit. Consistency turns experiments into trustworthy predictions.
💡
Log-space regression workflow: After collecting (N, loss) across six model sizes, compute x = log(N) and y = log(loss). Fit y = b0 + b1 x with linear regression. The slope b1 gives −α and intercept gives log(A). Plot the points and the fitted line to visually confirm the power-law relationship.
05Conclusion
Scaling laws give a simple, powerful way to understand and predict how language models improve as you scale parameters, data, and compute. The core form, loss = A × N^(−α), becomes a straight line in log space, making it easy to fit and extrapolate. Landmark studies found similar exponents for parameters and data, suggesting both are equally valuable, with compute tying them together through C ∝ N × D. Compute-optimal scaling advises growing model size and data together for a given budget, avoiding overfitting and underutilization. A basic theoretical view using learning curves and diminishing returns explains why power laws appear so consistently.
To put this into practice, design clean scaling experiments, train several points, and fit a line to log(loss) vs. log(scale). Use the exponent α to forecast gains from doubling size or data and to choose the best balance of N and D for your compute. Keep training recipes consistent, ensure convergence parity, and maintain a steady validation set—these make your fit reliable. Stay mindful of limits: at very large scales, emergent behaviors, data quality changes, and architecture tweaks can bend the curve, so update your fits as you scale.
For immediate practice, run a small scaling study: pick a model family, vary parameters across 5–7 sizes at a fixed dataset, and fit α_params. Then repeat by varying data size at a fixed model size. Try a compute-budget scenario using C ∝ N × D to plan the next run. As next steps, explore transfer scaling (pretrain size vs. downstream gains) and study how different tokenizers or training schedules shift A and α.
The core message: scaling laws turn high-stakes LLM training from guesswork into a predictable, budgetable process. Use them to plan, to decide where to spend compute, and to set realistic expectations. Respect their limits, refit when conditions change, and combine scaling with data quality and good engineering. With this approach, you can scale smarter, not just bigger.
✓
Re-fit after major changes: If you change architecture, data quality, or training recipes, remeasure points and refit α. Old exponents may no longer apply. This avoids misallocation of budgets based on outdated trends. Keep your scaling model current.
✓Prioritize data quality: Scaling assumes reasonably clean data; noisy or biased data can blunt gains. Invest in curation and filtering to keep the curve favorable. Quality often lowers A, improving the whole curve. It’s a multiplier on scaling benefits.
✓Beware of emergence at extreme scales: Large models can show new abilities that bend simple trends. Extrapolate with caution when going far beyond observed data. Plan intermediate checks and be ready to revise forecasts. Let measurements guide you.
✓Use compute-optimal planning: For fixed compute, increase N and D together rather than just one. This avoids overfitting (huge N, tiny D) and underuse (tiny N, huge D). The simplified square-root rule provides an intuitive starting point. Validate near your planned point to fine-tune the balance.
✓Communicate uncertainty: When sharing forecasts, include confidence intervals and note assumptions (fixed architecture, consistent data). This builds trust and makes plans robust. Stakeholders can then weigh risks and options. Responsible forecasting is part of good engineering.
✓Look for straightness and residuals: A straight log-log line and small, structureless residuals suggest a good power-law fit. Curved residuals hint at mixed regimes or setup changes. Use this diagnostic to refine experiments. Good fits lead to credible predictions.
✓Track real compute and time: The C ∝ N × D rule is a guide; track wall-clock time, GPU-hours, and memory too. System overhead can shift practical limits. This helps you plan infrastructure and scheduling. It ties theory to reality.
✓Combine scaling with algorithmic improvements: Scaling isn’t the only lever. Better optimization, architectures, or tokenization can reduce A or even change α. Use both scaling and innovation for the best results. It’s not either-or.
✓Start small, iterate: Begin with a few well-chosen points, fit α, then expand. Each round improves your forecasts. This iterative approach saves compute and avoids dead ends. It’s a practical path to confident scaling.
Model size (parameters)
The number of adjustable weights in a neural network. More parameters usually let a model learn more complex patterns. In scaling studies, this is often the main x-axis. It’s commonly measured in millions or billions. Making models bigger is a key way to reduce loss.
Data size (tokens)
The number of token pieces the model sees during training. More tokens give more examples to learn from. Token counts can be in billions or trillions for large LMs. Bigger datasets usually lower loss. Clean, diverse tokens help the most.
Compute (FLOPs)
The amount of calculation work done during training, often measured in floating-point operations (FLOPs). More compute lets you train larger models on more data or for more steps. Compute binds together parameters and data. It’s the main budget you spend in large-scale training. Careful planning maximizes gains per FLOP.
Loss (cross-entropy)
A number that measures how wrong the model’s predictions are. For language models, cross-entropy compares predicted probabilities to the true next token. Lower loss means better predictions. It’s smooth and comparable across scales. It’s the standard metric for scaling-law fits.
