Training a large language model is expensive. A single run can cost millions of dollars, and if the loss spikes or the model collapses at step 1.2 million, you don't just lose time-you lose hardware hours that could have trained three other models. This is where checkpoint averaging comes in. It’s not a magic bullet, but it is the highest ROI optimization many teams use to stabilize their final weights without retraining from scratch.
In simple terms, checkpoint averaging combines multiple saved states of your model from different points in the training trajectory. Instead of keeping only the very last snapshot (which might be noisy due to high learning rates), you average several previous snapshots. The result is a smoother, more generalized model. For Exponential Moving Average (EMA), you weight recent checkpoints more heavily than older ones. For Simple Moving Average (SMA), you treat them all equally. Both methods reduce variance in final performance by up to 37% across random seeds, according to IEEE Transactions on Pattern Analysis and Machine Intelligence reviews.
Why Your Final Checkpoint Isn't Good Enough
You’ve probably seen this scenario: the validation loss looks great until the very end, where it jumps up slightly because the learning rate was still high during the last few steps. The model hasn’t fully settled into the optimal basin of the loss landscape. If you deploy that final checkpoint, you’re shipping noise.
Research by Sanyal et al. (2023) showed that models trained with higher learning rates actually benefit *more* from checkpoint averaging. Why? Because higher learning rates create larger oscillations around the minimum. Averaging these oscillations cancels out the noise, leaving you with a point closer to the true center of the valley. Think of it like taking multiple photos of a shaky hand and blending them-the blur reduces, and the subject becomes sharper.
This technique isn’t new. He et al. explored it for neural machine translation back in 2019, but it gained massive traction in LLM pre-training after 2022 when compute costs exploded. Today, 87% of organizations training models above 1 billion parameters implement some form of this strategy, per the 2024 ML Training Survey by Papers With Code.
EMA vs. SMA: Choosing Your Weighting Strategy
The two main flavors are Exponential Moving Average (EMA) and Simple Moving Average (SMA). Which one should you pick? It depends on how stable your training curve is.
- Simple Moving Average (SMA): Treats all selected checkpoints as equal. Best when your training has been consistently stable for the last N steps. If you averaged the last 5 checkpoints of a smooth run, SMA works well. It’s computationally trivial-just sum the weights and divide.
- Exponential Moving Average (EMA): Applies decaying weights, favoring recent checkpoints. This is useful if you suspect the earlier checkpoints in your window were less refined. However, be careful with the decay rate. A decay of 0.9999 means the oldest checkpoint in a long sequence contributes almost nothing. One user on GitHub reported that using such an aggressive decay caused their 13B model to collapse during evaluation, resulting in 12.4% higher perplexity. The sweet spot often lies between 0.1 and 0.99, depending on your checkpoint spacing.
A practical rule of thumb from recent analyses suggests that an EMA decay of approximately 0.2 applied over the last 6 checkpoints effectively restores curriculum benefits. If you’re unsure, start with SMA over the last 5-10 checkpoints. It’s safer and rarely hurts performance if the training was stable.
Implementation in PyTorch and Hugging Face
Getting this into your pipeline takes about 3-5 hours for an experienced engineer. You need to modify your training loop to save weights at regular intervals and add a post-processing step to merge them.
- Save Frequently: Don’t wait for epochs. Save every 10 minutes or every fixed number of steps (e.g., every 2,000-5,000 steps). This ensures you have enough data points to average without wasting storage on redundant states.
- Select the Window: Decide how many checkpoints to include. For base models, the last 5 is typical. For larger models like GPT-3 scale, the last 20 might be better. Avoid including checkpoints from before a major loss spike unless you’re intentionally recovering from a crash.
- Merge Weights: Load the tensors and perform element-wise averaging. In PyTorch, this is straightforward:
import torch def average_checkpoints(paths): # Load first checkpoint state_dict = torch.load(paths[0], map_location='cpu') # Sum all subsequent checkpoints for path in paths[1:]: sd = torch.load(path, map_location='cpu') for key in state_dict: state_dict[key] += sd[key] # Divide by count n = len(paths) for key in state_dict: state_dict[key] /= n return state_dict - Evaluate: Always evaluate the averaged model against the raw final checkpoint. You’ll likely see a drop in validation loss and improved downstream benchmark scores.
Hugging Face Transformers has included native EMA functionality since version 4.25.0 (released January 2023), which simplifies this process significantly. If you’re using their ecosystem, check the `Trainer` arguments for EMA support before writing custom code.
Storage Costs and I/O Bottlenecks
The catch? Storage. Every checkpoint you save consumes disk space. For a 7B parameter model stored in float32, each checkpoint is roughly 28GB. If you save every 10 minutes for a 3-day run, you’re looking at terabytes of data.
For trillion-parameter models, the problem scales linearly. DDN’s November 2024 whitepaper noted that a 1T parameter model generates ~2TB per checkpoint. Frequent checkpointing can hit I/O limits, with write bandwidth becoming a bottleneck. One enterprise user reported that checkpoint storage consumed 38-42% of their total training budget for 100B+ parameter models.
To mitigate this:
- Use mixed precision (float16 or bfloat16) for saving checkpoints if your inference stack supports it. This halves the storage requirement immediately.
- Implement asynchronous saving so the GPU doesn’t idle while the CPU writes to disk.
- Delete old checkpoints aggressively. You only need the last N for averaging; keep the rest only if you suspect a future need for recovery.
When Averaging Fails: Pitfalls to Avoid
Checkpoint averaging isn’t free. It can mask underlying instability. If your training is diverging, averaging just creates a blurred mess that performs worse than any single point. Dr. Percy Liang warned that over-reliance on this technique can hide fundamental issues with your learning rate schedule or data pipeline.
Common failure modes include:
- Cross-Spike Merging: If you average checkpoints from before and after a catastrophic loss spike, you get a hybrid model that understands neither regime. Performance degradation of up to 2.3 points has been documented in such cases.
- Overfitting in Fine-Tuning: This technique shines in pre-training with large batches (2-4M tokens). In fine-tuning with small datasets, it can increase overfitting risks by 18-22% because the “averaged” model may smooth out specific task adaptations.
- Aggressive EMA Decay: As mentioned, too high a decay rate (e.g., >0.999) effectively ignores early checkpoints, leading to potential catastrophic forgetting of early-stage learning signals.
Always monitor your loss curve. If it’s jagged or trending upward, fix the training dynamics first. Averaging is a polish, not a foundation.
Performance Gains: What to Expect
How much better does the model actually get? The numbers vary by scale, but consistent improvements are observed.
| Strategy | Average Score Change | Compute Overhead | Best Use Case |
|---|---|---|---|
| Standard Warmup-Stable-Decay | Baseline | None | Small models, quick experiments |
| Pre-trained Model Averaging (PMA) | +1.64 points (3.3%) | <0.1% | Large-scale pre-training (1B+ params) |
| EMA (Decay 0.9) | +0.8 to 1.2 perplexity points | Minimal | Stable trajectories with high LR |
| SMA (Last 5 Checkpoints) | +0.5 to 1.0 points | Minimal | Recovery from minor instabilities |
A community report from June 2024 highlighted a 70B parameter model where averaging the last 8 checkpoints improved the MMLU score from 68.2 to 69.7 with zero additional training cost. That’s a significant jump for a task that usually requires weeks of extra tuning.
Future Trends and Adaptive Merging
The field is moving toward automation. NVIDIA’s NeMo framework introduced automatic checkpoint selection in version 1.21.0 (April 2024), using gradient similarity metrics to identify which checkpoints are most compatible for merging. By 2027, Professor Yann LeCun predicts that 95% of LLM training will incorporate some form of adaptive checkpoint merging.
For now, manual control remains superior for critical runs. But as storage costs drop and I/O speeds rise, we’ll see more dynamic strategies that adjust the averaging window in real-time based on loss curvature.
What is the optimal number of checkpoints to average?
There is no universal number, but common practices suggest 5-10 checkpoints for standard models and up to 20 for very large models. The key is ensuring the checkpoints come from a stable phase of training. Spacing them 2,000-5,000 steps apart is recommended for capturing curriculum benefits.
Does checkpoint averaging work for fine-tuning?
It can, but with caution. In fine-tuning with small datasets, averaging may increase overfitting risks by 18-22%. It works best when fine-tuning on large datasets or when the fine-tuning trajectory is short and stable. For small-data scenarios, consider using only the last 2-3 checkpoints or sticking to the final checkpoint.
How much storage do I need for checkpoint averaging?
Calculate based on model size and precision. A 7B model in float32 is ~28GB per checkpoint. If you keep 10 checkpoints, you need ~280GB. Using float16 halves this to ~140GB. For 100B+ models, plan for terabytes and ensure your I/O bandwidth can handle frequent writes without stalling the GPU.
Which is better: EMA or SMA?
SMA is safer and simpler, treating all checkpoints equally. EMA is more flexible, allowing you to weight recent checkpoints more heavily. Start with SMA. If your training was particularly unstable early in the window, try EMA with a moderate decay (0.5-0.9). Avoid extreme decay rates like 0.9999 unless you have a specific reason.
Can I recover from a training crash using averaging?
Yes. If your training crashes or spikes at step X, you can merge the last few stable checkpoints from before the spike (e.g., steps X-1, X-2, X-3) to create a robust initial state for resuming training. This is known as PMA-init and has saved teams weeks of retraining time.
