GPU Selection for LLM Inference: A100 vs H100 vs CPU Offloading

Choosing the right hardware for running large language models (LLMs) isn’t just about raw power-it’s about latency, cost per token, and whether your system can actually keep up when users expect answers in under a second. If you’re deploying LLMs in production, you’re not just picking a GPU-you’re betting on your application’s responsiveness, scalability, and long-term viability. Let’s cut through the noise and look at what actually matters: NVIDIA’s A100, H100, and CPU offloading.

Why This Choice Matters Right Now

In 2026, LLM inference isn’t a luxury. It’s the backbone of customer service bots, real-time content generators, code assistants, and internal knowledge engines. If your model takes 3 seconds to reply, users leave. If it costs $0.02 per query, your margins vanish. The hardware you pick today will dictate your performance for the next 2-3 years.

The market has shifted dramatically since 2023. Back then, A100s were the gold standard. Today? H100s are winning. And CPU offloading? It’s still around-but only for testing, not production.

The A100: Still Useful, But Falling Behind

The NVIDIA A100, launched in 2020, was built for general-purpose AI workloads. It has 80GB of HBM2e memory, 2.0 TB/s bandwidth, and third-generation Tensor Cores. For its time, it was a beast. Today, it’s the old workhorse.

When running Llama 3.1 70B with vLLM, the A100 hits about 1,148 tokens per second. That sounds fast-until you compare it to the H100’s 3,311 tokens per second. That’s nearly 3x faster. And the gap isn’t just in raw speed. The A100’s memory bandwidth is the bottleneck. As models grow beyond 70B parameters, the A100 starts stuttering. It can’t feed data to its cores fast enough.

Here’s the kicker: A100s are still widely available in cloud environments, and their hourly rates are lower. On AWS, you might pay $0.75/hour for an A100 versus $1.20/hour for an H100. But if the H100 processes 2.8x more tokens in that same hour, you’re actually paying less per token. A 2025 benchmark from Hyperstack.cloud found H100s delivered 18% lower cost per token than A100s on the same workload. That’s not a fluke-it’s architecture.

The H100: The New Standard for Production LLMs

The H100, built on NVIDIA’s Hopper architecture and released in late 2022, wasn’t just an upgrade. It was a redesign for transformers.

Its 80GB HBM3 memory runs at 3.35 TB/s bandwidth-67.5% faster than the A100. It has 14,592 CUDA cores (110% more). But the real game-changer is the Transformer Engine. This feature dynamically switches between FP8, FP16, and INT8 precision during inference. Most models don’t need full 16-bit precision. FP8 cuts memory use by half and doubles throughput. NVIDIA’s own data shows up to 6x faster training and 4x better inference performance over A100 when using FP8.

Real-world results back this up. A 30B model on H100 with FP8 runs 3.3x faster than on A100. For smaller models like Mistral 7B, the H100 delivers 247 tokens per second at $1.20/hour. The A100? Just 112 tokens per second at $0.75/hour. The H100 wins on cost per token, even with a higher hourly rate.

Enterprise users report higher concurrency too. One financial services team on the NVIDIA Developer Forum noted they handle 37 concurrent chat requests on H100 before latency hits 2 seconds. On A100? It’s 22. That’s not a small difference-it’s the difference between a usable chatbot and a frustrating one.

The H100 NVL variant, with two GPUs and 188GB of HBM3 memory, is now the go-to for models over 100B parameters. No other consumer-grade GPU comes close.

CPU Offloading: The Compromise No One Wants to Admit

CPU offloading sounds appealing. Run a 70B model on a server with 128GB RAM and a few cheap GPUs? Yes, it’s possible. Tools like vLLM’s PagedAttention and Hugging Face’s accelerate let you swap weights between GPU memory and CPU RAM.

But here’s the truth: it’s slow. And painfully so.

MLPerf benchmarks from late 2024 show CPU offloading adds 3-10x latency. A model that takes 200ms to respond on an H100 takes 2-5 seconds with offloading. That’s not a delay-it’s a dead end for real-time apps.

Users on Hugging Face forums reported 8-15 second response times for 7B models on high-end server CPUs. Even with AMD EPYC 9654 chips, throughput drops to 1-5 tokens per second. That’s slower than a human typing.

It’s useful for development, prototyping, or running tiny models on laptops. But if you’re deploying a public-facing service? Don’t even consider it. The engineering effort to stabilize it is high, and the user experience is terrible.

Software and Implementation Realities

It’s not just about the hardware. The software stack matters.

H100s need tuning. To get full FP8 benefits, you need to recompile your inference pipeline. NVIDIA says it takes 2-4 weeks of engineering work to fully optimize complex systems. That’s not trivial. But once done, the gains are massive.

A100s? They’re the easy option. Over 85% of inference frameworks-vLLM, TensorRT-LLM, DeepSpeed-work out of the box. You can get up and running in a day. But you’re leaving performance on the table.

CPU offloading? It’s a mess. Documentation is patchy. GitHub contributors report 5-7 days just to stabilize a 70B model. And even then, crashes are common under load.

Market Trends: Who’s Winning in 2026?

Gartner’s May 2025 report shows H100s now power 62% of new enterprise LLM deployments. A100s are down to 28%. CPU offloading? Just 10%-and most of that is in research labs, not production.

Why? Price. Cloud providers slashed H100 instance rates by 40% since January 2025. That made the cost per token drop below A100 levels for most workloads. AMD’s MI300X tried to compete, but it’s still 1.7x slower than H100 at 85% of the cost. Not enough.

Google’s TPU v5p is a wildcard. It can outperform H100 on some models, but only with Google’s own frameworks. If you’re locked into PyTorch or Hugging Face, it’s not an option.

For now, H100 is the only chip that balances speed, cost, and scalability for production LLMs. IDC predicts it’ll hold 75%+ of the market through 2027.

When Should You Choose What?

• Choose H100 if you’re running models over 13B parameters, need low latency, handle concurrent users, or care about long-term scalability. This is your default for production.
• Stick with A100 only if you’re on a tight budget, running small models (<13B), or have legacy systems that can’t be reconfigured. It’s a stopgap, not a strategy.
• Avoid CPU offloading for anything public-facing. It’s fine for learning, experimenting, or running 3B models on your workstation-but not for real users.

If you’re building a new system today, H100 is the only choice that makes sense. The performance gap is too wide. The cost per token is too favorable. And the software ecosystem is finally catching up.

Don’t optimize for yesterday’s hardware. The future of LLM inference isn’t about bigger RAM or cheaper CPUs. It’s about bandwidth, precision, and architecture-and the H100 delivers all three.