Tag: vLLM

Distributed Transformer Inference: Tensor vs Pipeline Parallelism for LLMs

Learn how to deploy large language models using distributed transformer inference. We compare Tensor and Pipeline Parallelism, discuss KV cache bottlenecks, and explore tools like vLLM and llm-d for efficient LLM serving.

Speculative Decoding Pipelines: Draft-and-Verify for Production LLMs

Learn how speculative decoding accelerates LLM inference using draft-and-verify architectures. Explore Medusa, vLLM implementation, and production tips for 2x-3x speedups.

Batched Generation in LLM Serving: How Request Scheduling Impacts Outputs

Batched generation in LLM serving uses dynamic request scheduling to boost throughput by 3-5x. Learn how continuous batching, PagedAttention, and learning-to-rank algorithms make AI responses faster and cheaper - and why most systems still get it wrong.