Tag: tensor parallelism

Distributed Transformer Inference: Tensor vs Pipeline Parallelism for LLMs

Learn how to deploy large language models using distributed transformer inference. We compare Tensor and Pipeline Parallelism, discuss KV cache bottlenecks, and explore tools like vLLM and llm-d for efficient LLM serving.

Read More

Throughput vs Latency: How Transformer Design Impacts LLM Inference Speed

Explore the critical tradeoff between throughput and latency in LLM inference. Learn how transformer design, batching strategies, and tensor parallelism impact speed and cost.

Read More