TL;DR

What Hardware Configuration Maximizes vLLM Throughput Per Dollar?

vLLM deployment fails most AI startups not because the model is wrong, but because the infrastructure layer was designed for batch processing, not production inference. This checklist covers the 12 critical decisions that separate a $40,000/month inference bill from a $12,000/month one running the same workload.

What Hardware Configuration Maximizes vLLM Throughput Per Dollar?

GPU selection determines your cost ceiling before you write a single line of code. NVIDIA H100 SXM nodes on AWS p5.48xlarge deliver 4.3x the throughput of A100 80GB on p4d.24xlarge for Llama 3 70B, but cost 2.8x more per hour. At 100 million tokens per day, the H100 setup costs $18,400/month versus $31,200/month for the A100 when you factor in total throughput—not just per-GPU pricing.

The calculation most CTOs skip: memory bandwidth math. vLLM's PagedAttention requires 2.2 bytes per parameter for KV cache in FP16. A 70B model needs 154GB for KV cache alone. On A100 80GB, you get 2 active sequences per GPU. On H100 80GB SXM, you get 4. The real question isn't "which GPU" but "how many concurrent users can I support per dollar of hardware."

For most AI startups under $5M ARR, 4x A100 80GB nodes running Mistral 7B or Llama 3 8B beats single H100 configurations. You serve 40 concurrent users at 45 tokens/second each for approximately $6,200/month on AWS. That's $0.00012 per token at scale—a number you can actually pitch to enterprise customers.

Specific configuration: Ubuntu 22.04, CUDA 12.1, vLLM 0.4.3, PyTorch 2.2.0. Use NCCL 2.19 for multi-GPU tensor parallelism. Avoid CUDA 12.2 with vLLM 0.4.x—known deadlock issues with prefix caching.

How Do You Configure vLLM's PagedAttention for Production Traffic?

PagedAttention's default configuration assumes development workloads. Production requires explicit tuning of three parameters: gpumemoryutilization, maxnumseqs, and block_size.

Set gpumemoryutilization=0.92 for A100 nodes. The 8% headroom handles KV cache spikes during burst traffic. Lower it to 0.85 on H100 if you're running concurrent batching with speculative decoding.

The blocksize parameter is where teams lose money. Default of 16 means 512 bytes of KV cache per block. For chat workloads with 2048-token contexts, increase blocksize to 32 or 64. This reduces internal fragmentation by 23% on typical conversational AI loads, translating to 15% more effective context slots per GPU.

maxnumseqs controls the scheduler's upper bound. Set it to 256 for A100, 512 for H100. Don't leave it at default (2560)—it creates scheduling overhead without benefit when your actual concurrency is 40-80 users.

Critical production flag: --enforce-eager. This forces CUDA graph enforcement, reducing latency variance from 45ms to 12ms at p95. The tradeoff is 8% lower throughput. For customer-facing latency SLAs, take the tradeoff.

> 📖 Related: Oxbotica PM referral how to get one and networking tips 2026

What Batching Strategy Balances Latency and Throughput?

vLLM's continuous batching outperforms static batching by 10x in production—not because of throughput gains, but because of tail latency elimination. Static batching creates 200-400ms p99 spikes when a single long-context request blocks a batch. Continuous batching with iteration-level scheduling keeps p99 under 80ms.

Configure chunked prefill with maxnumbatchedtokens=2048 and maxnum_seqs=256. This prevents a single prefill operation from blocking decode operations. For Llama 3 70B on 4xA100, chunked prefill reduces time-to-first-token from 2.3s to 0.8s while maintaining 38 tokens/second decode throughput.

Prefix caching deserves a separate strategy. Enable it with --enable-chunked-prefill --max-num-batched-tokens 2048. For multi-turn conversations where system prompts repeat, prefix caching delivers 3.2x throughput improvement on message 2+ of a conversation. At 50% cache hit rate, your effective cost per token drops from $0.00018 to $0.000056.

The metric that matters: prefill-to-decode ratio. Target 1:4 or lower. If your prefill time exceeds 25% of total generation time, increase maxnumbatched_tokens or reduce context window requirements.

How Do You Structure vLLM for Multi-Node Horizontal Scaling?

Single-node vLLM caps at 8 GPUs before tensor parallelism efficiency collapses. For 300+ concurrent users, you need distributed serving across 2+ nodes.

vLLM's tensor parallelism requires NVLink connectivity between all GPUs. On AWS p4d, you get 600GB/s NVLink within a node. Cross-node communication via EFA drops to 100GB/s. Route all tensor parallel traffic through intra-node NVLink—distribute tensor shards across 8 GPUs in one node before adding a second node.

Use Ray as the orchestration layer. Deploy vLLM with vllm.entrypoints.openai.api_server behind a Ray Serve HTTP adapter. This handles request routing, health checks, and graceful degradation when a GPU fails.

Load balancing configuration: send requests to the node with lowest current queue depth, not round-robin. On a 2-node cluster with 8 GPUs each, Node A serving 60 users and Node B serving 20 users should receive the next request. Average latency drops 34% versus round-robin under skewed load.

Zero redundancy design fails. Run 2x the minimum required capacity—3 nodes for a 2-node load. When a node fails, traffic reroutes without customer impact. The 50% overprovisioning costs $X; the customer churn from an outage costs 10X.

> 📖 Related: Coca-Cola SDE referral process and how to get referred 2026

What Monitoring Metrics Detect vLLM Degradation Before Customers Notice?

vLLM exposes Prometheus metrics at port 8000. Track these five above all others:

vllm:numrequestsrunning divided by vllm:numtotalgpumemoryutilized gives you GPU utilization efficiency. Drop below 0.6 means you're leaving throughput on the table. Above 0.95 means you're queueing.

vllm:timetofirsttokenseconds p95 alerts you to prefill congestion. Set threshold at 1.5x your baseline—typically 2.0 seconds for 70B models. Breach triggers auto-scaling.

vllm:timeperoutputtokenseconds p95 tracks decode performance. Degradation indicates KV cache thrashing or GPU thermal throttling. Alert threshold: 1.3x baseline.

Queue depth alone lies. A stable queue of 50 requests at 0.5s average is healthy. A growing queue from 50 to 200 in 5 minutes predicts pending saturation. Calculate queue growth rate, not absolute depth.

Set up GPU memory monitoring via nvidia-smi. vLLM memory leaks appear as gradual increases over 48-72 hours. When GPU memory utilization exceeds 95%, the engine enters degraded mode—requests queue instead of process. Daily memory increase of >0.5% triggers restart automation before customers hit errors.

Preparation Checklist

Hardware and Infrastructure:

  • Audit current GPU inventory against throughput requirements using token/second/gpu benchmarks for your specific model
  • Provision vLLM 0.4.3+ with CUDA 12.1, PyTorch 2.2.0, and NCCL 2.19 on Ubuntu 22.04
  • Configure gpumemoryutilization=0.92 (A100) or 0.85 (H100), blocksize=32, maxnum_seqs=256 (A100) or 512 (H100)
  • Enable chunked prefill with maxnumbatched_tokens=2048 and prefix caching with --enable-chunked-prefill

Production Safety:

  • Deploy behind Ray Serve with health checks and automatic failover for GPU failures
  • Set enforce_eager=true for latency SLA compliance, accepting 8% throughput reduction
  • Configure multi-node tensor parallelism to route all TP traffic through intra-node NVLink
  • Implement 2x minimum capacity with automatic scaling triggers at 80% GPU utilization

Monitoring Setup:

  • Instrument Prometheus metrics: vllm:numrequestsrunning, timetofirsttokenseconds, timeperoutputtokenseconds
  • Calculate queue growth rate, not absolute depth, for capacity planning alerts
  • Set GPU memory monitoring with nvidia-smi and automated restart at 95% utilization or >0.5% daily growth

Cost Optimization:

  • Run the memory bandwidth math: KV cache requirements = 2.2 bytes × model parameters
  • Target prefill-to-decode ratio below 1:4 to maximize effective throughput
  • Enable prefix caching for multi-turn conversation workloads (3.2x throughput on cached turns)

For teams evaluating inference infrastructure alongside model selection, the vLLM deployment patterns and GPU selection frameworks in the AI Infrastructure Playbook (specifically the chapters on tensor parallelism and continuous batching) provide deeper benchmarking data across H100/A100/L40S configurations with actual workload profiles.

Mistakes to Avoid

BAD: Setting gpumemoryutilization to default 0.9 without context

The default assumes single-user development loads. In production with 50+ concurrent users, 0.9 causes OOM crashes during traffic spikes. Every vLLM deployment failure I've reviewed in 2024 started with teams using defaults.

GOOD: Setting gpumemoryutilization=0.92 with 8% headroom for burst traffic

The small efficiency loss prevents production crashes. On A100 80GB with Llama 3 70B, 0.92 gives you 73GB for KV cache versus 72GB at 0.9—negligible throughput difference, massive stability gain.

BAD: Ignoring block_size for conversational AI workloads

Default block_size=16 creates 23% internal fragmentation with 2048-token contexts. Your GPU memory looks full at 70% utilization when you're actually wasting 23% on fragmentation overhead.

GOOD: Setting block_size=32 for chat workloads, 64 for long-context applications

Matching block size to your actual context length eliminates fragmentation. A 70B model on 4xA100 supports 120 concurrent users at blocksize=32 versus 92 at blocksize=16.

BAD: Round-robin load balancing across multi-node vLLM clusters

Round-robin ignores actual load. Node A serving 60 users and Node B serving 20 users receive equal traffic, causing Node A queue buildup while Node B idles.

GOOD: Queue-depth-aware load balancing that routes to lowest-occupancy node

Distribute by current request count, not by node count. On a 2-node cluster, the node with 20 active requests receives the next request over the node with 60. Average latency drops 34% under skewed traffic.

More PM Career Resources

Explore frameworks, salary data, and interview guides from a Silicon Valley Product Leader.

Visit sirjohnnymai.com →

FAQ

How does vLLM's continuous batching compare to static batching for real-time chat applications?

Continuous batching outperforms static batching by eliminating tail latency spikes that destroy user experience. Static batching creates 200-400ms p99 latency when a single long-context request blocks the batch. Continuous batching with iteration-level scheduling maintains p99 under 80ms. For real-time chat where 2-second response times feel broken, continuous batching is non-negotiable. Configure with maxnumbatched_tokens=2048 to prevent prefill from blocking decode operations.

What's the realistic cost per token for vLLM deployment on AWS at 100M tokens/day?

At 100 million tokens/day with Llama 3 70B on 4xA100 nodes, expect $18,400/month in AWS costs. That's $0.000184 per token input plus $0.00022 per token output. With prefix caching hitting 50% of messages in multi-turn conversations, effective cost drops to $0.00012 per token. Enterprise pricing at $0.003/token input delivers 65% gross margins at this infrastructure cost. The math only works if you optimize block_size, enable prefix caching, and use queue-depth-aware load balancing.

When should you choose vLLM over alternative inference servers like TRT-LLM or Text Generation Inference?

Choose vLLM for conversational AI with variable-length contexts and multi-turn memory, where PagedAttention's efficiency gains are largest. Choose TRT-LLM when you need maximum throughput on fixed-length inference with known batch sizes—for batch processing jobs where latency doesn't matter. Choose TGI when you need HuggingFace compatibility without optimization and can tolerate 20-30% lower throughput. For AI startups building customer-facing chat or agentic applications, vLLM wins on 80% of production workloads. The exception is extremely latency-insensitive batch pipelines where TRT-LLM's throughput advantages outweigh its inflexibility.

Related Reading