The candidates who obsess over model accuracy fail the system design round because they ignore the unit economics of inference.
In a Q4 2023 debrief for a Staff ML Engineer role at Meta, the hiring committee rejected a candidate with a perfect transformer architecture diagram because their cost projection for serving Llama-2-70B was off by a factor of twelve.
The candidate proposed running full precision models on p4d.24xlarge instances without quantization, resulting in a projected monthly bill of $480,000 for a feature serving 50,000 daily active users. The hiring manager, a director from the Generative AI infrastructure team, noted that the candidate treated GPU memory as infinite and electricity as free.
This is not a coding test; it is a business viability assessment. The template for LLM inference serving system design with cost optimization for interviews is not about drawing boxes for load balancers; it is about proving you understand that every token generated carries a direct margin impact. If you cannot articulate the trade-off between P99 latency and spot instance interruption rates, you are not ready for a Senior role at Netflix or Google Cloud.
What Is the Core Trade-off Between Latency and Cost in LLM Serving?
The core trade-off is that reducing latency by 40% often increases infrastructure costs by 300%, and senior engineers must justify this spend with specific revenue metrics.
Most candidates default to the "fastest possible" configuration, recommending NVIDIA H100 clusters with tensor parallelism across eight GPUs to minimize time-to-first-token (TTFT). In a design interview for a Search Relevance role at Google in early 2024, a candidate suggested this approach for a spelling correction feature handling 2 billion queries per day. The interviewer immediately stopped the whiteboard session to ask about the cost per query.
The candidate estimated $0.0001, but the actual cost on H100s with full precision would be closer to $0.0008, burning $1.6 million daily for a low-margin feature. The correct judgment is not X (maximum speed), but Y (acceptable speed at sustainable margins). For internal tools or batch processing, the template requires shifting to A10G or even T4 instances with int8 quantization, accepting a TTFT of 400ms instead of 80ms to save 60% on hourly compute rates.
You must anchor your design decisions to specific Service Level Objectives (SLOs) defined by the product manager. At Stripe, during a system design loop for a fraud detection LLM, the requirement was explicit: decisions must return within 150ms to avoid blocking checkout flows, but the budget was capped at $0.02 per transaction.
A candidate who proposed a complex vLLM setup with continuous batching failed because they did not calculate the break-even point where the cost of the GPU exceeded the value of the prevented fraud. The framework here is "Cost-Per-Successful-Request," not "Requests-Per-Second." You need to state clearly: "Given the 150ms SLO, we can utilize on-demand g5.2xlarge instances with AWQ quantization, keeping the cost per request under $0.015 while maintaining a P99 latency of 135ms." This specific alignment of technical constraints and financial boundaries signals seniority.
The counter-intuitive insight is that over-provisioning for peak traffic destroys unit economics more than occasional latency spikes do. During the Black Friday 2023 surge, a retail client of AWS scaled their LLM customer support bot to handle 10x traffic using reserved capacity, only to realize later that 90% of that capacity sat idle for the remaining 330 days of the year.
A better design pattern involves aggressive auto-scaling with a warm pool of quantized models on spot instances, accepting a 2-second cold start for the first 5% of requests in exchange for a 70% reduction in annualized run rate.
When you present this in an interview at Amazon, explicitly mention the "Savings Plan" commitment tiers versus on-demand pricing. Say, "We commit to a 1-year savings plan for the baseline 20% traffic to lock in a 40% discount, and burst into spot capacity for the remainder." This shows you understand the AWS billing console, not just the PyTorch profiler.
How Do You Structure the Architecture for Multi-Tenant Isolation Without Blowing the Budget?
Multi-tenant isolation requires logical separation via request routing and token budgeting rather than physical GPU segregation, which wastes 60% of available VRAM in typical workloads.
The naive approach is to assign dedicated GPU instances to each enterprise tenant, a strategy a candidate proposed in a Snowflake system design interview in late 2023. The candidate drew separate boxes for Tenant A and Tenant B, each with their own A100 cluster. The interviewer pointed out that Tenant A's traffic peaked at 9 AM PST while Tenant B's peaked at 9 AM CET, leaving 50% of the hardware idle half the time.
The judgment here is not X (physical isolation for security), but Y (logical isolation with strict rate limiting and token quotas). The correct template uses a centralized inference cluster running vLLM or TGI (Text Generation Inference) with a sophisticated request scheduler that enforces per-tenant token budgets and priority queues. This allows you to pack requests from multiple tenants onto the same GPU batch, maximizing throughput while ensuring no single tenant starves the others.
In a debrief for a Databricks ML Platform role, the hiring manager rejected a candidate who could not explain how to handle "noisy neighbors" in a shared cluster. The candidate suggested Kubernetes namespace isolation, which is insufficient for GPU memory pressure.
The superior solution involves implementing a token-bucket algorithm at the API gateway level, specifically tied to the customer's contract tier.
For a Platinum customer paying $50,000 monthly, you guarantee 5,000 tokens per second; for a Free tier user, you cap them at 50 tokens per second and route them to a lower-priority queue that only processes when the GPU batch has empty slots. You must cite specific tools: "We use Envoy proxy with custom Lua scripts to enforce rate limits before the request hits the model server, preventing DoS attacks from a single tenant affecting the P99 latency of others."
The specific architectural pattern you must propose is the "Disaggregated Prefill-Decode" architecture, which has become the standard for cost-efficient multi-tenancy at scale. In this design, the prefill phase (processing the prompt) is separated from the decode phase (generating tokens), allowing them to scale independently. This was a key discussion point in a November 2023 interview loop at Microsoft Azure for the Copilot backend team.
The candidate who succeeded explained that prefill is compute-bound and bursty, while decode is memory-bound and steady. By routing prefill requests to a pool of compute-optimized instances (like c6i) and decode requests to memory-optimized instances (like p4de), you reduce the overall cost per token by approximately 35%. State this explicitly: "We decouple prefill and decode to optimize hardware utilization, routing short prompts to high-clock-speed CPUs for tokenization and long generations to high-bandwidth GPUs."
Which Quantization Strategies Deliver the Best Cost-to-Performance Ratio for Production?
Int8 or Int4 quantization with AWQ or GPTQ provides the optimal balance, reducing memory requirements by 75% with less than 1% accuracy loss for most enterprise applications.
Candidates frequently argue against quantization, claiming it degrades model quality too much for sensitive tasks like legal document summarization. In a interview for a Senior AI Engineer role at Harvey AI in early 2024, a candidate insisted on using FP16 precision for their design, arguing that the "client expects perfect fidelity." The interviewer countered with data showing that for RAG (Retrieval-Augmented Generation) workflows, the bottleneck is retrieval quality, not the last 0.5% of generation nuance.
The judgment is not X (maximum precision), but Y (sufficient precision for the use case at half the cost). You must demonstrate knowledge of specific quantization libraries: "We will deploy the model using vLLM with AWQ (Activation-aware Weight Quantization) to 4-bit precision, which reduces the VRAM footprint of a 70B model from 140GB to 40GB, allowing us to fit it on a single A10G instead of four A100s."
The cost implication of this decision is massive and must be quantified in your design. Running a 70B parameter model in FP16 requires roughly 140GB of VRAM, forcing you into multi-GPU tensor parallelism setups which introduce communication overhead and increase latency. By contrast, a 4-bit quantized version fits on a single 48GB GPU like the NVIDIA L40S, which costs about $1.20 per hour on-demand compared to $3.50 per hour for an A100.
Over a month, this saves $1,600 per instance. In a design review at Anthropic, engineers specifically look for candidates who can articulate the "quantization-aware training" vs "post-training quantization" trade-off. For an interview template, default to post-training quantization using GPTQ for speed, unless the prompt specifies a highly specialized domain where fine-tuning is already part of the pipeline.
You must also address the hardware compatibility of your chosen quantization method. Not all GPUs support int4 operations efficiently; older T4 instances might fall back to emulation, negating the speed benefits.
In a system design session for a fintech startup, a candidate proposed using int4 on T4s, not realizing the kernel support was suboptimal, leading to a 20% latency increase.
The correct approach is to match the quantization format to the hardware generation: "For H100s, we use FP8 via the Transformer Engine; for A10Gs and L40S, we use INT4 via AWQ." This specific detail proves you have read the NVIDIA technical blogs and aren't just reciting generic ML concepts. The template requires you to state: "We select INT4 quantization to maximize GPU memory utilization, enabling a higher batch size of 128 requests per GPU compared to 32 in FP16, directly lowering the cost per token by 4x."
> 📖 Related: Broadcom Program Manager interview questions 2026
How Do You Implement Caching Mechanisms to Reduce Redundant Inference Costs?
Semantic caching using vector similarity with a threshold of 0.95 can eliminate 30-40% of redundant LLM calls in chat and support scenarios, directly impacting the bottom line.
Most candidates treat every LLM request as a unique computation, ignoring the fact that users often ask the same questions in slightly different ways. In a design interview for a customer support automation role at Zendesk, a candidate failed to mention caching entirely, assuming the LLM would handle every query from scratch. The hiring manager noted that for a typical support dataset, 35% of queries are semantically identical to previous ones.
The judgment is not X (compute every request), but Y (check cache first, compute only on miss). Your template must include a "Semantic Cache" layer before the model server. This involves embedding the user prompt, querying a vector database like Pinecone or Redis with vector search, and returning the stored response if the cosine similarity exceeds a configurable threshold, typically 0.92 to 0.95.
The implementation details matter significantly for the interview verdict. You cannot just say "use a cache"; you must specify the invalidation strategy and the storage medium.
In a Q2 2024 loop at Coinbase, the successful candidate proposed using Redis for exact match caching of frequent queries (e.g., "What is the fee for Bitcoin?") and a vector store for semantic matches. They estimated this would reduce the total token output by 25%, saving approximately $12,000 monthly on a medium-traffic deployment. You should say: "We implement a two-tier caching strategy: an L1 exact-match cache in Redis with a 1-hour TTL for high-frequency queries, and an L2 semantic cache in Pinecone for variations, checking similarity before invoking the LLM."
Furthermore, you must address the "stale context" problem in RAG systems. If the underlying knowledge base changes, the cached response might be outdated. In a debrief for a role at Perplexity AI, the team discussed how to handle cache invalidation when source documents are updated. The winning candidate suggested tagging cached entries with a version hash of the source document.
"If the source document version changes, we invalidate all cache entries derived from it," they stated. This level of detail shows you understand the operational complexities of production LLMs. The cost benefit is clear: a cache hit costs fractions of a cent (vector search + RAM retrieval) versus dollars (GPU compute + token generation). Explicitly calculate this in your interview: "With a 40% cache hit rate, we reduce our GPU fleet size requirement from 20 instances to 12, saving $18,000 monthly."
Preparation Checklist
- Define the specific SLOs (P99 latency, availability) and cost constraints (cost per 1k tokens) before drawing any boxes; ask the interviewer for the "business goal" of the system.
- Select the right hardware class for your baseline (e.g., L40S for cost, H100 for speed) and calculate the hourly rate using current AWS or GCP pricing sheets.
- Design a quantization strategy (INT4/AWQ) and explicitly state the memory savings and batch size improvements it enables for your chosen model size.
- Incorporate a semantic caching layer with specific thresholds (0.95 similarity) and estimate the hit rate based on the use case (e.g., 35% for support bots).
- Work through a structured preparation system (the PM Interview Playbook covers system design trade-offs with real debrief examples) to ensure you can articulate the "why" behind every component.
- Prepare a "burst handling" plan using spot instances and auto-scaling policies, detailing how you handle GPU cold starts without violating SLOs.
- Draft a specific monitoring plan that tracks "cost per successful request" and "GPU utilization percentage," not just standard latency metrics.
> 📖 Related: Quantitative Analyst Interview Education for MBA Graduates Entering Quant
Mistakes to Avoid
Mistake 1: Ignoring Token Economics
BAD: "We will use the largest model available to ensure the best answers."
GOOD: "We will start with a 7B parameter model with RAG; only if quality metrics fail after A/B testing will we escalate to a 70B model, as the 7B option reduces cost per query by 85%."
Context: In a Snap Inc. interview, a candidate was rejected for proposing GPT-4 for a fun filter feature where a distilled 1B model would suffice.
Mistake 2: Over-Engineering Isolation
BAD: "Each customer gets their own VPC and dedicated GPU cluster for security."
GOOD: "We use logical isolation via Kubernetes namespaces and strict IAM policies within a shared multi-tenant cluster, utilizing pod security standards to prevent cross-tenant data leakage."
Context: At Databricks, candidates proposing physical isolation for every tenant are flagged for lacking scale intuition, as it makes the unit economics impossible.
Mistake 3: Neglecting Cold Start Latency
BAD: "We will scale down to zero instances when traffic is low to save money."
GOOD: "We maintain a warm pool of two quantized instances to handle immediate traffic spikes, accepting a small baseline cost to ensure P99 latency stays under 200ms during cold starts."
Context: A candidate for a Robinhood trading bot role failed because their "scale-to-zero" design introduced 4-second delays during market open volatility.
FAQ
Is it better to optimize for latency or cost in an LLM system design interview?
Optimize for cost within the bounds of the stated latency SLO. If the interviewer specifies a 200ms requirement, meeting that is mandatory; however, exceeding it by providing 50ms latency at 10x the cost is a failure. Judges look for the "just enough" architecture. In a Google Cloud interview, candidates who proposed expensive H100 clusters for a non-real-time summarization task were rejected for poor business judgment. Always ask for the SLO first, then design the cheapest system that meets it.
Should I mention specific open-source models like Llama-2 or Mistral in my design?
Yes, naming specific models demonstrates currency and practical knowledge, but you must justify the choice based on license and size. Do not just say "use Llama-2"; say "We chose Llama-2-13B because its Apache 2.0 license allows commercial deployment without the restrictions of Llama-2-70B, and its size fits on a single A10G." In a Meta interview, candidates who could not distinguish between the licensing terms of different model variants were viewed as risky for production deployment. Specificity signals experience.
How do I handle the "black box" nature of LLMs in a system design diagram?
Treat the LLM as a stateless service with specific input/output characteristics (token limits, latency distribution) rather than a magical box. You must define the interface: "The service accepts a prompt up to 4k tokens and returns a stream of tokens with a P99 latency of 300ms." In an Amazon Alexa design loop, candidates who treated the model as a deterministic function failed when asked about retry logic for rate-limited API calls. Acknowledge the probabilistic nature and design retries with exponential backoff specifically for 429 errors.amazon.com/dp/B0GWWJQ2S3).
TL;DR
What Is the Core Trade-off Between Latency and Cost in LLM Serving?