LLM System Design Interview Template: Batching Strategy for Inference
The candidates who prepare the most often perform the worst. In a Google DeepMind interview on 12 Oct 2024, the top‑scoring applicant spent the entire 30‑minute design segment sketching a monolithic transformer diagram, yet the hiring committee rejected him 5‑2 because his batching logic ignored the 150 ms 99th‑percentile latency target that the SSD rubric explicitly penalises.
How should I structure a batch inference design for an LLM interview?
Answer: Begin with a two‑layer diagram that separates request routing from GPU‑level batching, then quantify the batch‑size‑to‑latency curve before presenting cost estimates.
In the Q3 2024 hiring cycle at Google DeepMind, the interview panel asked the candidate to “design a system that can serve 10 k concurrent LLM inference requests while keeping 99th‑percentile latency under 150 ms.” The candidate immediately drew a single “Inference Service” box, cited the NVIDIA A100 40 GB GPU, and claimed a batch size of 64 tokens would be sufficient.
The hiring manager, Maya L., interrupted: “You have not shown how the router decides which requests join a batch, nor how you handle stragglers that break the latency SLA.” The debrief vote later reflected a 5‑2 split, with the two dissenters citing the missing “batch orchestration layer” as a fatal omission.
The judgment is that a correct answer must articulate a router → batcher → executor pipeline, provide a concrete batch‑size‑vs‑latency curve (e.g., 32‑token batches achieve 120 ms, 64‑token batches spike to 190 ms), and reference the Google SSD rubric item “Scalability of batching”. Not “just a monolith”, but a two‑stage orchestration that isolates latency‑critical path.
What metrics do interviewers actually evaluate in a batching strategy?
Answer: Interviewers score latency distribution, GPU utilisation, cost per token, and the clarity of failure‑mode handling.
During a Meta LLM System Design loop on 5 Nov 2023, the senior PM asked the candidate to “explain how you would monitor and mitigate tail latency in a 12‑engineer inference cluster.” The candidate listed “average latency” and “GPU memory usage”, but omitted the 99th‑percentile latency and cost per 1 k tokens ($0.12).
The hiring committee applied the STAR+ metric from Amazon Alexa, which awards a point for each of the four pillars: Specificity, Tail‑latency, Affordability, Resilience, and + (future‑proofing). The candidate earned 2/5, leading to a 4‑3 “hire” vote that was later rescinded when the panel revisited the cost model.
The judgment is that interviewers care about tail latency and cost efficiency, not merely average numbers. Not “GPU utilisation alone”, but a balanced scorecard that includes the 99th‑percentile, cost per token, and explicit fallback paths for batch‑size overflow.
Which trade‑offs are decisive in a Google DeepMind system design loop?
Answer: The decisive trade‑off is between batch size (throughput) and per‑request latency (SLA), with a secondary focus on memory fragmentation.
In the same DeepMind interview, the candidate argued for a static batch size of 128 tokens to maximise GPU throughput, citing the NVIDIA A100’s peak TFLOPs.
The hiring manager, Ravi K., counter‑questioned: “If a high‑priority request arrives while the batch is full, how do you guarantee the 150 ms SLA?” The candidate responded, “We’ll pre‑empt lower‑priority batches,” but offered no concrete algorithm. The debrief notes recorded a 5‑2 pass because the panel valued dynamic batch sizing that adapts to request priority, as outlined in the internal “Adaptive Batching” design doc dated 3 Mar 2024.
The judgment is that dynamic batching—e.g., a “sliding‑window” that caps batch latency at 50 ms—wins over static high‑throughput configurations. Not “maximise GPU utilisation”, but preserve latency guarantees while still achieving ≥70 % utilisation.
> 📖 Related: Yardi PM behavioral interview questions with STAR answer examples 2026
Why does the candidate’s answer often miss the real judgment signal?
Answer: The real judgment signal is a candidate’s ability to expose and mitigate failure modes, not merely to list components.
At an Amazon Alexa interview on 22 Sep 2023, the interviewee described a “simple FIFO queue” for request aggregation, then quoted a paper on “large‑batch inference”. The hiring lead, Priya M., noted in the debrief that the candidate “never mentioned what happens when a batch stalls due to a single long‑running prompt”. The committee recorded a 4‑3 reject because the “failure‑mode awareness” rubric item carried double weight. The candidate’s quote—“I’d just A/B test it” when asked about handling out‑of‑memory errors—was flagged as lack of concrete mitigation.
The judgment is that interviewers penalise candidates who omit explicit error handling, even if they name cutting‑edge techniques. Not “knowing the latest paper”, but demonstrating operational robustness.
When does a debrief turn a borderline candidate into a hire for LLM inference design?
Answer: A debrief flips a borderline case when a senior engineer champions the candidate’s nuanced cost model, outweighing a single metric shortfall.
In a Meta hiring committee on 8 Dec 2023, the candidate’s design missed the 150 ms target by 20 ms but included a detailed cost per token breakdown showing a $0.10 saving versus the baseline. Senior engineer Luis G. argued that the cost advantage justified a hire despite the latency miss, resulting in a 5‑2 vote after a brief “cost‑first” discussion. The final offer package listed $210,000 base, $30,000 sign‑on, and 0.04 % equity, reflecting the market rate for LLM PMs in the Bay Area.
The judgment is that a strong business case can outweigh a modest latency deficit, but only when the candidate backs it with precise numbers. Not “perfect latency”, but quantified cost trade‑offs that align with product ROI.
> 📖 Related: Payment for Order Flow at Robinhood: System Design Challenges for Senior Engineers
Preparation Checklist
- Review the Google SSD rubric and note the latency‑distribution items.
- Memorise the STAR+ metric used by Amazon Alexa for system‑design evaluation.
- Build a quick‑draw sketch of a router → batcher → executor pipeline on a whiteboard.
- Calculate a batch‑size‑to‑latency curve for an NVIDIA A100 40 GB (e.g., 32‑token batch = 120 ms, 64‑token batch = 190 ms).
- Draft a cost per 1 k token model using $0.12 as the baseline inference price.
- Prepare a failure‑mode checklist (out‑of‑memory, straggler request, GPU pre‑empt).
- Work through a structured preparation system (the PM Interview Playbook covers “Dynamic Batching with Real‑World De‑brief Examples” with actual debrief excerpts).
Mistakes to Avoid
BAD: “I would just increase the batch size until the GPU memory fills up.”
GOOD: “I would start with a 32‑token batch, monitor GPU memory, and dynamically shrink the batch if a request exceeds the 150 ms SLA, as demonstrated in the Adaptive Batching paper (Mar 2024).”
BAD: “Latency is the only metric that matters.”
GOOD: “Latency, cost per token, and failure‑mode handling each carry weight in the SSD rubric; I would present a balanced scorecard that includes 99th‑percentile latency, $0.12 per 1 k tokens, and a fallback path for OOM errors.”
BAD: “My design is a single monolithic service.”
GOOD: “I split the system into a lightweight router that assigns priority, a batcher that groups requests up to 32 tokens, and an executor that runs on an NVIDIA A100, achieving 70 % utilisation while respecting the 150 ms SLA.”
FAQ
What concrete numbers should I quote to prove I understand batching costs?
State the per‑token inference price ($0.12 per 1 k tokens), the GPU utilisation target (≥70 %), and the exact latency SLA (150 ms 99th‑percentile). Interviewers treat these figures as the “judgment signal”, not vague cost‑saving claims.
How do I demonstrate failure‑mode awareness without sounding rehearsed?
Quote a specific scenario: “If a request exceeds the batch timeout, the router immediately forwards it to a dedicated low‑latency executor, preventing SLA breach.” Mention the exact timeout (e.g., 50 ms) and the fallback executor (a separate A100 instance).
When is it acceptable to trade latency for higher throughput?
Only when you can justify the trade‑off with a cost model that shows a >5 % reduction in inference spend, and when the product team accepts a higher SLA (e.g., 200 ms instead of 150 ms). The debrief will penalise un‑justified latency increases.amazon.com/dp/B0GWWJQ2S3).
TL;DR
How should I structure a batch inference design for an LLM interview?