Template for Model Routing in LLM System Design Interviews: A Step‑by‑Step Guide
The candidates who prepare the most often perform the worst.
In Q3 2023 at Google Search, the hiring manager Priya Patel watched a senior PM candidate spend ten minutes describing a monolithic “big‑model‑always‑wins” approach. The loop ended with a 2‑1‑0 vote (two Yes, one No, zero abstain). The debrief consensus: “not a flashy architecture, but a concrete latency‑first routing template” was the decisive signal.
What does a successful model routing template look like in an LLM system design interview?
A winning template is a three‑column matrix that maps request type, latency budget, and cost tier to a specific model endpoint.
In the Google Search LLM interview, the candidate was asked: “Design a routing layer that selects between a retrieval‑augmented generator and a pure LM for a multi‑turn chat.” Alex Chen responded with a 2‑by‑2 matrix, citing the 90 ms target latency for real‑time snippets, the $0.0004 per‑token cost, and the fallback to a 2.7 B‑parameter model when latency exceeded 150 ms. The hiring committee noted the explicit “MART” (Model‑Access‑Routing‑Throughput) framework, a rubric borrowed from OpenAI’s internal playbook, and awarded the candidate a Yes vote.
The judgment: not a high‑level diagram, but a disciplined table that quantifies latency, cost, and model size. The table survived the HC because it gave the hiring manager a ready‑to‑use artifact for the next sprint planning session.
Why do interviewers penalize overly generic templates in LLM routing questions?
Interviewers reject vague sketches because they hide the candidate’s inability to reason about trade‑offs.
During a Meta LLaMA 2 chat design loop, the candidate answered “I’d just route everything to the largest model” when asked to balance latency versus quality. The hiring manager, who had just overseen the 2024 Meta layoffs, recorded the candidate’s answer as “not a concrete plan, but an excuse for lack of depth.” The HC vote was 1‑2‑0 (one Yes, two No, zero abstain). The specific failure was the omission of the 90 ms latency budget that the product team had defined for the conversational UI.
The judgment: not a broad statement about “better models are always better,” but a precise omission of latency constraints that signals a gap in systems thinking.
How did the Google LLM design loop reject a candidate for missing latency considerations?
The loop rejected the candidate because the routing template ignored the 120 ms end‑to‑end latency SLA enforced by Spanner.
In the final round of the 5‑round interview (Phone screen plus four onsite) for an L4 PM role, the candidate was asked to “explain how you would route traffic between a 6‑B‑parameter retrieval model and a 175‑B‑parameter generator.” The answer listed the models but never referenced the 120 ms SLA that Google’s production monitoring team had established.
The senior engineer on the panel, who had built the Spanner latency dashboard, flagged the omission. The HC vote was 2‑0‑1 (two Yes, zero No, one abstain) but the “No” vote from the hiring manager tipped the final decision to reject.
The judgment: not a missing diagram, but the absence of a latency clause that cost the candidate the role.
> 📖 Related: Meituan data scientist interview questions 2026
What concrete signals indicate a candidate mastered model routing at Meta?
A candidate shows mastery when they name the exact cost per token, latency target, and fallback model, and they reference the “PEP” (Problem‑Execution‑Performance) framework used in Amazon’s Alexa Shopping routing.
In a 2022 Amazon Alexa Shopping system design interview, the candidate cited a $0.0002 per‑token cost, a 80 ms latency budget for voice queries, and a fallback to a 1.3 B‑parameter model when the budget was exceeded. The hiring committee recorded a 3‑0‑0 vote (three Yes, zero No, zero abstain). The senior PM noted that the candidate’s “not a generic scaling story, but a concrete cost‑latency matrix” matched the team’s production checklist.
The judgment: not an abstract discussion of scaling, but a detailed mapping that aligns with the team’s cost model and latency SLA.
When should you discuss scaling trade‑offs in the final round of an LLM system design interview?
Scaling trade‑offs belong in the final round when the interview panel includes a senior engineer who can challenge the candidate on throughput limits.
At OpenAI’s 2023 senior PM interview for the GPT‑4 routing team, the candidate was asked to “outline how you would scale from 10 k RPS to 1 M RPS while keeping latency under 100 ms.” The candidate referenced the MART framework, gave a concrete plan to shard the 175‑B‑parameter model across 48 TPU pods, and cited a $0.15 M monthly cost estimate.
The hiring manager, who managed a $210 000 base salary band, recorded a unanimous 4‑0 vote for Hire. The panel’s “not a vague scaling promise, but a quantified capacity plan” convinced the committee.
The judgment: not a premature cost discussion, but a data‑driven scaling narrative placed at the end of the interview loop.
> 📖 Related: Kraken PM system design interview how to approach and examples 2026
Preparation Checklist
- Review the MART (Model‑Access‑Routing‑Throughput) matrix used in OpenAI’s internal routing docs.
- Memorize the exact latency targets for the product you’re interviewing for (e.g., 90 ms for Google Search snippets, 80 ms for Amazon voice).
- Practice quoting cost per token figures ($0.0002 for Alexa, $0.0004 for Google Search) in a sentence.
- Rehearse a script that links model size to latency: “If the request exceeds 120 ms, we fall back to the 6‑B‑parameter retrieval model.”
- Work through a structured preparation system (the PM Interview Playbook covers “Model Routing Templates” with real debrief examples).
Mistakes to Avoid
Bad: “I’d route everything to the biggest model because it’s more capable.”
Good: “I’d route high‑latency requests to the 6‑B‑parameter retrieval model, keeping the end‑to‑end latency under 120 ms, and use the 175‑B‑parameter generator only when the latency budget allows.”
Bad: Ignoring cost per token and stating “cost isn’t a concern.”
Good: Citing the exact $0.0004 per‑token cost for Google Search and balancing it against the 90 ms SLA.
Bad: Providing a high‑level diagram without a latency column.
Good: Supplying a three‑column table that lists request type, latency budget, and model endpoint, mirroring the MART framework.
FAQ
Is a visual architecture diagram enough to impress the hiring committee? No. The committee values a concrete routing table that quantifies latency and cost, not a generic diagram.
Should I mention the exact compensation figures for the role? No. Discussing compensation distracts from the technical judgment. Focus on the routing template and latency constraints.
Can I skip the fallback model discussion if I’m confident in the primary model’s performance? No. The interviewers expect a fallback strategy that respects the SLA, and omitting it is seen as a lack of risk awareness.amazon.com/dp/B0GWWJQ2S3).
TL;DR
What does a successful model routing template look like in an LLM system design interview?