Kubernetes GPU Scheduling Issues in LLM Training: A PM's Guide to Debugging

The hiring manager’s inbox pinged at 09:13 UTC on 15 May 2024. Priya Patel from Google Cloud AI slammed a PM candidate’s screen share: “Your pod is still Pending after you asked for eight NVIDIA A100s. Explain why.” The candidate, fresh from a Harvard‑style ML bootcamp, muttered “nodeSelector = 8” while the loop timer hit the 45‑minute mark. The debrief later that afternoon logged a 5‑1 No Hire vote. The problem wasn’t the answer. It was the candidate’s missing judgment signal about Kubernetes device‑plugin mechanics.

Why do GPU pods get stuck in Pending in Kubernetes for LLM workloads?

Answer: Pods stay Pending when the scheduler cannot match the requested GPU count to any node because the device plugin has not advertised enough allocatable GPUs, or because node‑level taints block the pods.

Details: Google Cloud AI, Q2 2024 PM loop for Vertex AI Training; interview question “Explain why your Kubernetes job stayed Pending when you requested 8 GPUs for a GPT‑3 fine‑tune”; candidate answer “I set nodeSelector to nvidia.com/gpu=8”; debrief vote 5‑1 No Hire; hiring manager Priya Patel cited “lack of awareness of device plugin constraints”; framework used Google’s Resource Allocation Rubric (RAR); compensation $190,000 base, 0.04% equity.

The debrief room smelled of burnt coffee and stale PowerPoint slides. Priya Patel opened with “The problem isn’t your nodeSelector syntax — it’s that you ignored the device plugin’s allocatable field.” Jake Liu from Amazon SageMaker had previously warned that “nodeSelector alone never forces the scheduler to see GPUs; the plugin must first publish the resource.” The RAR checklist asked for a “Device‑Plugin Health Check” item, which the candidate skipped.

Not “just a YAML typo,” but “a missing verification of plugin readiness.” The loop’s senior PM, Anika Shah, added that the scheduler will silently reject the pod if the node’s kubelet reports zero allocatable GPUs. The interviewers then noted the candidate’s lack of a concrete signal: no “kubectl get nodes -o jsonpath” dump, no “nvidia‑smi” output, no “DevicePluginList” API call. The verdict: a No Hire because the candidate treated the GPU request as a static label rather than a dynamic resource advertised by the plugin.

How can I verify that the device plugin is correctly reporting GPU resources?

Answer: Run kubectl get nodes -o jsonpath='{.status.allocatable}' and cross‑check the numbers against nvidia‑smi on each node; any mismatch signals a plugin mis‑registration.

Details: Amazon SageMaker, March 2023 PM interview; question “How would you confirm the GPU count reported by the NVIDIA device plugin?”; candidate reply “I’d kubectl describe node”; debrief vote 4‑2 No Hire; hiring manager Jake Liu flagged “no cross‑validation with driver output”; framework Amazon Fault Isolation Checklist (FIC); compensation $185,000 base.

In the debrief, Jake Liu replayed the candidate’s screen: “You ran kubectl describe node and saw nvidia.com/gpu: 0. Did you then run nvidia‑smi?” The candidate shook his head.

The FIC required a “Dual‑Source Validation” step, which the candidate omitted. Not “just a missing flag,” but “a missing sanity check that reveals a dangling device‑plugin socket.” Priya Patel reminded the panel that the device plugin registers via a gRPC server; if the server crashes, the plugin reports zero GPUs while the node still shows GPU hardware. Carlos Gómez from Microsoft Azure ML later added that “the plugin’s health endpoint must return Ready before the scheduler will consider the node.” The interviewers logged a decisive No Hire because the candidate’s diagnostic method lacked the required two‑pronged verification, a signal that he would not catch subtle plugin regressions in production.

> 📖 Related: [](https://sirjohnnymai.com/blog/meta-vs-uber-pm-role-comparison-2026)

What is the impact of pod priority and preemption on LLM training throughput?

Answer: High‑priority pods can preempt lower‑priority workloads, but preemption incurs a cold‑start delay of 2–3 minutes per GPU, which dramatically lowers overall LLM training throughput.

Details: Meta AI Infra, June 2024 HC for Infra PM; question “What happens if you schedule a high‑priority LLM pod that exceeds cluster capacity?”; candidate answer “Scheduler will evict lower‑priority pods”; debrief vote 3‑3 Tie, final decision No Hire; hiring manager Anika Shah cited “no quantification of preemption cost”; framework Meta Preemption Impact Matrix; compensation $200,000 base.

Anika Shah opened the debrief with a slide showing a 2‑minute GPU warm‑up curve captured on a 40‑node cluster on 12 May 2024. “The candidate said eviction is free,” she said.

“Not “just a scheduling detail,” but “an invisible latency that adds up to hours of wasted compute.” The Preemption Impact Matrix required candidates to model the cost of pod termination, checkpoint reload, and GPU memory re‑initialization. The candidate offered no numbers, only a vague “it’s fast.” The panel referenced a Meta internal benchmark where a preempted 8‑GPU BERT‑large job lost 12 % of epoch time due to cold‑starts. The tie resulted from a senior PM arguing that “preemption is a feature, not a bug,” while another argued it “is a throughput killer if not quantified.” The final No Hire came because the candidate could not demonstrate the ability to measure and mitigate preemption penalties, a core requirement for LLM‑scale infra.

When should I switch from the default scheduler to a custom scheduler for LLM GPU allocation?

Answer: Deploy a custom scheduler like Volcano when you need advanced features such as gang scheduling, node affinity, and GPU topology awareness that the default scheduler lacks.

Details: Microsoft Azure ML, September 2023 PM interview; question “When would you replace the default scheduler with Volcano for GPU jobs?”; candidate answer “When you need node affinity”; debrief vote 4‑2 Yes Hire (later rescinded for unrelated reasons); hiring manager Carlos Gómez; framework Azure Scheduler Selection Guide; compensation $192,500 base.

Carlos Gómez presented a screenshot of an Azure ML pipeline from 28 Oct 2023 where the default scheduler failed to co‑locate four A100 GPUs for a GPT‑2 fine‑tune, resulting in a “Insufficient resources” error. He asked the candidate to articulate the trigger point.

The candidate said “once you need node affinity.” Gómez added, “Not ‘just a preference flag,’ but ‘a need for gang scheduling that ensures all GPUs start together.’” The Scheduler Selection Guide required a justification that includes “GPU topology constraints” and “cross‑node bandwidth considerations.” The candidate’s answer matched the guide’s first bullet, earning a Yes Hire vote. However, the later rescind was unrelated to the scheduling discussion. The key judgment: the candidate recognized the precise scenario—when gang scheduling and topology‑aware placement are essential—showing the depth expected of a PM overseeing LLM training pipelines.

> 📖 Related: kubernetes-vs-serverless-security-for-faang-cloud-engineer-interview

Preparation Checklist

  • Review the Kubernetes Device Plugin API version v1.0.0 release notes (June 2024) for NVIDIA‑specific fields.
  • Simulate a pending pod on a 4‑node test cluster using kubectl apply -f pending‑gpu‑pod.yaml.
  • Run nvidia‑smi -L on each node and capture the output into a gpu‑inventory.txt file.
  • Compare kubectl get nodes -o jsonpath='{.status.allocatable}' against gpu‑inventory.txt to spot mismatches.
  • Practice explaining the Preemption Impact Matrix using the Meta internal benchmark from 12 May 2024.
  • Work through a structured preparation system (the PM Interview Playbook covers “GPU Scheduling Debugging” with real debrief examples).
  • Draft a one‑page cheat sheet of Volcano vs. default scheduler trade‑offs for LLM workloads.

Mistakes to Avoid

BAD: “Just set nodeSelector: nvidia.com/gpu: 8 and hope the scheduler finds a node.” GOOD: Verify the device plugin’s allocatable count first, then use resourceRequests with limits to guarantee scheduling.

BAD: “Assume preemption has no cost because the scheduler will automatically rebalance.” GOOD: Quantify the cold‑start delay (2–3 minutes per GPU) and include it in throughput calculations.

BAD: “Switch to Volcano only when the default scheduler fails with a generic error.” GOOD: Deploy Volcano when you need gang scheduling, node affinity, and topology‑aware placement, as outlined in the Azure Scheduler Selection Guide.

FAQ

When does a Pending status indicate a device‑plugin issue versus a node‑capacity problem?

If kubectl get nodes shows nvidia.com/gpu: 0 while nvidia‑smi lists physical GPUs, the device plugin is mis‑reporting; that is the decisive signal.

How can I prove to a hiring panel that I understand preemption costs?

Present a latency chart from a Meta internal benchmark (12 May 2024) showing a 2‑minute GPU warm‑up per preempted pod, and calculate the aggregate epoch time loss.

Is Volcano the only custom scheduler for LLM GPU jobs?

No, Volcano is a common choice, but the Azure Scheduler Selection Guide also lists YuniKorn and Kube‑Scheduler‑Framework as viable alternatives when gang scheduling is required.amazon.com/dp/B0GWWJQ2S3).

Related Reading

Why do GPU pods get stuck in Pending in Kubernetes for LLM workloads?