Data Engineer Interview Pipeline Design Template: Batch and Streaming
In a Q2 2024 hiring debrief at Netflix for a Senior L6 Data Engineer role in the Content Engineering group, a candidate lost a 315,000 dollar base salary offer because they designed a real-time Apache Kafka pipeline for data that only needed daily reconciliation. The hiring committee voted 4-to-1 against hiring because the candidate prioritized architectural trendiness over business-aligned execution.
Most candidates treat data pipeline design as an exercise in drawing boxes and connecting them with arrows. In high-stakes loops, your choice of technology matters far less than your ability to justify its operational cost, resource footprint, and failure recovery characteristics.
The problem is not your architectural choice, but your inability to justify its operational cost. System design interviewers at FAANG-level companies are not looking for a perfect architecture; they are looking for your engineering trade-offs under constraint.
You must prove that you understand how data scale impacts storage cost, how network latency dictates processing choices, and how failure modes corrupt data integrity. This guide provides the exact architectural templates, decision frameworks, and communication scripts used by successful candidates to pass L5 and L6 data engineering loops at companies like Meta, Google, and Stripe.
How do FAANG companies grade data pipeline design interviews?
FAANG hiring committees grade pipeline design interviews on your ability to map business requirements to hardware constraints, not on your familiarity with specific cloud vendor services. During a Q1 2024 interview loop at Google in Sunnyvale, a candidate was asked to design a log ingestion pipeline handling 10 petabytes of data daily.
The candidate immediately started listing Google Cloud Platform services like BigQuery and Cloud Pub/Sub without calculating the network bandwidth or storage costs. The committee flagged this as a lack of senior-level engineering judgment, resulting in a down-level to L4. Under the Google SWE-DE L5 Competency Matrix, candidates are evaluated on resource efficiency, data consistency guarantees, and operational simplicity.
The grading differentiator is not the selection of a trendy tool like Apache Druid, but the cost-to-value ratio of your storage and compute choices. To pass, you must demonstrate that you can calculate the network egress costs, disk I/O bottlenecks, and CPU requirements of your pipeline before choosing your tools.
If you cannot explain why you chose parquet files over raw JSON for cold storage based on storage savings and query performance, you will fail. Your design must show a clear separation of storage and compute to prevent runaway resource costs at scale.
When an interviewer asks you to defend your architecture, you must use a structured response that links technical decisions directly to operational metrics. In that Sunnyvale loop, the successful candidate used this exact script to defend their storage format:
We will store the raw logs in Google Cloud Storage using the Apache Parquet format partitioned by event date and application ID. Given our daily ingestion volume of 10 petabytes, raw JSON would cost us approximately 200,000 dollars per month in storage alone.
By applying Snappy compression and columnar layout, we reduce this data footprint by 75 percent to 2.5 petabytes. This reduces our storage bill to 50,000 dollars per month and decreases query response times for downstream Apache Spark analytical jobs by 12 times because we can push down filters directly to the storage metadata layer.
What is the best architectural template for a batch data pipeline interview question?
The optimal batch architectural template isolates the ingestion, processing, and serving layers using immutable object storage coupled with an ACID transaction layer like Delta Lake or Apache Iceberg. During a late 2023 interview loop at Meta in Menlo Park, a candidate was tasked with designing an ad click processing pipeline handling 5 billion events per day for downstream machine learning models.
The candidate designed a brittle system where Apache Spark jobs wrote directly to a relational database, which caused write locks and corrupted data during job failures. The hiring manager noted that the design lacked basic data lakehouse principles, leading to a No-Hire decision.
A robust batch design must use a three-tier storage architecture, commonly referred to as the Medallion Architecture, to ensure data reliability and clean separation of concerns. The bronze layer stores raw, unaltered source data ingested directly from message queues or database change logs, preserving the original schema. The silver layer cleans, deduplicates, and conforms the data to a unified schema using distributed compute engines like Apache Spark or AWS Glue. The gold layer contains business-level aggregates optimized for analytical queries, stored in high-performance columnar formats.
To demonstrate mastery during the interview, you must walk through this layered architecture while addressing late-arriving data and backfill scenarios. Use this script to explain your data flow to the interviewer:
Our batch pipeline will ingest ad click events from Amazon S3 into a bronze Delta Lake table using Apache Spark structured streaming with a trigger-once interval of six hours. We will write these records using append-only semantics to prevent write amplification.
In the silver layer, we run an Apache Spark job that deduplicates events based on the click ID and event timestamp, using a 24-hour watermark to handle late-arriving data from slow mobile clients. Finally, the gold layer aggregates these clicks by advertiser ID and hour, saving the results as partitioned Parquet files, which allows the Meta Ads ML training models to query the processed features with zero read-contention.
How should I design a real-time streaming pipeline during a 45-minute interview?
Real-time streaming pipeline designs must rely on partitioned message brokers like Apache Kafka or AWS Kinesis paired with stateful processing engines like Apache Flink, focusing on exactly-once processing semantics. In February 2024, a candidate interviewing for Uber's Rider Pricing team in San Francisco was asked to design a surge pricing calculation engine handling 100,000 requests per second.
The candidate proposed using a single Redis instance to hold global state and calculated surge multipliers using a basic Python script. The interviewer rejected this design because it lacked horizontal scalability, state recovery mechanisms, and partition tolerance.
The core challenge of streaming is not tool selection, but managing out-of-order events using event-time watermarking rather than processing-time timestamps. When designing for high throughput, you must show how you partition your message queues to match your downstream processing parallelism. Your design must also account for backpressure, where the processing engine cannot keep up with the ingestion rate, by implementing rate limiting or dynamic scaling policies. If you do not mention state serialization and checkpointing in Apache Flink, the interviewer will assume you do not understand production streaming operations.
To prove your streaming expertise, you must explain how you maintain state across distributed nodes while ensuring data consistency. Use this script to describe your stream processing logic:
To compute the Uber surge pricing multiplier with sub-second latency, we will ingest location updates into an Apache Kafka topic partitioned by geohash-5, ensuring that all coordinates within a specific geographic area land on the same broker. We will consume this stream using Apache Flink, utilizing a sliding event-time window of 10 minutes with a 1-minute slide.
To handle GPS drift and network delays from rider phones, we will emit a watermark of event-time minus 30 seconds. Flink will maintain the active ride request count in its RocksDB state backend, checkpointing to Amazon S3 every 10 seconds to guarantee exactly-once processing even if a processing node fails mid-window.
> đź“– Related: Color Health PM system design interview how to approach and examples 2026
When should I choose batch processing over streaming in a system design round?
You must choose batch processing by default unless the business requirement explicitly demands latency under 10 seconds, as streaming introduces massive operational overhead, state management complexity, and elevated cloud infrastructure costs. During a Q3 2024 interview loop at Stripe in Seattle, a candidate was asked to design a financial balance reconciliation system.
The candidate proposed a real-time streaming pipeline using Apache Flink to reconcile transactions as they occurred. The hiring committee flagged this as a critical error, noting that financial reconciliation requires absolute consistency and deterministic auditability, which is highly complex to implement in streaming systems compared to scheduled batch runs.
The decision is not speed, but data consistency and deterministic reprocessing. Batch pipelines allow you to run heavy computation, complex table joins, and historical backfills with lower compute costs and simpler recovery mechanisms. If a batch job fails, you simply truncate the target partition and rerun the job for that specific execution date. In contrast, backfilling historical data in a streaming system requires rebuilding state from raw message logs, which can cost thousands of dollars in compute and introduce severe lag into your live production streams.
Your ability to push back on real-time requirements when batch is more appropriate is a strong signal of senior-level engineering maturity. Use this script to steer the interviewer toward a batch design:
While real-time streaming sounds appealing for the Stripe balance reconciliation system, I recommend a daily batch processing pipeline using Apache Spark and Airflow. Financial reporting requires absolute data consistency, transaction completeness, and deterministic reprocessing for audits.
A batch approach allows us to perform multi-way joins between the payment processor logs, bank settlement files, and our internal ledger at midnight when all source files are finalized. This avoids the state storage overhead of keeping millions of unsettled transactions in memory and simplifies backfills, as we can easily re-run a specific day's Spark job using a simple date partition overwrite.
How do interviewers evaluate scale and fault tolerance in pipeline designs?
Interviewers evaluate scale and fault tolerance by forcing you to break your own system, specifically checking how you handle consumer lag, network partitions, and poison pill messages. In January 2024, a candidate for Amazon's Prime Video Analytics team in Seattle was asked to design a quality-of-service monitoring pipeline for 50 million concurrent streams.
During the deep dive, the interviewer asked what would happen if the database target became unavailable for two hours. The candidate suggested pausing the upstream ingestion, which would have caused millions of client devices to drop connections and lose telemetry data, leading to a weak-hire rating.
Fault tolerance is not about avoiding failures, but about defining idempotent write paths and dead-letter queues. You must show that your pipeline can handle duplicate data delivery, which is common in distributed systems that guarantee at least-once processing. This means your downstream databases or processing engines must perform deduplication using unique transaction IDs or upsert operations. Additionally, you must design a mechanism to isolate malformed payloads, known as poison pills, so they do not block the processing of valid data across your entire cluster.
When explaining your fault tolerance strategy, you must walk through a concrete recovery scenario step-by-step. Use this script to demonstrate how your pipeline recovers from downstream database outages:
If our target Snowflake database experiences a two-hour outage, our Apache Spark streaming application will begin experiencing write failures. To prevent data loss, we will configure our Spark writers with an exponential backoff retry policy of up to 5 attempts.
If the outage persists, the Spark job will fail, but our upstream Apache Kafka cluster, configured with a 7-day data retention policy, will continue to buffer the incoming Prime Video telemetry events safely. Once Snowflake is back online, we will restart the Spark application from its last committed Kafka offsets. To handle the sudden surge in data, we will scale up our Spark executor count from 50 to 150 instances, leveraging dynamic allocation to clear the consumer lag without dropping any records.
> 📖 Related: Review: Does SWE面试Playbook Actually Help with Palantir FDE Interview Data Modeling Questions?
Preparation Checklist
Review the system design and technical execution modules in the PM Interview Playbook to align your engineering architecture with product-driven business metrics and user-experience constraints.
Practice calculating storage and network egress costs for a 100-terabyte daily ingestion scale, assuming a 3-to-1 compression ratio for columnar formats like Apache Parquet.
Memorize the specific architectural trade-offs between Apache Kafka, AWS Kinesis, Apache Spark Streaming, and Apache Flink, focusing on state management and windowing capabilities.
Write out a step-by-step backfill strategy for a batch pipeline, explaining how to handle partition overwrites and downstream dependency trigger patterns.
Prepare three real-world examples of how you handled data quality issues, such as schema drift or late-arriving data, in your previous engineering roles.
Draw a complete data pipeline template on a physical whiteboard or digital tool like Excalidraw, clearly labeling the ingestion, storage, processing, and serving layers.
Mistakes to Avoid
Over-engineering with streaming when batch is sufficient
Candidates often design complex, real-time streaming architectures using Apache Flink and Kafka for use cases that only require daily or hourly reporting, leading to high operational costs and fragile state management.
BAD: Designing a real-time streaming pipeline using Apache Kafka and Apache Flink to calculate monthly active users for an internal dashboard that managers only view on Monday mornings.
GOOD: Designing an hourly batch pipeline using Apache Airflow to orchestrate Apache Spark SQL jobs that write aggregated monthly active user metrics to a Snowflake data warehouse partition.
Lacking concrete data volume and cost calculations
Proposing architectural designs without calculating the underlying data volume, network bandwidth, storage footprint, and compute costs makes your system design feel theoretical rather than practical.
BAD: Saying "We will ingest all user clickstream events into BigQuery and run SQL queries to calculate real-time analytics for our application users."
GOOD: Saying "With 1 billion daily events averaging 1 kilobyte each, we ingest 1 terabyte of raw data daily. We will stage this in Google Cloud Storage for 20 dollars per month before running daily Spark aggregation jobs, which limits our expensive BigQuery interactive query scans to pre-aggregated gold tables."
Ignoring the mechanism for handling duplicate data
Assuming that your pipeline will always process data exactly once without explicitly designing for deduplication or idempotent writes, which leads to corrupted metrics and inaccurate data states.
BAD: Writing records directly to a relational database using simple INSERT statements, assuming the upstream message broker guarantees exactly-once delivery.
GOOD: Writing records to the target database using an UPSERT command keyed on a unique transaction UUID, or using an ACID storage layer like Delta Lake to perform merge operations that prevent duplicate writes.
FAQ
How do I handle schema drift in an interview design?
You must design your ingestion layer to use a schema registry like Confluent Schema Registry for streaming or enforce schema evolution rules in your storage layer like Delta Lake. When a producer changes the schema, the registry validates compatibility, preventing poison pills from breaking downstream Spark processing jobs.
What is the best database for a real-time serving layer?
The best database depends on the query pattern, but for real-time analytics, choose a columnar OLAP database like ClickHouse, Apache Pinot, or Amazon Redshift. These databases support high-throughput writes and sub-second aggregations over millions of rows, unlike transactional databases like PostgreSQL which bottleneck under analytical workloads.
How do I handle late-arriving data in streaming?
Handle late-arriving data using event-time watermarks and side outputs in Apache Flink or Apache Spark Streaming. The watermark defines how long the system waits for delayed events before closing a time window, while any data arriving after the watermark is routed to a side-output dead-letter queue for manual reconciliation.amazon.com/dp/B0GWWJQ2S3).
TL;DR
How do FAANG companies grade data pipeline design interviews?