🏭Data engineering concepts

This is DataChef's day job: data platforms on AWS with Spark, Kafka, Airflow and dbt. The contest may have conceptual questions or pipeline-logic problems, and the interview will certainly cover this. Learn the vocabulary and the "why".

Pipelines: ETL vs ELT, batch vs streaming

Medallion / layered architecture

Data lake: files in object storage (S3) plus a catalog. Warehouse: managed SQL with columnar storage. Lakehouse: a lake plus table formats (Delta Lake, Apache Iceberg, Hudi) that add ACID, time travel, MERGE and schema evolution on top of Parquet.

Data Mesh (DataChef's specialty)

1. Domain ownership: the teams that produce data own it end-to-end.
2. Data as a product: discoverable, addressable, trustworthy, with an SLA, schema and docs.
3. Self-serve data platform: a central platform team provides the tooling (e.g. DataChef's AWS Data Landing Zone).
4. Federated computational governance: global standards (security, interoperability) enforced automatically.

The properties of a good pipeline

CDC & delivery semantics

• CDC (change data capture): read the DB's log (MySQL binlog, Postgres WAL) with Debezium → Kafka. You get insert/update/delete events, with op, before, after and ts.
• At-most-once (may lose), at-least-once (may duplicate; the most common), exactly-once (idempotent producer, transactions, or at-least-once plus an idempotent sink). In practice, choose at-least-once and dedupe on a key.
• Late and out-of-order events: use the event time vs processing time distinction, plus watermarks ("I've probably seen everything before T−10 min"). Window types are tumbling (fixed, non-overlapping), sliding/hopping (overlapping) and session (gap-based).

Apache Spark essentials

from pyspark.sql import SparkSession, functions as F, Window
spark = SparkSession.builder.getOrCreate()
orders = spark.read.parquet("s3://lake/silver/orders/")            # lazy
w = Window.partitionBy("customer_id").orderBy(F.col("updated_at").desc())
latest = (orders
          .withColumn("rn", F.row_number().over(w))
          .filter("rn = 1").drop("rn"))                             # transformations: lazy
daily = (latest.groupBy(F.to_date("order_ts").alias("dt"))
               .agg(F.sum("amount").alias("revenue"),
                    F.countDistinct("customer_id").alias("buyers")))
(daily.write.mode("overwrite").partitionBy("dt")
      .parquet("s3://lake/gold/daily_revenue/"))                    # action: triggers the job

• Lazy evaluation: transformations (select, filter, withColumn, join, groupBy) build a DAG. Actions (count, collect, show, write) execute it.
• Narrow transformations (map, filter) need no data movement. Wide ones (groupBy, join, distinct, repartition) shuffle data and create stage boundaries. Shuffles are expensive.
• Partitions = units of parallelism. repartition(n) does a full shuffle; coalesce(n) only reduces partitions with no full shuffle. Too many tiny output files is the small files problem.
• Joins: broadcast hash join (small table under ~10 MB by default; hint with F.broadcast) vs sort-merge join (big × big). Data skew (one hot key) can be fixed with salting or AQE skew-join handling.
• cache()/persist() pays off when a DataFrame is reused. Avoid Python UDFs (row-by-row serialization) and prefer built-in F.* functions.
• Catalyst optimizer and predicate/projection pushdown into Parquet. AQE (Adaptive Query Execution) re-optimizes at runtime.

File formats

Partitioning on disk (Hive-style): s3://bucket/events/dt=2026-10-23/hour=16/part-0000.parquet. Partition on low-cardinality columns that queries filter on (date), never on user_id.

Kafka in 60 seconds

• Topic → partitions (ordered, append-only logs) → messages with offsets. Ordering is guaranteed per partition only, and the message key decides the partition.
• Consumer group: each partition goes to exactly one consumer in the group, so parallelism ≤ #partitions. Consumers commit offsets.
• Retention by time/size, or log compaction (keeps the latest value per key, giving you a changelog/table).
• Replication factor, ISR, acks=all for durability. The schema registry holds Avro/Protobuf schemas.

Orchestration & transformation tools

• Airflow: DAGs of tasks, scheduler, operators/sensors, execution_date/logical date, retries, backfill and catchup. Tasks should be idempotent and atomic. Use XCom only for small metadata. Alternatives: Dagster, Prefect, AWS Step Functions.
• dbt: SQL SELECT models plus Jinja {{ ref('stg_orders') }}, which builds the dependency DAG. Materializations: view / table / incremental / ephemeral. Tests: unique, not_null, relationships, accepted_values. Snapshots give you SCD2.

AWS data services map

What these look like in code

An idempotent Airflow DAG

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

def load_orders(data_interval_start, data_interval_end, **_):
    # read ONLY this run's interval, write ONLY this run's partition → reruns are safe
    dt = data_interval_start.strftime("%Y-%m-%d")
    df = extract(since=data_interval_start, until=data_interval_end)
    df.to_parquet(f"s3://lake/bronze/orders/dt={dt}/", engine="pyarrow")   # overwrite the partition

with DAG("orders_daily", start_date=datetime(2026, 10, 1), schedule="@daily", catchup=True,
         default_args={"retries": 3, "retry_delay": timedelta(minutes=5)}) as dag:
    load = PythonOperator(task_id="load_orders", python_callable=load_orders)
    dq = PythonOperator(task_id="check_orders", python_callable=check_row_counts)
    load >> dq                                   # dependency: load runs before the check

A dbt incremental model

-- models/silver/orders.sql
{{ config(materialized='incremental', unique_key='order_id') }}

SELECT order_id, customer_id, status, amount, updated_at
FROM {{ ref('stg_orders') }}
{% if is_incremental() %}
  WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})   -- watermark on the target table
{% endif %}
-- with unique_key set, dbt MERGEs new versions over old rows instead of appending duplicates

Spark Structured Streaming with a watermark

events = (spark.readStream.format("kafka")
          .option("subscribe", "clicks").load()
          .select(F.from_json(F.col("value").cast("string"), schema).alias("e")).select("e.*"))
counts = (events
          .withWatermark("event_time", "10 minutes")                 # accept events up to 10 min late
          .groupBy(F.window("event_time", "5 minutes"), "page")      # tumbling 5-min windows
          .count())
(counts.writeStream.outputMode("append")
       .option("checkpointLocation", "s3://lake/_chk/clicks/")       # offsets + state → exactly-once sink
       .format("delta").start("s3://lake/silver/click_counts/"))

Architecture vocabulary

• Lambda architecture: a batch layer (complete, slow) plus a speed layer (approximate, fast), merged at query time. Two code paths to maintain.
• Kappa architecture: one streaming path; to reprocess, replay the log (Kafka) from the beginning.
• Reverse ETL: push warehouse results back into operational tools (CRM, ads).
• Data contract: an agreed schema + semantics + SLA between a producer and its consumers, checked in CI.
• Schema evolution: backward compatible = new readers can read old data (add a field with a default); forward compatible = old readers can read new data (remove an optional field). Renaming a field breaks both.
• Partition pruning: the engine skips partitions that the WHERE clause rules out. It only works when you filter on the partition column itself, not on a function of it.
• Backpressure: a slow consumer makes the system slow down the producer (or buffer), instead of falling over.
• Lineage: which jobs and tables produced this column. Needed for impact analysis and GDPR.

Whiteboard warm-up: three classic design questions