Interview preparation

PySpark Interview Questions and Answers

Practical PySpark questions for data engineers, from DataFrame fundamentals to joins, shuffles, partitioning and production performance.

  • 20 questions
  • Beginner → Senior
  • Production-focused

PySpark interviews quickly move past syntax to how Spark actually runs your code: what triggers work, where data moves, why one task takes an hour while the rest finish in seconds. Answers refer to open-source Apache Spark 3.x and 4.x behavior, and each is written to be spoken in your own words: a short answer, why it matters in production, what a strong answer usually demonstrates, the traps to avoid, and where to go deeper.

Ce guide est disponible en anglais.

PySpark interview preparation workspace with DataFrames, joins, shuffles, partitions, Spark execution plans and performance troubleshooting.

At a glance

How the levels work

Q01BeginnerPySpark Fundamentals

What is PySpark?

Short answer

PySpark is the Python API for Apache Spark, a distributed engine for processing large datasets across a cluster. You write Python, mostly with DataFrames and Spark SQL, and Spark turns it into an optimized plan that runs in parallel on many executors. Transformations describe the work, and actions trigger it. That makes it useful when data is too big for one machine, or when you need to process it in parallel with fault tolerance.

Why it matters in production

Most PySpark code is not executed by Python line by line. The driver builds a plan, the JVM-based engine optimizes and runs it, and data stays distributed across executors. Understanding that split explains most performance behavior: why built-in functions are fast, why collect() is dangerous, and why moving data between machines is the main cost.

What the interviewer is looking for

A strong answer usually demonstrates that you know PySpark is a client API for a distributed engine, that the DataFrame and SQL APIs are the normal way to use it, and that work happens lazily until an action runs.

Watch out

Saying PySpark is “pandas for big data.” The APIs look similar in places, but the execution model is completely different: distributed, lazy and optimized as a whole plan. Code written in a pandas style (row loops, collecting to the driver, Python functions everywhere) is usually slow in Spark.

Back to all questions ↑

Q02BeginnerPySpark Fundamentals

What is the difference between a transformation and an action in PySpark?

Short answer

A transformation describes a new DataFrame from an existing one, such as select, filter, withColumn, join or groupBy. It doesn’t process any data; it adds a step to the plan. An action asks for a result, such as count, collect, show, or writing to storage, and that’s what makes Spark actually run the plan. This is lazy evaluation: Spark waits until an action so it can optimize the whole chain at once.

Why it matters in production

Every action runs the plan again from its sources unless intermediate results are cached. A notebook that calls count() after every step to “check progress” can read the same data many times. Knowing which calls are actions explains where time is really spent.

What the interviewer is looking for

A strong answer usually demonstrates that you can classify common calls correctly and connect actions to jobs: each action triggers at least one Spark job, which you can see in the Spark UI.

Watch out

Assuming df.cache() or df.repartition() does work immediately. Both are lazy and only take effect when an action runs. Also, writing a DataFrame is an action, so a pipeline with several writes from the same lineage recomputes shared steps unless you plan for it.

Example

orders = spark.read.table("orders")                 # no data read yet
recent = orders.filter("order_date >= '2026-01-01'") # transformation
totals = recent.groupBy("customer_id").sum("total")  # transformation
totals.write.mode("overwrite").saveAsTable("customer_totals")  # action: the job runs here

Back to all questions ↑

Q03BeginnerSpark Execution

What does lazy evaluation mean in Spark?

Short answer

Transformations aren’t executed when you call them. Spark records them as a logical plan, and only when an action runs does it optimize the whole plan and execute it. The optimizer, Catalyst, can then reorder and combine steps: pushing filters down to the source, reading only the columns you use, and choosing join strategies. So lazy evaluation is what lets Spark see the full job before running any of it.

Why it matters in production

It explains several surprises: an error in a transformation only appears when an action runs, sometimes many cells later; timing a single transformation cell measures almost nothing; and two actions on the same DataFrame run the work twice unless it is cached.

What the interviewer is looking for

A strong answer usually demonstrates that you connect laziness to optimization (filter and column pushdown, join selection) and to debugging, and that you can read the result with df.explain().

Watch out

Saying “nothing happens until an action.” Some work does happen on the driver earlier: Spark analyzes the plan, checks that columns and tables exist, and may read metadata or infer a schema, which can scan files when reading CSV or JSON without a schema. Lazy means data processing is deferred, not that the driver is idle.

Example

df = spark.read.parquet("/data/events").filter("country = 'ES'").select("user_id", "ts")
df.explain()   # shows the plan: the filter and column list are pushed into the Parquet scan

Back to all questions ↑

Q04BeginnerDataFrames & Transformations

What is a Spark DataFrame?

Short answer

A Spark DataFrame is a distributed collection of rows with a schema: named, typed columns, split into partitions across the cluster. You work with it through column expressions, like col("price") * col("qty"), or SQL. Because Spark knows the schema and sees the whole plan, the Catalyst optimizer can rewrite it, and the engine can store and process the data efficiently. Unlike a pandas DataFrame, it isn’t held in one process’s memory, and it’s immutable: every transformation returns a new DataFrame.

Why it matters in production

The schema and expressions are what make Spark fast. When you stay within DataFrame operations and built-in functions, Spark can optimize and generate efficient code; when you drop to Python functions or the RDD API, much of that is lost.

What the interviewer is looking for

A strong answer usually demonstrates that you understand schema + distributed partitions + optimizable expressions, and the practical differences from pandas: immutability, laziness, no row order unless you sort, and no positional index.

Watch out

Describing it as “a distributed Python list” or “a big pandas DataFrame.” Rows are not Python objects inside the engine, and there is no implicit order. Also avoid relying on schema inference in production: explicit schemas make reads faster and catch source changes.

Back to all questions ↑

Q05BeginnerPySpark Fundamentals

When would you use PySpark instead of pandas?

Short answer

I’d use PySpark when the data or the processing doesn’t fit comfortably on one machine. That means datasets larger than memory, heavy joins and aggregations over billions of rows, or work that benefits from running in parallel on a cluster. It also fits when the platform is already Spark-based, such as a lakehouse with Delta tables. For small or medium data, pandas, or another single-machine library, is usually simpler and faster, because Spark has startup, planning and coordination overhead.

Why it matters in production

Choosing the wrong tool costs in both directions: pandas jobs that crash with out-of-memory errors as data grows, or Spark clusters spinning up for a few megabytes of data and taking minutes for work that would take a second locally.

What the interviewer is looking for

A strong answer usually demonstrates that you choose based on data size, growth, the platform and the team, and know that the two can work together: Spark for the heavy lifting, then a small aggregated result converted to pandas for analysis or plotting.

Watch out

Saying PySpark is automatically faster. On small data it is usually slower. Also avoid converting large Spark DataFrames to pandas with toPandas(): everything is pulled into the driver’s memory, which brings back exactly the limit Spark was meant to remove.

Back to all questions ↑

Q06WorkingDataFrames & Transformations

What is the difference between select, withColumn and filter in PySpark?

Short answer

select is a projection: it chooses the columns or expressions in the result, so it can rename, compute and drop columns in one step. withColumn adds a column or replaces one with the same name, keeping all the others. filter, or its alias where, keeps the rows that match a condition. All three are transformations, and Spark combines a chain of them into one optimized plan, so the order you write them matters less than you might expect.

Why it matters in production

Readable chains are fine, but a common anti-pattern is calling withColumn in a loop to add dozens of columns. Each call adds a projection to the plan, and very long plans can slow analysis on the driver and even cause stack overflow errors. A single select (or withColumns, available since Spark 3.3) is cleaner.

What the interviewer is looking for

A strong answer usually demonstrates that you know what each does to columns versus rows, that they are lazy and optimized together, and the practical habits: select only the columns you need, and filter early so less data reaches joins and shuffles.

Watch out

Thinking that filtering after a join is always slower because of the written order. Catalyst often pushes simple filters below joins automatically. It cannot do that for filters on columns computed after the join, or filters hidden inside Python UDFs.

Example

from pyspark.sql import functions as F

result = (orders
    .filter(F.col("status") == "shipped")                       # rows
    .withColumn("net", F.col("total") - F.col("discount"))      # add a column
    .select("order_id", "customer_id", "net"))                  # keep only what is needed

Back to all questions ↑

Q07WorkingDataFrames & Transformations

Why should you avoid Python UDFs when built-in Spark functions can do the same work?

Short answer

Built-in functions run inside Spark’s engine, and the optimizer understands them. A regular Python UDF is a black box: each executor has to send rows to a separate Python worker process, run your function, and send the results back, with serialization costs both ways. The optimizer can’t look inside it, so it can’t push filters through it or optimize around it. Pandas UDFs reduce the cost by processing batches with Apache Arrow, but built-in expressions are still the first choice when they can do the job.

Why it matters in production

A single row-by-row UDF on a large table can be the slowest step of a job by a wide margin, and it hides logic from the plan, which blocks optimizations such as predicate pushdown.

What the interviewer is looking for

A strong answer usually demonstrates that you know why UDFs are slower (the Python/JVM boundary, serialization, optimizer opacity) and know the alternatives in order: built-in functions in pyspark.sql.functions or SQL expressions, higher-order functions for arrays and maps, then vectorized pandas UDFs, and only then a plain Python UDF.

Watch out

Saying UDFs should never be used. They are fine for logic that has no built-in equivalent, calls to a Python library, or small data. Measure them, prefer pandas UDFs for large volumes, and keep them deterministic. Arrow-based execution for regular Python UDFs is also available in newer Spark versions (3.5 and later), which reduces but does not remove the overhead.

Example

from pyspark.sql import functions as F

# Instead of a Python UDF that lowercases and trims emails:
df = df.withColumn("email", F.lower(F.trim("email")))

Back to all questions ↑

Q08WorkingDebugging & Production

What happens when you call collect() on a large DataFrame?

Short answer

collect() is an action that runs the whole plan and brings every resulting row back to the driver as Python objects. On a large DataFrame, that sends all the data over the network into one process, and the driver can run out of memory and fail, or become extremely slow. It also turns distributed data into single-machine data, throwing away the reason for using Spark. For inspection I’d use show, take or limit; for results, write to storage or aggregate first.

Why it matters in production

It is one of the most common causes of driver out-of-memory failures, and it often passes testing because development data is small. The same applies to toPandas() and to iterating with toLocalIterator() over huge results without care.

What the interviewer is looking for

A strong answer usually demonstrates that you know where the data ends up (the driver) and pick the right alternative for the goal:

  • Look at a sample: show(20), take(10), limit(100).
  • Get a number: count() or an aggregation that returns one row.
  • Keep the result: write it to a table or files, in parallel, from the executors.
  • Use a small lookup on the driver: collect only after aggregating or filtering to a known small size.

Watch out

Using collect() inside a loop, or collecting just to call len() or build a Python list for a later filter. Spark settings such as spark.driver.maxResultSize limit the damage but are a safety net, not a design.

Back to all questions ↑

Q09WorkingJoins & Shuffles

What is a shuffle in Spark, and why is it expensive?

Short answer

A shuffle redistributes data across partitions, usually by key, so that rows that belong together end up in the same partition, on possibly a different executor. Joins, groupBy aggregations, distinct, repartition and global orderBy commonly cause one. It’s expensive because every row is serialized, written to local disk, sent over the network and read back by the next stage. If partitions are too large, data also spills to disk. Each shuffle is a stage boundary in the job.

Why it matters in production

Shuffles usually dominate the runtime and cost of a Spark job. Reducing how much data is shuffled, or avoiding a shuffle entirely, is often the biggest single optimization.

What the interviewer is looking for

A strong answer usually demonstrates that you can identify shuffles in a plan (Exchange operators in explain(), and stage boundaries in the Spark UI) and know the levers: filter and select before the shuffle, broadcast small join sides, aggregate before joining when possible, and avoid unnecessary repartition and orderBy.

Watch out

Thinking shuffles can always be avoided. Joins and aggregations on large data need them; the goal is to shuffle less data, in sensibly sized partitions, without skew. Also, the default number of shuffle partitions (spark.sql.shuffle.partitions, 200 in Apache Spark) is not tuned for your data; Adaptive Query Execution, enabled by default since Spark 3.2, can coalesce small shuffle partitions at runtime.

Back to all questions ↑

Q10WorkingSpark Execution

What is the difference between narrow and wide transformations?

Short answer

In a narrow transformation, each output partition depends on a single input partition, so Spark can process partitions independently with no data movement. Examples are select, filter, withColumn and union. In a wide transformation, an output partition needs data from many input partitions, so Spark has to shuffle. Examples are groupBy, joins that aren’t broadcast, distinct and repartition. Spark chains narrow steps together within one stage, and every wide transformation starts a new stage.

Why it matters in production

Stages are how you read a job in the Spark UI. Knowing which operations create stage boundaries tells you where shuffles happen, where data might spill, and where a slow job is likely spending its time.

What the interviewer is looking for

A strong answer usually demonstrates the dependency definition (one input partition versus many), correct examples, and the link to stages, shuffles and fault recovery: losing a partition after a narrow chain is cheap to recompute, while wide dependencies may require recomputing data from many parents.

Watch out

“Narrow is good, wide is bad.” Wide transformations are how joins and aggregations work, and an early aggregation (wide) can make everything after it much cheaper. A join with a broadcast small table is also effectively narrow on the large side. Judge by how much data moves, not by the label.

Back to all questions ↑

Q11SeniorJoins & Shuffles

How do you investigate a Spark job with a large shuffle?

Short answer

First I find the stage with the big shuffle in the Spark UI, look at its shuffle read and write sizes, and map it back to the operation that caused it through the SQL tab or explain(). It’s usually a join or a groupBy. Then I ask whether that much data needs to move. Can I filter or drop columns earlier, aggregate before joining, or broadcast the smaller side? Next I check the task distribution: if a few tasks take far longer than the median, it’s skew. Finally I check partition sizes and spill.

Why it matters in production

Large shuffles drive runtime, cost and failures (executor out-of-memory errors, lost shuffle files, disk filling up). Tuning settings without finding the operation that causes the shuffle rarely helps.

What the interviewer is looking for

A strong answer usually demonstrates a method from evidence to fix:

  1. Locate the stage. In the Spark UI, sort stages by duration and compare shuffle read/write, input and spill.
  2. Map it to the code. The SQL/DataFrame tab or df.explain("formatted") shows which Exchange belongs to which join or aggregation.
  3. Measure the data. Rows and bytes going into the shuffle versus what the job actually needs. Unused columns and late filters are common causes.
  4. Check the distribution. The task summary (min, median, max duration and shuffle read) shows skew: a max far above the median.
  5. Check partition count and size. Too few partitions means huge tasks and spill; too many means scheduling overhead and tiny files.
  6. Look for spill. Memory and disk spill per stage show partitions too large for executor memory.
  7. Apply the matching fix. Filter and project before the shuffle, broadcast a small side, pre-aggregate, handle skewed keys, adjust partitioning, then rerun and compare the same metrics.

Adaptive Query Execution (on by default since Spark 3.2) already coalesces shuffle partitions and can switch to a broadcast join at runtime, so check what it did in the final plan.

Watch out

Increasing executor memory or cluster size first. It can hide the problem without reducing the data moved, and it does nothing for skew, where one task does most of the work regardless of cluster size.

Back to all questions ↑

Q12WorkingJoins & Shuffles

How do joins work in PySpark, and what join strategies might Spark use?

Short answer

A join matches rows from two DataFrames on keys, and Spark has to bring matching keys together. If one side is small enough, it uses a broadcast hash join: it copies the small side to every executor, so the large side doesn’t move. Otherwise, the usual choice for two large inputs is a sort-merge join: both sides are shuffled by key, sorted, and merged. Spark can also use a shuffle hash join, and nested-loop strategies for non-equality conditions. The optimizer decides from size estimates, hints and configuration.

Why it matters in production

The join strategy often decides whether a job takes minutes or hours. A broadcast join avoids shuffling the large table; a sort-merge join on two large inputs is robust but shuffles both; a non-equi join that falls back to a nested loop can be extremely slow.

What the interviewer is looking for

A strong answer usually demonstrates that you know the main strategies and how Spark chooses:

  • Size estimates from table statistics or file sizes, compared with spark.sql.autoBroadcastJoinThreshold (10 MB by default in Apache Spark 3.x and 4.x).
  • Join hints such as broadcast, merge, shuffle_hash and shuffle_replicate_nl, which override the default choice where supported.
  • Adaptive Query Execution, which can change a sort-merge join to a broadcast join at runtime when a side turns out to be small.
  • Join type and condition: equality conditions allow hash and merge joins; inequality conditions do not.

You confirm the choice with explain(): look for BroadcastHashJoin, SortMergeJoin or ShuffledHashJoin.

Watch out

Assuming Spark always picks the best strategy. Estimates can be wrong after filters, with missing statistics, or with UDFs. Also watch for duplicate keys: a join to a non-unique key multiplies rows, which is a correctness bug before it is a performance one.

Back to all questions ↑

Q13WorkingJoins & Shuffles

What is a broadcast join, and when is it useful?

Short answer

In a broadcast join, Spark sends a full copy of the smaller DataFrame to every executor, builds a hash table from it, and joins each partition of the large side locally. The large side never has to be shuffled. It’s ideal for joining a big fact table to small dimension or lookup tables. Spark does it automatically when its size estimate is below a configured threshold, and I can force it with a broadcast() hint. The catch is memory: the small side must fit comfortably on the driver and on every executor.

Why it matters in production

Replacing a sort-merge join with a broadcast join can remove the largest shuffle in a job. Broadcasting something that is not actually small, though, causes driver or executor out-of-memory errors and broadcast timeouts.

What the interviewer is looking for

A strong answer usually demonstrates when it helps (one side small, the other large), how Spark decides (size estimate versus spark.sql.autoBroadcastJoinThreshold, 10 MB by default in Apache Spark 3.x and 4.x; Adaptive Query Execution can also switch at runtime), and the risks: the broadcast is collected on the driver first, then copied to each executor.

Watch out

  • Hinting broadcast() on a table that grows over time. It works today and fails in six months.
  • Broadcasting the wrong side, or a side that is small only after a filter Spark cannot estimate well.
  • Outer joins: Spark can only broadcast the side that is not preserved (for example, the right side of a left outer join).

Example

from pyspark.sql.functions import broadcast

enriched = sales.join(broadcast(countries), "country_code")   # countries: a few hundred rows
enriched.explain()   # look for BroadcastHashJoin

Back to all questions ↑

Q14SeniorJoins & Shuffles

What is data skew, and how can it make a join slow?

Short answer

Skew is when a few key values hold a large share of the rows, such as one huge customer, a default value, or NULL. In a shuffle join, all rows for a key go to the same partition, so one or a few tasks do most of the work while the rest finish quickly and wait. The symptoms are a stage stuck at 199 of 200 tasks, a max task time far above the median, and spill or out-of-memory errors on those tasks. I’d confirm the hot keys first, then pick a fix that fits the cause.

Why it matters in production

A skewed join does not get faster with a bigger cluster: the slowest task decides the stage time. It is also a common cause of jobs that worked for months and suddenly time out when one key grows.

What the interviewer is looking for

A strong answer usually demonstrates diagnosis before cure: the Spark UI task summary (duration and shuffle read per task) and a quick groupBy(key).count() on the join keys. Then the mitigations, roughly in order of simplicity:

  • Filter keys that should not be joined at all: NULL keys, placeholder values, test data.
  • Broadcast the other side if it is small, which removes the shuffle and the skew with it.
  • Adaptive Query Execution skew join handling (enabled by default with AQE in Spark 3.x), which splits oversized partitions for sort-merge joins at runtime. Check the final plan to see whether it applied.
  • Handle hot keys separately: join the few large keys with a different strategy and union the results.
  • Salting: add a random suffix to hot keys on the large side and replicate those keys on the other side. Effective, but it adds complexity and data, so it comes last.

Watch out

Reaching for salting first, or increasing partition counts. More partitions do not help when all the rows share one key; they just add empty tasks.

Back to all questions ↑

Q15WorkingPartitioning

What is the difference between repartition() and coalesce()?

Short answer

repartition(n) can increase or decrease the number of partitions, and it does a full shuffle, so it produces evenly sized partitions. It can also partition by columns. coalesce(n) only reduces partitions, by merging existing ones without a full shuffle, so it’s cheaper but can leave uneven partition sizes. I’d use coalesce to cut down many small output partitions before a write, and repartition when I need more parallelism, even distribution, or data grouped by key.

Why it matters in production

Both are mostly used to control output file counts and task parallelism. Using the wrong one either adds a needless full shuffle or quietly cripples parallelism.

What the interviewer is looking for

A strong answer usually demonstrates the shuffle difference and the less obvious side effect: because coalesce avoids a shuffle, it can push the reduced partition count upstream into the same stage. coalesce(1) before a write can make the entire preceding computation run in a single task. If that happens, repartition (with its shuffle) can be faster overall because the expensive work still runs in parallel.

Watch out

  • Saying coalesce can increase partitions. On a DataFrame it cannot; asking for more simply keeps the current count.
  • Using repartition everywhere “to be safe”, which adds full shuffles.
  • Writing one file per partition with coalesce(1) on large data just to get a single output file.
  • Forgetting that Adaptive Query Execution may already coalesce shuffle partitions at runtime.

Back to all questions ↑

Q16WorkingPartitioning

How do you choose the number of Spark partitions?

Short answer

There’s no universal formula. I aim for partitions big enough that task overhead doesn’t dominate, small enough to fit in executor memory without spilling, and numerous enough to keep all cores busy, usually a small multiple of the total cores. So I start from the data size in each stage, especially shuffle size, and the cluster’s cores. Then I check the Spark UI: task durations, spill and skew. Adaptive Query Execution helps by coalescing small shuffle partitions at runtime, so I tune from evidence rather than magic numbers.

Why it matters in production

Too few partitions means long tasks, spill and idle cores; too many means scheduling overhead, tiny tasks and, on write, thousands of small files. The default spark.sql.shuffle.partitions (200 in Apache Spark) is only a starting point and is wrong for both very small and very large jobs.

What the interviewer is looking for

A strong answer usually demonstrates the trade-offs and the signals to tune from:

  • Input partitions come from file sizes and the maximum bytes per input split (spark.sql.files.maxPartitionBytes), so file layout matters.
  • Shuffle partitions come from spark.sql.shuffle.partitions, adjusted at runtime by AQE when enabled.
  • Output partitions decide file counts and sizes on write.
  • Signals: median task time, spill, a max task time far above the median (skew), idle executors.

Watch out

Fixed rules such as “always 200” or “always 128 MB per partition” applied without measuring, and adding partitions to fix skew: if one key dominates, more partitions do not split it.

Back to all questions ↑

Q17WorkingCaching & Persistence

When should you cache a DataFrame?

Short answer

I cache a DataFrame when it’s expensive to compute and reused by more than one action, for example a cleaned dataset that feeds several aggregations, or data reused across iterations of a loop or a model. Caching is lazy: it’s stored when the first action runs, and later actions read it instead of recomputing. But it uses executor memory, spilling to disk by default, and that memory competes with joins and shuffles. So I don’t cache DataFrames used once, and I call unpersist() when I’m done.

Why it matters in production

Missing a cache can mean reading and transforming the same source several times. Caching too much causes memory pressure, evictions and spill, which slows everything, and stale caches in long-running sessions can hide new data.

What the interviewer is looking for

A strong answer usually demonstrates cost versus benefit: reuse count, recompute cost and size. Also:

  • Check the Storage tab in the Spark UI to confirm what is cached and how much fits in memory.
  • persist() lets you choose a storage level; cache() uses the default.
  • For a long lineage reused many times, writing an intermediate table (or checkpointing) can be more robust than caching, because it survives failures and session restarts.

Watch out

  • Caching “for performance” everywhere, including DataFrames read once.
  • Caching before a filter, which stores far more data than needed.
  • Expecting df.cache() alone to do anything: nothing is stored until an action runs.

Back to all questions ↑

Q18SeniorPerformance & Scale

A PySpark notebook is suddenly three times slower, but the code did not change. What would you check?

Short answer

If the code didn’t change, something around it did, so I compare today’s run with a good one rather than guessing. First I split startup time from processing time. Then I check the data: has the volume grown, has a backfill widened the window, have small files accumulated, or has a key become skewed? In the Spark UI, I compare stages for new shuffles, spill or straggler tasks. Then the platform: cluster contention, capacity limits, runtime or library upgrades, and lost caches. I change one thing and measure again.

Why it matters in production

Slowdowns without code changes are common on shared platforms, and teams waste days rewriting code when the cause was data growth, file layout or a busy cluster.

What the interviewer is looking for

A strong answer usually demonstrates a structured comparison against a baseline:

  1. Startup versus processing. Slow session or cluster startup points at pools, environments, libraries or capacity, not the code.
  2. Data volume and shape. Input rows and bytes, the processing window, new partitions, skewed keys.
  3. File layout. File counts and sizes on the source tables; missing compaction makes reads and merges slower over time.
  4. Execution plan and stages. Compare explain() output and stage metrics with a good run: a join that stopped broadcasting because a table grew past the threshold, new shuffles, spill, stragglers.
  5. Platform. Concurrent jobs, queueing, throttling or capacity limits on managed platforms, executor loss.
  6. Runtime and dependencies. A Spark or runtime upgrade, a changed library version, new default settings.
  7. Caching and state. A cache that used to exist, or a source that is now read twice.
  8. Source systems. A slower database, API or storage account feeding the job.

Watch out

Comparing a cold run with a warm one, tuning Spark settings before checking data growth, and assuming “the code did not change” means the plan did not change. Growing data can flip join strategies without any code edit.

Back to all questions ↑

Q19SeniorPerformance & Scale

How would you optimize a PySpark job that processes hundreds of millions of rows?

Short answer

I start by measuring: which stages take the time, and how much data each one reads, shuffles and writes. Then I reduce the work. I process only new data incrementally where possible, filter early, and select only the columns I need. I fix the expensive steps: broadcast small join sides, handle skew, and right-size partitions and files. I replace Python UDFs with built-in functions, remove unnecessary actions, and cache only what’s genuinely reused. Then I check the physical plan and compare the same metrics before and after.

Why it matters in production

At this scale, small inefficiencies multiply: an extra shuffle, a full reload or a Python UDF can add hours and real cost. The biggest wins usually come from processing less data, not from tuning settings.

What the interviewer is looking for

A strong answer usually demonstrates that you optimize from evidence, biggest bottleneck first:

  • Process less: incremental loads instead of full history; partition pruning and data skipping on sources.
  • Read less: filter and project early, so less data reaches joins and shuffles; pushdown into Parquet or Delta.
  • Join well: broadcast small dimensions, avoid joins that multiply rows, pre-aggregate before joining where possible.
  • Distribute well: handle skewed keys, size shuffle partitions sensibly (AQE helps), avoid needless repartition and global orderBy.
  • Compute efficiently: built-in functions over Python UDFs; avoid repeated count() and collect() calls.
  • Write well: sensible file sizes, compaction, and MERGE conditions that let Spark prune target files.
  • Reuse wisely: cache only results used several times, and unpersist them.
  • Verify: explain("formatted"), stage metrics and spill, before and after each change.

Watch out

“Add more executors” as the answer. More hardware helps when work is evenly distributed and resources are the limit; it does nothing for skew, full reloads or row-multiplying joins, and it raises cost permanently.

Back to all questions ↑

Q20SeniorDebugging & Production

A PySpark pipeline works in development but fails or slows down in production. How do you troubleshoot it?

Short answer

I assume production is different in some concrete way and find out which. First I capture the failure: the run, its parameters, the error, and the failing stage in the Spark UI or execution log. Then I compare environments. Data volume and distribution are the usual culprits: skewed keys, nulls, schema drift or bad files that never existed in development. Then runtime and library versions, permissions, configuration and cluster contention. I reproduce on a production-like sample, fix one cause, rerun safely, and validate the output before declaring it done.

Why it matters in production

“It works on my machine” is the most common Spark incident. Development data is small, clean and evenly distributed; production is large, messy and skewed, and that changes join strategies, memory use and failure modes.

What the interviewer is looking for

A strong answer usually demonstrates a repeatable workflow:

  1. Capture evidence. The failing run, parameters as received, the full error and stack trace, the stage and task that failed, and the execution log entry with row counts and source window.
  2. Classify the failure. Code or data error (schema, nulls, bad records), resource problem (driver or executor out of memory, spill, disk full, lost executors), access problem (permissions, secrets, network paths), or platform problem (capacity, queueing).
  3. Compare data. Volume, key distribution and skew, new or renamed columns, malformed files, late or duplicate inputs.
  4. Compare environments. Spark and runtime versions, library versions, Spark configuration, cluster size, identity and permissions, paths and parameters.
  5. Read the plan. explain() and the Spark UI in production: a join that broadcasts in development may become a sort-merge join with skew in production, or a broadcast that no longer fits.
  6. Reproduce narrowly. A production-like sample, or the failing partition or date, rather than the whole pipeline.
  7. Fix and rerun safely. Idempotent writes, the same source window, no watermark advanced by the failed run.
  8. Validate and prevent. Reconcile row counts and key totals, then add a schema check, a data quality rule, an alert or a test with realistic data.

Watch out

Increasing memory until the error disappears, rerunning a partial load without checking idempotency, and testing only with small, clean samples. Pin library versions and test with production-like volumes and distributions.

Back to all questions ↑

Keep preparing

New questions are added as they are written. Interview questions for other topics live on the interview questions hub. The best preparation is still hands-on: start with Explore Data with PySpark, work through the notebook performance troubleshooting guide, and keep the PySpark performance cheat sheet close for quick reference.