Browse cheat sheets · PySpark
PySpark Performance Quick Reference
Troubleshoot and optimize PySpark workloads by separating scan, shuffle, skew, partitioning, caching, joins, driver pressure, and file-layout problems before changing cluster size.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Measure an action, not lazy transformation setup
- 2. Inspect the physical plan
- 3. Reduce columns early
- 4. Filter early
- 5. Inspect partition count
- 6. Inspect row distribution by partition
- 7. Check key skew
- 8. Do not fix skew by adding compute first
- 9. Check join cardinality before tuning join performance
- 10. Broadcast a genuinely small lookup
- 11. Project before broadcasting
- 12. Avoid collecting lookup data to the driver
- 13. Avoid large collect()
- 14. Avoid unnecessary toPandas()
- 15. Prefer built-in functions over Python UDFs
- 16. Use Python UDFs only when necessary
- 17. Cache only reused expensive intermediates
- 18. Do not cache one-time intermediates
- 19. Check whether data is persisted
- 20. Avoid repeated actions over the same expensive lineage
- 21. Aggregate multiple metrics in one pass
- 22. Repartition only when you need redistribution
- 23. Repartition by key for repeated keyed operations
- 24. Use coalesce() when reducing partitions
- 25. Avoid repartition(1)
- 26. Check shuffle partition setting
- 27. Inspect Adaptive Query Execution
- 28. Check broadcast threshold
- 29. Avoid unnecessary sorts
- 30. Avoid wide window partitions
- 31. Filter before window functions when semantics allow
- 32. Check file count / small-file patterns
- 33. Compact Delta small files when justified
- 34. Use partition pruning
- 35. Avoid wrapping partition columns unnecessarily
- 36. Avoid high-cardinality physical partitioning
- 37. Compare file format choices
- 38. Keep exact decimals for financial workloads
- 39. Avoid exploding arrays before filtering when possible
- 40. Validate row growth after explode()
- 41. Use semi/anti joins for existence checks
- 42. Avoid full joins when you only need existence
- 43. Check stage/task behavior, not only notebook duration
- 44. Interpret stragglers carefully
- 45. Track input size when comparing runs
- 46. Separate startup from execution
- 47. Do not use cluster size as the first fix
- 48. Tune one variable at a time
- 49. Preserve correctness checks during tuning
- 50. Use a compact troubleshooting flow
- What the result means
- What to look for
- Runtime increased after a join
- One task dominates a stage
- Driver memory grows
- Same lineage executes repeatedly
- Reads are slow even though transformations are simple
- Shuffle has thousands of tiny tasks
- Shuffle has a few huge tasks
- Production notes
- Performance tuning is workload-specific
- Spark is lazy
- AQE can already fix some runtime problems
- Caching competes for memory
- Shuffle is not inherently bad
- File layout is part of performance
- Scale changes the answer
- Common mistakes
- 1. Increasing cluster size before checking correctness
- 2. Timing lazy transformations
- 3. Caching everything
- 4. Broadcasting because one table has fewer rows
- 5. Repartitioning everywhere
- 6. Using repartition(1) for one output file
- 7. Ignoring skew
- 8. Using Python UDFs for built-in operations
- 9. Comparing runtimes without comparing input size
- 10. Changing many tuning variables at once
- Next step
- Related deep dives
Problem
A PySpark job is slow, inconsistent, memory-heavy, or expensive.
The cause may be:
- reading too much data
- carrying too many columns
- shuffle-heavy joins or aggregations
- skewed keys
- too many or too few partitions
- unnecessary repartitioning
- repeated recomputation
- overuse of
cache() - Python UDF overhead
- large
collect()/toPandas()operations - many small files
- poor Delta layout
- concurrent workloads competing for resources
- simply processing more data than previous runs
The goal is to identify the dominant cost before changing Spark configuration or adding compute.
Quick answer
Use this order:
1. Verify correctness first
2. Measure a real Spark action
3. Inspect the physical plan
4. Reduce rows and columns early
5. Check join strategy
6. Check shuffle volume
7. Check skew / straggling tasks
8. Check partition counts
9. Remove driver-side collection
10. Cache only reused expensive intermediates
11. Inspect file layout / small files
12. Compare against input volume and concurrency
13. Change one thing and measure again
Start with:
from time import perf_counter
start = perf_counter()
result = (
spark.table("sales")
.filter("sale_date >= '2026-10-01'")
.groupBy("merchant_id")
.sum("amount")
)
result.count()
elapsed = perf_counter() - start
print(f"Elapsed: {elapsed:.2f} seconds")
Then inspect the plan:
result.explain("formatted")
Do not tune Spark blindly.
Code / commands
1. Measure an action, not lazy transformation setup
This:
start = perf_counter()
result = (
df
.filter("amount > 100")
.groupBy("merchant_id")
.sum("amount")
)
print(perf_counter() - start)
mostly measures logical-plan construction.
Force execution:
start = perf_counter()
result = (
df
.filter("amount > 100")
.groupBy("merchant_id")
.sum("amount")
)
row_count = result.count()
print(
"Rows:",
row_count,
)
print(
"Elapsed:",
perf_counter() - start,
)
Use an action representative of the real workload.
2. Inspect the physical plan
result.explain("formatted")
Look for:
- table/file scans
- filters
Exchange- sorts
- broadcast joins
- sort-merge joins
- hash aggregations
- repeated scans
- window operators
Exchange usually means data redistribution.
That is not automatically bad, but large repeated exchanges deserve attention.
3. Reduce columns early
Prefer:
sales = (
spark.table("sales")
.select(
"sale_id",
"merchant_id",
"sale_date",
"amount",
)
)
over:
sales = spark.table("sales")
when the table is wide and most columns are unused.
Narrower DataFrames reduce:
- scan I/O
- serialization
- shuffle width
- memory pressure
4. Filter early
sales = (
spark.table("sales")
.filter(
"sale_date >= '2026-10-01'"
)
.select(
"merchant_id",
"sale_id",
"amount",
)
)
Push filters before joins and aggregations when the business logic permits it.
Do not filter away historical rows if downstream logic needs them.
5. Inspect partition count
print(
df.rdd.getNumPartitions()
)
Use this as context, not as a verdict.
A “good” partition count depends on:
- data size
- executor resources
- task size
- workload type
- shuffle stage
- skew
6. Inspect row distribution by partition
from pyspark.sql import functions as F
partition_rows = (
df
.withColumn(
"_partition_id",
F.spark_partition_id(),
)
.groupBy("_partition_id")
.count()
.orderBy(
F.desc("count")
)
)
partition_rows.show(
100,
truncate=False,
)
Use this when you suspect uneven partitions.
Row count is only one signal because rows can have different sizes.
7. Check key skew
(
df
.groupBy("merchant_id")
.count()
.orderBy(
F.desc("count")
)
.show(
50,
truncate=False,
)
)
Look for a small number of keys holding a disproportionate share of rows.
Common skew sources:
- one large tenant
- null/sentinel keys
"UNKNOWN"0- malformed keys
- dominant customers
8. Do not fix skew by adding compute first
If one task processes a huge hot key, more executors may not fix the bottleneck.
First determine whether you can:
- filter bad/sentinel keys
- pre-aggregate
- isolate the hot key
- use a different business key
- salt only the problematic distribution
Do not add salting until skew is proven.
9. Check join cardinality before tuning join performance
If a join unexpectedly multiplies rows:
(
right
.groupBy("merchant_id")
.count()
.filter(
F.col("count") > 1
)
.show(
50,
truncate=False,
)
)
A bad many-to-many join can create more work than any Spark setting can compensate for.
Correctness first.
10. Broadcast a genuinely small lookup
result = sales.join(
F.broadcast(merchants),
on="merchant_id",
how="left",
)
Then verify:
result.explain("formatted")
Look for:
BroadcastHashJoin
Broadcasting can avoid shuffling the large side.
Do not broadcast a large or very wide DataFrame.
11. Project before broadcasting
Prefer:
merchant_lookup = merchants.select(
"merchant_id",
"merchant_name",
)
Then:
result = sales.join(
F.broadcast(merchant_lookup),
"merchant_id",
"left",
)
Broadcast size is about bytes, not just row count.
12. Avoid collecting lookup data to the driver
Bad:
merchant_ids = [
row["merchant_id"]
for row in merchants.collect()
]
Then:
sales.filter(
F.col("merchant_id").isin(
merchant_ids
)
)
Better:
sales.join(
merchants.select("merchant_id"),
"merchant_id",
"left_semi",
)
Keep distributed data distributed.
13. Avoid large collect()
Risky:
rows = df.collect()
This moves all result rows to the driver.
For inspection:
df.show(
20,
truncate=False,
)
or:
rows = (
df.limit(100)
.collect()
)
Only collect when the result is deliberately small.
14. Avoid unnecessary toPandas()
Risky:
pdf = df.toPandas()
The entire result must fit in driver memory.
For inspection:
pdf = (
df.limit(1000)
.toPandas()
)
For large transformations, stay in Spark.
15. Prefer built-in functions over Python UDFs
Prefer:
df = df.withColumn(
"email_normalized",
F.lower(
F.trim(
F.col("email")
)
),
)
over a Python UDF for operations already available in Spark SQL functions.
Built-ins avoid Python serialization overhead and give Spark more optimization opportunities.
16. Use Python UDFs only when necessary
If custom logic cannot be expressed with built-ins, a Python UDF may be appropriate.
But first ask whether the logic can be expressed using:
- Spark SQL functions
- higher-order array/map functions
- SQL expressions
- pandas UDFs where vectorization is suitable
Do not use Python UDFs as the default transformation mechanism.
17. Cache only reused expensive intermediates
filtered = (
df
.filter(
F.col("sale_date") >= "2026-10-01"
)
.select(
"merchant_id",
"amount",
)
.cache()
)
Materialize:
filtered.count()
Reuse:
by_merchant = (
filtered
.groupBy("merchant_id")
.sum("amount")
)
totals = (
filtered
.agg(
F.sum("amount")
)
)
When done:
filtered.unpersist()
18. Do not cache one-time intermediates
Bad pattern:
df = df.cache()
df.count()
# use once
Caching introduces:
- materialization cost
- memory consumption
- possible spill
Use it only when reuse justifies the cost.
19. Check whether data is persisted
print(
filtered.storageLevel
)
Remember that cache() is lazy until an action runs.
20. Avoid repeated actions over the same expensive lineage
This may trigger repeated recomputation:
print(df.count())
print(
df
.filter("status = 'failed'")
.count()
)
print(
df
.agg(
F.sum("amount")
)
.collect()
)
If the same expensive prefix is reused several times, consider caching or restructuring into fewer scans.
21. Aggregate multiple metrics in one pass
Instead of separate jobs:
row_count = df.count()
null_count = (
df.filter(
F.col("merchant_id").isNull()
)
.count()
)
prefer:
metrics = (
df
.agg(
F.count("*").alias("row_count"),
F.sum(
F.when(
F.col("merchant_id").isNull(),
1,
).otherwise(0)
).alias("null_merchant_count"),
F.sum("amount").alias("amount_total"),
)
)
This can reduce repeated scans.
22. Repartition only when you need redistribution
df = df.repartition(
200,
"merchant_id",
)
This creates a shuffle.
Use it when the redistribution has a clear downstream benefit.
Do not repartition blindly before every join or write.
23. Repartition by key for repeated keyed operations
If several expensive operations use the same key:
by_merchant = df.repartition(
"merchant_id"
)
may help when the resulting partitioning can be reused.
Measure whether the initial shuffle is worth it.
24. Use coalesce() when reducing partitions
df_small = df.coalesce(20)
This is useful when data has already become much smaller and you want fewer output partitions.
Because it generally avoids a full reshuffle while reducing partitions, it can be cheaper than repartition().
But it can produce uneven partitions.
25. Avoid repartition(1)
Risky:
df.repartition(1)
before a large write.
This funnels work into one partition.
Use one-file output only as a deliberate export step for a small dataset.
26. Check shuffle partition setting
print(
spark.conf.get(
"spark.sql.shuffle.partitions"
)
)
This setting influences many shuffle stages.
Do not change it globally because a single job looks slow.
Measure task sizes and stage behavior first.
27. Inspect Adaptive Query Execution
print(
spark.conf.get(
"spark.sql.adaptive.enabled"
)
)
Adaptive Query Execution can change plans at runtime based on observed statistics.
This can include:
- coalescing shuffle partitions
- changing join strategies
- handling skewed partitions in supported cases
Inspect the executed behavior rather than assuming the original plan is final.
28. Check broadcast threshold
print(
spark.conf.get(
"spark.sql.autoBroadcastJoinThreshold"
)
)
Spark may automatically broadcast sufficiently small inputs.
Do not raise the threshold aggressively without checking executor memory and actual table size.
29. Avoid unnecessary sorts
Global sort:
df.orderBy(
F.col("event_time")
)
can be expensive because it requires distributed ordering.
Use it only when output order is actually required.
For top-N:
df.orderBy(
F.col("revenue").desc()
).limit(100)
may be appropriate, but use specialized patterns such as window/top-N where group semantics matter.
30. Avoid wide window partitions
Example:
Window.partitionBy("country")
can produce huge partitions when a country contains a large fraction of the dataset.
Use the business entity key when that is the real partitioning requirement.
31. Filter before window functions when semantics allow
recent = (
df
.filter(
F.col("event_date") >= "2026-10-01"
)
.select(
"customer_id",
"event_time",
"amount",
)
)
Then apply the window.
Do not filter away history needed for lifetime calculations.
32. Check file count / small-file patterns
Many tiny source files can increase:
- planning overhead
- file-open overhead
- task scheduling overhead
For Delta tables, inspect the table/file layout when scans are slower than expected.
Do not assume the DataFrame code is the only issue.
33. Compact Delta small files when justified
On supported Delta environments:
OPTIMIZE sales;
Use this when small-file fragmentation is measurably affecting reads.
Do not run OPTIMIZE after every write by default.
34. Use partition pruning
If the Delta table is physically partitioned by:
sale_year
sale_month
filter directly on those columns when appropriate:
df = (
spark.table("sales")
.filter(
(F.col("sale_year") == 2026)
& (F.col("sale_month") == 10)
)
)
Inspect:
df.explain("formatted")
Look for partition filters.
35. Avoid wrapping partition columns unnecessarily
If sale_year exists, prefer:
F.col("sale_year") == 2026
over:
F.year(
F.col("sale_date")
) == 2026
when the goal is partition pruning.
36. Avoid high-cardinality physical partitioning
Poor partition columns often include:
- transaction IDs
- exact timestamps
- customer IDs with millions of values
This can create many tiny directories/files.
Partitioning should support a real pruning or write-isolation pattern.
37. Compare file format choices
For repeated Spark analytics, prefer columnar formats such as:
Parquet
Delta
over CSV/JSON when you control the storage layer.
Benefits include:
- column pruning
- compression
- typed schema
- predicate pushdown support
Do not use CSV as the internal analytical format just because it is easy to inspect manually.
38. Keep exact decimals for financial workloads
Use:
from pyspark.sql.types import DecimalType
df = df.withColumn(
"amount",
F.col("amount").cast(
DecimalType(18, 2)
),
)
Avoid unnecessary floating-point conversion when exact reconciliation matters.
Correct data types can also reduce downstream ambiguity.
39. Avoid exploding arrays before filtering when possible
If an array contains many items, this:
df.select(
"id",
F.explode("items").alias("item"),
)
can multiply rows significantly.
If possible, reduce/filter earlier.
Always understand the post-explode() grain.
40. Validate row growth after explode()
before = df.count()
exploded = (
df.select(
"id",
F.explode("items").alias("item"),
)
)
after = exploded.count()
print(
"Expansion factor:",
after / before
if before
else None,
)
A valid row expansion can still be expensive.
41. Use semi/anti joins for existence checks
Instead of full enrichment:
valid = sales.join(
merchants.select("merchant_id"),
"merchant_id",
"left_semi",
)
For unmatched rows:
invalid = sales.join(
merchants.select("merchant_id"),
"merchant_id",
"left_anti",
)
These avoid carrying unnecessary right-side payload.
42. Avoid full joins when you only need existence
A left join followed by:
filter(right.key.isNotNull())
is often less clear than a semi join.
Use the join type that matches the business question.
43. Check stage/task behavior, not only notebook duration
A slow notebook may have:
one slow stage
one straggling task
large shuffle
executor retry
driver pause
Use Spark UI / platform monitoring to inspect:
- stage duration
- task duration
- shuffle read/write
- executor behavior
- failed tasks
- input/output
44. Interpret stragglers carefully
If most tasks finish in:
20–30 seconds
but one task takes:
15 minutes
suspect:
- skew
- oversized partition
- bad input file
- executor issue
- spill/resource pressure
Do not average that away.
45. Track input size when comparing runs
Performance comparisons are weak without workload context.
Record when useful:
input rows
input files
bytes
date range
partitions
output rows
concurrent workloads
A run can be slower simply because it processed much more data.
46. Separate startup from execution
For notebook platforms, total duration may include:
session startup
environment initialization
library setup
Spark execution
write/commit time
Do not optimize Spark code to fix platform/session startup.
47. Do not use cluster size as the first fix
Larger compute may help a healthy parallel workload.
It does not fix:
- accidental many-to-many joins
- hot keys
- driver collection
- repeated scans
- high-cardinality partitions
- bad file layout
Fix avoidable inefficiency first.
48. Tune one variable at a time
Bad tuning approach:
change shuffle partitions
enable cache
broadcast join
increase executors
repartition
change file layout
all at once.
Better:
measure baseline
→ change one thing
→ compare
→ keep or revert
Otherwise you do not know what helped.
49. Preserve correctness checks during tuning
For example, compare:
before_total = (
baseline
.agg(
F.sum("amount").alias("amount")
)
.first()["amount"]
)
with:
after_total = (
optimized
.agg(
F.sum("amount").alias("amount")
)
.first()["amount"]
)
A faster wrong result is not an optimization.
50. Use a compact troubleshooting flow
Slow job
↓
Scan too large?
→ filter/project earlier
Large shuffle?
→ inspect joins/groupBy/windows
One task dominates?
→ inspect skew
Driver pressure?
→ remove collect/toPandas/Python loops
Repeated stages?
→ check recomputation/cache
Many tiny files?
→ inspect write layout/OPTIMIZE
Everything slow at once?
→ inspect shared compute/capacity
What the result means
| Signal | Likely interpretation |
|---|---|
| Large scan | Too much input or insufficient pruning |
Many Exchange operators |
Significant shuffle |
| Broadcast hash join | Small side distributed to executors |
| Sort-merge join | Both sides redistributed/sorted |
| One long task | Skew or oversized partition |
| Many repeated stages | Recomputed lineage or repeated actions |
| Driver OOM | Too much data collected locally |
| Many tiny files | File-layout overhead |
| High shuffle read/write | Join/aggregation/window redistribution |
| Large cache footprint | Persisted data consuming executor memory |
| Slow notebook but fast Spark stages | Startup/orchestration overhead |
| Many workloads slow together | Shared resource/capacity pressure |
The core performance principle is:
do less work
before
asking Spark to do the same work faster
What to look for
Runtime increased after a join
Check:
- row multiplication
- broadcast choice
- shuffle volume
- key skew
- unnecessary columns
One task dominates a stage
Check:
- hot keys
- null/sentinel keys
- oversized partitions
- bad file distribution
Driver memory grows
Look for:
collect()toPandas()- Python lists
- local loops
- huge displayed results
Same lineage executes repeatedly
Check repeated actions and whether a reused expensive intermediate should be cached.
Reads are slow even though transformations are simple
Inspect:
- file count
- partition pruning
- storage latency
- schema width
- small-file fragmentation
Shuffle has thousands of tiny tasks
Shuffle partitioning may be too high for the data volume.
Measure task sizes before reducing it.
Shuffle has a few huge tasks
Partitioning may be too low or skew may be dominating.
Do not assume one fixed partition count solves both problems.
Production notes
Performance tuning is workload-specific
There is no universal best value for:
spark.sql.shuffle.partitions
executor count
partition count
broadcast threshold
cache strategy
Use representative production-scale data.
Spark is lazy
Transformation code can appear fast until an action is triggered.
Benchmark end-to-end actions.
AQE can already fix some runtime problems
Adaptive Query Execution can coalesce partitions and change some join behavior dynamically.
Do not fight the optimizer without evidence.
Caching competes for memory
Persisting large DataFrames can increase spill or reduce memory available for active stages.
Use cache only for meaningful reuse.
Shuffle is not inherently bad
Joins, aggregations, and windows often require it.
The goal is to reduce unnecessary shuffle and avoid pathological skew, not eliminate every Exchange.
File layout is part of performance
Spark code and storage layout are one system.
A well-written query over thousands of tiny files can still be slow.
Scale changes the answer
A pattern that works well at:
1 million rows
may behave differently at:
1 billion rows
because shuffle, skew, and file layout become dominant.
Common mistakes
1. Increasing cluster size before checking correctness
Bad joins and row explosion simply become more expensive.
2. Timing lazy transformations
Measure a real action.
3. Caching everything
Cache has cost.
4. Broadcasting because one table has fewer rows
Check size in bytes and width.
5. Repartitioning everywhere
Every repartition is a shuffle.
6. Using repartition(1) for one output file
This creates a single-partition bottleneck.
7. Ignoring skew
Average partition size can hide one massive hot key.
8. Using Python UDFs for built-in operations
This adds avoidable serialization and optimizer limitations.
9. Comparing runtimes without comparing input size
Larger input can explain slower execution.
10. Changing many tuning variables at once
You lose causal understanding.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.