Browse cheat sheets · PySpark

Technology: PySparkCategory: PerformanceDifficulty: IntermediateUpdated

PySpark Performance Quick Reference

Troubleshoot and optimize PySpark workloads by separating scan, shuffle, skew, partitioning, caching, joins, driver pressure, and file-layout problems before changing cluster size.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Measure an action, not lazy transformation setup
  5. 2. Inspect the physical plan
  6. 3. Reduce columns early
  7. 4. Filter early
  8. 5. Inspect partition count
  9. 6. Inspect row distribution by partition
  10. 7. Check key skew
  11. 8. Do not fix skew by adding compute first
  12. 9. Check join cardinality before tuning join performance
  13. 10. Broadcast a genuinely small lookup
  14. 11. Project before broadcasting
  15. 12. Avoid collecting lookup data to the driver
  16. 13. Avoid large collect()
  17. 14. Avoid unnecessary toPandas()
  18. 15. Prefer built-in functions over Python UDFs
  19. 16. Use Python UDFs only when necessary
  20. 17. Cache only reused expensive intermediates
  21. 18. Do not cache one-time intermediates
  22. 19. Check whether data is persisted
  23. 20. Avoid repeated actions over the same expensive lineage
  24. 21. Aggregate multiple metrics in one pass
  25. 22. Repartition only when you need redistribution
  26. 23. Repartition by key for repeated keyed operations
  27. 24. Use coalesce() when reducing partitions
  28. 25. Avoid repartition(1)
  29. 26. Check shuffle partition setting
  30. 27. Inspect Adaptive Query Execution
  31. 28. Check broadcast threshold
  32. 29. Avoid unnecessary sorts
  33. 30. Avoid wide window partitions
  34. 31. Filter before window functions when semantics allow
  35. 32. Check file count / small-file patterns
  36. 33. Compact Delta small files when justified
  37. 34. Use partition pruning
  38. 35. Avoid wrapping partition columns unnecessarily
  39. 36. Avoid high-cardinality physical partitioning
  40. 37. Compare file format choices
  41. 38. Keep exact decimals for financial workloads
  42. 39. Avoid exploding arrays before filtering when possible
  43. 40. Validate row growth after explode()
  44. 41. Use semi/anti joins for existence checks
  45. 42. Avoid full joins when you only need existence
  46. 43. Check stage/task behavior, not only notebook duration
  47. 44. Interpret stragglers carefully
  48. 45. Track input size when comparing runs
  49. 46. Separate startup from execution
  50. 47. Do not use cluster size as the first fix
  51. 48. Tune one variable at a time
  52. 49. Preserve correctness checks during tuning
  53. 50. Use a compact troubleshooting flow
  54. What the result means
  55. What to look for
  56. Runtime increased after a join
  57. One task dominates a stage
  58. Driver memory grows
  59. Same lineage executes repeatedly
  60. Reads are slow even though transformations are simple
  61. Shuffle has thousands of tiny tasks
  62. Shuffle has a few huge tasks
  63. Production notes
  64. Performance tuning is workload-specific
  65. Spark is lazy
  66. AQE can already fix some runtime problems
  67. Caching competes for memory
  68. Shuffle is not inherently bad
  69. File layout is part of performance
  70. Scale changes the answer
  71. Common mistakes
  72. 1. Increasing cluster size before checking correctness
  73. 2. Timing lazy transformations
  74. 3. Caching everything
  75. 4. Broadcasting because one table has fewer rows
  76. 5. Repartitioning everywhere
  77. 6. Using repartition(1) for one output file
  78. 7. Ignoring skew
  79. 8. Using Python UDFs for built-in operations
  80. 9. Comparing runtimes without comparing input size
  81. 10. Changing many tuning variables at once
  82. Next step
  83. Related deep dives

Problem

A PySpark job is slow, inconsistent, memory-heavy, or expensive.

The cause may be:

  • reading too much data
  • carrying too many columns
  • shuffle-heavy joins or aggregations
  • skewed keys
  • too many or too few partitions
  • unnecessary repartitioning
  • repeated recomputation
  • overuse of cache()
  • Python UDF overhead
  • large collect() / toPandas() operations
  • many small files
  • poor Delta layout
  • concurrent workloads competing for resources
  • simply processing more data than previous runs

The goal is to identify the dominant cost before changing Spark configuration or adding compute.

Quick answer

Use this order:

1. Verify correctness first
2. Measure a real Spark action
3. Inspect the physical plan
4. Reduce rows and columns early
5. Check join strategy
6. Check shuffle volume
7. Check skew / straggling tasks
8. Check partition counts
9. Remove driver-side collection
10. Cache only reused expensive intermediates
11. Inspect file layout / small files
12. Compare against input volume and concurrency
13. Change one thing and measure again

Start with:

from time import perf_counter

start = perf_counter()

result = (
    spark.table("sales")
    .filter("sale_date >= '2026-10-01'")
    .groupBy("merchant_id")
    .sum("amount")
)

result.count()

elapsed = perf_counter() - start

print(f"Elapsed: {elapsed:.2f} seconds")

Then inspect the plan:

result.explain("formatted")

Do not tune Spark blindly.

Code / commands

1. Measure an action, not lazy transformation setup

This:

start = perf_counter()

result = (
    df
    .filter("amount > 100")
    .groupBy("merchant_id")
    .sum("amount")
)

print(perf_counter() - start)

mostly measures logical-plan construction.

Force execution:

start = perf_counter()

result = (
    df
    .filter("amount > 100")
    .groupBy("merchant_id")
    .sum("amount")
)

row_count = result.count()

print(
    "Rows:",
    row_count,
)

print(
    "Elapsed:",
    perf_counter() - start,
)

Use an action representative of the real workload.


2. Inspect the physical plan

result.explain("formatted")

Look for:

  • table/file scans
  • filters
  • Exchange
  • sorts
  • broadcast joins
  • sort-merge joins
  • hash aggregations
  • repeated scans
  • window operators

Exchange usually means data redistribution.

That is not automatically bad, but large repeated exchanges deserve attention.


3. Reduce columns early

Prefer:

sales = (
    spark.table("sales")
    .select(
        "sale_id",
        "merchant_id",
        "sale_date",
        "amount",
    )
)

over:

sales = spark.table("sales")

when the table is wide and most columns are unused.

Narrower DataFrames reduce:

  • scan I/O
  • serialization
  • shuffle width
  • memory pressure

4. Filter early

sales = (
    spark.table("sales")
    .filter(
        "sale_date >= '2026-10-01'"
    )
    .select(
        "merchant_id",
        "sale_id",
        "amount",
    )
)

Push filters before joins and aggregations when the business logic permits it.

Do not filter away historical rows if downstream logic needs them.


5. Inspect partition count

print(
    df.rdd.getNumPartitions()
)

Use this as context, not as a verdict.

A “good” partition count depends on:

  • data size
  • executor resources
  • task size
  • workload type
  • shuffle stage
  • skew

6. Inspect row distribution by partition

from pyspark.sql import functions as F

partition_rows = (
    df
    .withColumn(
        "_partition_id",
        F.spark_partition_id(),
    )
    .groupBy("_partition_id")
    .count()
    .orderBy(
        F.desc("count")
    )
)

partition_rows.show(
    100,
    truncate=False,
)

Use this when you suspect uneven partitions.

Row count is only one signal because rows can have different sizes.


7. Check key skew

(
    df
    .groupBy("merchant_id")
    .count()
    .orderBy(
        F.desc("count")
    )
    .show(
        50,
        truncate=False,
    )
)

Look for a small number of keys holding a disproportionate share of rows.

Common skew sources:

  • one large tenant
  • null/sentinel keys
  • "UNKNOWN"
  • 0
  • malformed keys
  • dominant customers

8. Do not fix skew by adding compute first

If one task processes a huge hot key, more executors may not fix the bottleneck.

First determine whether you can:

  • filter bad/sentinel keys
  • pre-aggregate
  • isolate the hot key
  • use a different business key
  • salt only the problematic distribution

Do not add salting until skew is proven.


9. Check join cardinality before tuning join performance

If a join unexpectedly multiplies rows:

(
    right
    .groupBy("merchant_id")
    .count()
    .filter(
        F.col("count") > 1
    )
    .show(
        50,
        truncate=False,
    )
)

A bad many-to-many join can create more work than any Spark setting can compensate for.

Correctness first.


10. Broadcast a genuinely small lookup

result = sales.join(
    F.broadcast(merchants),
    on="merchant_id",
    how="left",
)

Then verify:

result.explain("formatted")

Look for:

BroadcastHashJoin

Broadcasting can avoid shuffling the large side.

Do not broadcast a large or very wide DataFrame.


11. Project before broadcasting

Prefer:

merchant_lookup = merchants.select(
    "merchant_id",
    "merchant_name",
)

Then:

result = sales.join(
    F.broadcast(merchant_lookup),
    "merchant_id",
    "left",
)

Broadcast size is about bytes, not just row count.


12. Avoid collecting lookup data to the driver

Bad:

merchant_ids = [
    row["merchant_id"]
    for row in merchants.collect()
]

Then:

sales.filter(
    F.col("merchant_id").isin(
        merchant_ids
    )
)

Better:

sales.join(
    merchants.select("merchant_id"),
    "merchant_id",
    "left_semi",
)

Keep distributed data distributed.


13. Avoid large collect()

Risky:

rows = df.collect()

This moves all result rows to the driver.

For inspection:

df.show(
    20,
    truncate=False,
)

or:

rows = (
    df.limit(100)
    .collect()
)

Only collect when the result is deliberately small.


14. Avoid unnecessary toPandas()

Risky:

pdf = df.toPandas()

The entire result must fit in driver memory.

For inspection:

pdf = (
    df.limit(1000)
    .toPandas()
)

For large transformations, stay in Spark.


15. Prefer built-in functions over Python UDFs

Prefer:

df = df.withColumn(
    "email_normalized",
    F.lower(
        F.trim(
            F.col("email")
        )
    ),
)

over a Python UDF for operations already available in Spark SQL functions.

Built-ins avoid Python serialization overhead and give Spark more optimization opportunities.


16. Use Python UDFs only when necessary

If custom logic cannot be expressed with built-ins, a Python UDF may be appropriate.

But first ask whether the logic can be expressed using:

  • Spark SQL functions
  • higher-order array/map functions
  • SQL expressions
  • pandas UDFs where vectorization is suitable

Do not use Python UDFs as the default transformation mechanism.


17. Cache only reused expensive intermediates

filtered = (
    df
    .filter(
        F.col("sale_date") >= "2026-10-01"
    )
    .select(
        "merchant_id",
        "amount",
    )
    .cache()
)

Materialize:

filtered.count()

Reuse:

by_merchant = (
    filtered
    .groupBy("merchant_id")
    .sum("amount")
)

totals = (
    filtered
    .agg(
        F.sum("amount")
    )
)

When done:

filtered.unpersist()

18. Do not cache one-time intermediates

Bad pattern:

df = df.cache()
df.count()
# use once

Caching introduces:

  • materialization cost
  • memory consumption
  • possible spill

Use it only when reuse justifies the cost.


19. Check whether data is persisted

print(
    filtered.storageLevel
)

Remember that cache() is lazy until an action runs.


20. Avoid repeated actions over the same expensive lineage

This may trigger repeated recomputation:

print(df.count())

print(
    df
    .filter("status = 'failed'")
    .count()
)

print(
    df
    .agg(
        F.sum("amount")
    )
    .collect()
)

If the same expensive prefix is reused several times, consider caching or restructuring into fewer scans.


21. Aggregate multiple metrics in one pass

Instead of separate jobs:

row_count = df.count()

null_count = (
    df.filter(
        F.col("merchant_id").isNull()
    )
    .count()
)

prefer:

metrics = (
    df
    .agg(
        F.count("*").alias("row_count"),
        F.sum(
            F.when(
                F.col("merchant_id").isNull(),
                1,
            ).otherwise(0)
        ).alias("null_merchant_count"),
        F.sum("amount").alias("amount_total"),
    )
)

This can reduce repeated scans.


22. Repartition only when you need redistribution

df = df.repartition(
    200,
    "merchant_id",
)

This creates a shuffle.

Use it when the redistribution has a clear downstream benefit.

Do not repartition blindly before every join or write.


23. Repartition by key for repeated keyed operations

If several expensive operations use the same key:

by_merchant = df.repartition(
    "merchant_id"
)

may help when the resulting partitioning can be reused.

Measure whether the initial shuffle is worth it.


24. Use coalesce() when reducing partitions

df_small = df.coalesce(20)

This is useful when data has already become much smaller and you want fewer output partitions.

Because it generally avoids a full reshuffle while reducing partitions, it can be cheaper than repartition().

But it can produce uneven partitions.


25. Avoid repartition(1)

Risky:

df.repartition(1)

before a large write.

This funnels work into one partition.

Use one-file output only as a deliberate export step for a small dataset.


26. Check shuffle partition setting

print(
    spark.conf.get(
        "spark.sql.shuffle.partitions"
    )
)

This setting influences many shuffle stages.

Do not change it globally because a single job looks slow.

Measure task sizes and stage behavior first.


27. Inspect Adaptive Query Execution

print(
    spark.conf.get(
        "spark.sql.adaptive.enabled"
    )
)

Adaptive Query Execution can change plans at runtime based on observed statistics.

This can include:

  • coalescing shuffle partitions
  • changing join strategies
  • handling skewed partitions in supported cases

Inspect the executed behavior rather than assuming the original plan is final.


28. Check broadcast threshold

print(
    spark.conf.get(
        "spark.sql.autoBroadcastJoinThreshold"
    )
)

Spark may automatically broadcast sufficiently small inputs.

Do not raise the threshold aggressively without checking executor memory and actual table size.


29. Avoid unnecessary sorts

Global sort:

df.orderBy(
    F.col("event_time")
)

can be expensive because it requires distributed ordering.

Use it only when output order is actually required.

For top-N:

df.orderBy(
    F.col("revenue").desc()
).limit(100)

may be appropriate, but use specialized patterns such as window/top-N where group semantics matter.


30. Avoid wide window partitions

Example:

Window.partitionBy("country")

can produce huge partitions when a country contains a large fraction of the dataset.

Use the business entity key when that is the real partitioning requirement.


31. Filter before window functions when semantics allow

recent = (
    df
    .filter(
        F.col("event_date") >= "2026-10-01"
    )
    .select(
        "customer_id",
        "event_time",
        "amount",
    )
)

Then apply the window.

Do not filter away history needed for lifetime calculations.


32. Check file count / small-file patterns

Many tiny source files can increase:

  • planning overhead
  • file-open overhead
  • task scheduling overhead

For Delta tables, inspect the table/file layout when scans are slower than expected.

Do not assume the DataFrame code is the only issue.


33. Compact Delta small files when justified

On supported Delta environments:

OPTIMIZE sales;

Use this when small-file fragmentation is measurably affecting reads.

Do not run OPTIMIZE after every write by default.


34. Use partition pruning

If the Delta table is physically partitioned by:

sale_year
sale_month

filter directly on those columns when appropriate:

df = (
    spark.table("sales")
    .filter(
        (F.col("sale_year") == 2026)
        & (F.col("sale_month") == 10)
    )
)

Inspect:

df.explain("formatted")

Look for partition filters.


35. Avoid wrapping partition columns unnecessarily

If sale_year exists, prefer:

F.col("sale_year") == 2026

over:

F.year(
    F.col("sale_date")
) == 2026

when the goal is partition pruning.


36. Avoid high-cardinality physical partitioning

Poor partition columns often include:

  • transaction IDs
  • exact timestamps
  • customer IDs with millions of values

This can create many tiny directories/files.

Partitioning should support a real pruning or write-isolation pattern.


37. Compare file format choices

For repeated Spark analytics, prefer columnar formats such as:

Parquet
Delta

over CSV/JSON when you control the storage layer.

Benefits include:

  • column pruning
  • compression
  • typed schema
  • predicate pushdown support

Do not use CSV as the internal analytical format just because it is easy to inspect manually.


38. Keep exact decimals for financial workloads

Use:

from pyspark.sql.types import DecimalType

df = df.withColumn(
    "amount",
    F.col("amount").cast(
        DecimalType(18, 2)
    ),
)

Avoid unnecessary floating-point conversion when exact reconciliation matters.

Correct data types can also reduce downstream ambiguity.


39. Avoid exploding arrays before filtering when possible

If an array contains many items, this:

df.select(
    "id",
    F.explode("items").alias("item"),
)

can multiply rows significantly.

If possible, reduce/filter earlier.

Always understand the post-explode() grain.


40. Validate row growth after explode()

before = df.count()

exploded = (
    df.select(
        "id",
        F.explode("items").alias("item"),
    )
)

after = exploded.count()

print(
    "Expansion factor:",
    after / before
    if before
    else None,
)

A valid row expansion can still be expensive.


41. Use semi/anti joins for existence checks

Instead of full enrichment:

valid = sales.join(
    merchants.select("merchant_id"),
    "merchant_id",
    "left_semi",
)

For unmatched rows:

invalid = sales.join(
    merchants.select("merchant_id"),
    "merchant_id",
    "left_anti",
)

These avoid carrying unnecessary right-side payload.


42. Avoid full joins when you only need existence

A left join followed by:

filter(right.key.isNotNull())

is often less clear than a semi join.

Use the join type that matches the business question.


43. Check stage/task behavior, not only notebook duration

A slow notebook may have:

one slow stage
one straggling task
large shuffle
executor retry
driver pause

Use Spark UI / platform monitoring to inspect:

  • stage duration
  • task duration
  • shuffle read/write
  • executor behavior
  • failed tasks
  • input/output

44. Interpret stragglers carefully

If most tasks finish in:

20–30 seconds

but one task takes:

15 minutes

suspect:

  • skew
  • oversized partition
  • bad input file
  • executor issue
  • spill/resource pressure

Do not average that away.


45. Track input size when comparing runs

Performance comparisons are weak without workload context.

Record when useful:

input rows
input files
bytes
date range
partitions
output rows
concurrent workloads

A run can be slower simply because it processed much more data.


46. Separate startup from execution

For notebook platforms, total duration may include:

session startup
environment initialization
library setup
Spark execution
write/commit time

Do not optimize Spark code to fix platform/session startup.


47. Do not use cluster size as the first fix

Larger compute may help a healthy parallel workload.

It does not fix:

  • accidental many-to-many joins
  • hot keys
  • driver collection
  • repeated scans
  • high-cardinality partitions
  • bad file layout

Fix avoidable inefficiency first.


48. Tune one variable at a time

Bad tuning approach:

change shuffle partitions
enable cache
broadcast join
increase executors
repartition
change file layout

all at once.

Better:

measure baseline
→ change one thing
→ compare
→ keep or revert

Otherwise you do not know what helped.


49. Preserve correctness checks during tuning

For example, compare:

before_total = (
    baseline
    .agg(
        F.sum("amount").alias("amount")
    )
    .first()["amount"]
)

with:

after_total = (
    optimized
    .agg(
        F.sum("amount").alias("amount")
    )
    .first()["amount"]
)

A faster wrong result is not an optimization.


50. Use a compact troubleshooting flow

Slow job
  ↓
Scan too large?
  → filter/project earlier

Large shuffle?
  → inspect joins/groupBy/windows

One task dominates?
  → inspect skew

Driver pressure?
  → remove collect/toPandas/Python loops

Repeated stages?
  → check recomputation/cache

Many tiny files?
  → inspect write layout/OPTIMIZE

Everything slow at once?
  → inspect shared compute/capacity

What the result means

Signal Likely interpretation
Large scan Too much input or insufficient pruning
Many Exchange operators Significant shuffle
Broadcast hash join Small side distributed to executors
Sort-merge join Both sides redistributed/sorted
One long task Skew or oversized partition
Many repeated stages Recomputed lineage or repeated actions
Driver OOM Too much data collected locally
Many tiny files File-layout overhead
High shuffle read/write Join/aggregation/window redistribution
Large cache footprint Persisted data consuming executor memory
Slow notebook but fast Spark stages Startup/orchestration overhead
Many workloads slow together Shared resource/capacity pressure

The core performance principle is:

do less work
before
asking Spark to do the same work faster

What to look for

Runtime increased after a join

Check:

  • row multiplication
  • broadcast choice
  • shuffle volume
  • key skew
  • unnecessary columns

One task dominates a stage

Check:

  • hot keys
  • null/sentinel keys
  • oversized partitions
  • bad file distribution

Driver memory grows

Look for:

  • collect()
  • toPandas()
  • Python lists
  • local loops
  • huge displayed results

Same lineage executes repeatedly

Check repeated actions and whether a reused expensive intermediate should be cached.

Reads are slow even though transformations are simple

Inspect:

  • file count
  • partition pruning
  • storage latency
  • schema width
  • small-file fragmentation

Shuffle has thousands of tiny tasks

Shuffle partitioning may be too high for the data volume.

Measure task sizes before reducing it.

Shuffle has a few huge tasks

Partitioning may be too low or skew may be dominating.

Do not assume one fixed partition count solves both problems.

Production notes

Performance tuning is workload-specific

There is no universal best value for:

spark.sql.shuffle.partitions
executor count
partition count
broadcast threshold
cache strategy

Use representative production-scale data.

Spark is lazy

Transformation code can appear fast until an action is triggered.

Benchmark end-to-end actions.

AQE can already fix some runtime problems

Adaptive Query Execution can coalesce partitions and change some join behavior dynamically.

Do not fight the optimizer without evidence.

Caching competes for memory

Persisting large DataFrames can increase spill or reduce memory available for active stages.

Use cache only for meaningful reuse.

Shuffle is not inherently bad

Joins, aggregations, and windows often require it.

The goal is to reduce unnecessary shuffle and avoid pathological skew, not eliminate every Exchange.

File layout is part of performance

Spark code and storage layout are one system.

A well-written query over thousands of tiny files can still be slow.

Scale changes the answer

A pattern that works well at:

1 million rows

may behave differently at:

1 billion rows

because shuffle, skew, and file layout become dominant.

Common mistakes

1. Increasing cluster size before checking correctness

Bad joins and row explosion simply become more expensive.

2. Timing lazy transformations

Measure a real action.

3. Caching everything

Cache has cost.

4. Broadcasting because one table has fewer rows

Check size in bytes and width.

5. Repartitioning everywhere

Every repartition is a shuffle.

6. Using repartition(1) for one output file

This creates a single-partition bottleneck.

7. Ignoring skew

Average partition size can hide one massive hot key.

8. Using Python UDFs for built-in operations

This adds avoidable serialization and optimizer limitations.

9. Comparing runtimes without comparing input size

Larger input can explain slower execution.

10. Changing many tuning variables at once

You lose causal understanding.

Next step

Use the dominant signal to choose the next action.

  • Large scan → filter/project earlier and inspect partition pruning.
  • Large shuffle → inspect joins, aggregations, windows, and unnecessary repartitioning.
  • One straggling task → investigate skew.
  • Driver pressure → remove large collect() / toPandas() / Python-side loops.
  • Repeated work → consider caching only the reused expensive prefix.
  • Join slow → validate cardinality, then inspect broadcast vs shuffle strategy.
  • Many small files → review Delta write layout and maintenance.
  • Everything slowed together → inspect shared compute/capacity and concurrency.
  • Code is correct but still slow → inspect Spark UI, stage metrics, and runtime configuration before adding compute.

← All cheat sheets