Browse cheat sheets · Microsoft Fabric
Fabric Notebook Performance
Troubleshoot and improve Microsoft Fabric notebook performance by separating Spark session startup, data scan, shuffle, skew, driver pressure, concurrency, and write-path bottlenecks.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Measure the Spark action, not only DataFrame construction
- 2. Inspect the physical plan
- 3. Check the current number of partitions
- 4. Inspect partition row distribution
- 5. Filter before expensive transformations
- 6. Avoid select("*") when only a few columns are needed
- 7. Inspect join strategy
- 8. Broadcast a genuinely small lookup table
- 9. Check for duplicate join keys before blaming Spark
- 10. Reduce data before a large join
- 11. Check for skewed join/group keys
- 12. Avoid collect() for large results
- 13. Avoid unnecessary toPandas()
- 14. Avoid Python loops over Spark rows
- 15. Prefer built-in Spark expressions over Python UDFs
- 16. Cache only reused DataFrames
- 17. Check cache state
- 18. Avoid unnecessary repeated actions
- 19. Inspect Spark configuration relevant to the session
- 20. Inspect important adaptive execution settings
- 21. Repartition when you need a real redistribution
- 22. Coalesce when reducing partitions without a full reshuffle
- 23. Avoid repartition(1) for large production outputs
- 24. Check whether small files are part of the problem
- 25. Write only the columns you need
- 26. Be deliberate with overwrite
- 27. Stop comparing notebook runs that use different input volumes
- 28. Separate startup latency from Spark execution
- 29. Use Fabric starter pools for faster session startup where appropriate
- 30. Use High Concurrency when compatible notebooks can share a session
- 31. Use a session tag for pipeline notebook reuse when High Concurrency is enabled
- 32. Do not create excessive parallel notebook activity
- 33. Use autoscale and dynamic executor allocation for variable workloads when appropriate
- 34. Consider Native Execution Engine only where supported and validated
- 35. Use Spark monitoring to find the stage that dominates runtime
- What the result means
- What to look for
- Startup dominates total runtime
- A stage has one or a few straggling tasks
- Shuffle dominates runtime
- Driver fails or becomes slow
- Runtime changes dramatically for similar code
- Output creates many small files
- Production notes
- Fabric notebook performance has two different layers
- High Concurrency has compatibility rules
- Current High Concurrency density is configurable
- Spark DMV-style snapshots do not exist in the same way as SQL Server
- Capacity and pool sizing matter after code-level issues are understood
- Queueing can be a capacity problem rather than a notebook-code problem
- Autotune is version- and feature-dependent
- Measure with representative production-scale data
- Common mistakes
- 1. Increasing Spark compute before finding the bottleneck
- 2. Timing only DataFrame transformations
- 3. Using repartition(1) to get one output file
- 4. Caching every DataFrame
- 5. Broadcasting a table without checking its size
- 6. Using Python UDFs for operations Spark already supports
- 7. Ignoring data skew
- 8. Increasing notebook concurrency blindly
- 9. Treating session startup as query runtime
- 10. Optimizing code while ignoring file layout
- Next step
- Related deep dives
Problem
A Microsoft Fabric notebook is slow, inconsistent, or consumes more Spark capacity than expected.
The delay may come from very different places:
- Spark session startup
- reading too much data
- too many small files
- poor partitioning
- shuffle-heavy joins or aggregations
- data skew
- Python code running on the driver
- unnecessary
collect()/toPandas() - repeated recomputation
- excessive notebook concurrency
- undersized or oversized Spark compute
- slow Delta writes
- several notebooks starting separate Spark sessions unnecessarily
The goal is to identify where the time is going before changing Spark settings.
Quick answer
Use this flow:
1. Separate session startup time from notebook execution time
2. Identify the slow Spark action/stage
3. Inspect the physical plan
4. Check how much data is being scanned
5. Check partitions and shuffle
6. Check for skew
7. Check for driver-side collection / Python loops
8. Check repeated recomputation
9. Check file layout and Delta write pattern
10. Check pool sizing, autoscale, concurrency, and session reuse
11. Change one thing and measure again
Start with the operation you believe is slow:
from time import perf_counter
start = perf_counter()
result = (
spark.read.format("delta")
.load("Tables/sales")
.filter("sale_date >= '2026-10-01'")
.groupBy("merchant_id")
.sum("amount")
)
result.count() # force execution
elapsed = perf_counter() - start
print(f"Elapsed: {elapsed:.2f} seconds")
Then inspect the plan:
result.explain("formatted")
Do not tune Spark configuration until you know whether the bottleneck is scan, shuffle, skew, compute, startup, or the driver.
Code / commands
1. Measure the Spark action, not only DataFrame construction
Spark DataFrames are lazy.
This:
start = perf_counter()
result = (
df.filter("amount > 100")
.groupBy("merchant_id")
.sum("amount")
)
print(perf_counter() - start)
mostly measures logical-plan construction.
Force an action:
from time import perf_counter
start = perf_counter()
result = (
df.filter("amount > 100")
.groupBy("merchant_id")
.sum("amount")
)
row_count = result.count()
elapsed = perf_counter() - start
print("Rows:", row_count)
print(f"Elapsed: {elapsed:.2f} seconds")
Use a representative action.
Do not add a large collect() merely to benchmark.
2. Inspect the physical plan
result.explain("formatted")
Use this before changing partition counts or compute size.
Look for:
- scans
- filters
- joins
- exchanges
- sorts
- aggregations
- broadcast joins
- repeated scans
An Exchange usually means Spark must redistribute data across partitions.
That is not automatically bad, but large or repeated exchanges deserve attention.
3. Check the current number of partitions
print(df.rdd.getNumPartitions())
After a shuffle:
result = (
df.groupBy("merchant_id")
.sum("amount")
)
print(result.rdd.getNumPartitions())
Partition count alone does not tell you whether the workload is healthy.
You need to compare it with:
- data volume
- partition-size distribution
- executor count
- task duration
- skew
4. Inspect partition row distribution
For targeted troubleshooting:
from pyspark.sql import functions as F
partition_rows = (
df.withColumn(
"_partition_id",
F.spark_partition_id(),
)
.groupBy("_partition_id")
.count()
.orderBy(F.desc("count"))
)
partition_rows.show(50, truncate=False)
Use this when you suspect skew.
If a small number of partitions contain far more rows than the rest, those tasks can dominate stage duration.
Row count is only one signal: variable-width rows can have different byte sizes.
5. Filter before expensive transformations
Prefer:
filtered = (
spark.read.format("delta")
.load("Tables/sales")
.filter(F.col("sale_date") >= "2026-10-01")
.select(
"merchant_id",
"sale_date",
"amount",
)
)
over loading wide data and carrying every column through joins and aggregations.
Push down:
- filters
- column selection
as early as practical.
This reduces the amount of data processed downstream.
6. Avoid select("*") when only a few columns are needed
Instead of:
df = (
spark.read.format("delta")
.load("Tables/sales")
.select("*")
)
prefer:
df = (
spark.read.format("delta")
.load("Tables/sales")
.select(
"sale_id",
"merchant_id",
"sale_date",
"amount",
)
)
Column pruning matters most on wide analytical tables.
7. Inspect join strategy
joined = sales.join(
merchants,
on="merchant_id",
how="left",
)
joined.explain("formatted")
Look for whether Spark chooses:
- broadcast hash join
- sort-merge join
- another strategy
Do not force a broadcast until you know the smaller side is appropriate for broadcasting.
8. Broadcast a genuinely small lookup table
When you have verified that the lookup is small enough:
from pyspark.sql import functions as F
joined = sales.join(
F.broadcast(merchants),
on="merchant_id",
how="left",
)
Then inspect again:
joined.explain("formatted")
Broadcasting avoids shuffling the large side for some join patterns.
Do not broadcast a large table merely because a join is slow.
9. Check for duplicate join keys before blaming Spark
A join may be slow because it multiplies rows.
Check key cardinality:
duplicate_keys = (
merchants.groupBy("merchant_id")
.count()
.filter(F.col("count") > 1)
)
duplicate_keys.show(50, truncate=False)
If a supposed dimension table is not unique, a join can:
- multiply rows
- increase shuffle volume
- increase memory pressure
- inflate downstream metrics
Fix the data relationship before tuning the cluster.
10. Reduce data before a large join
Prefer:
sales_small = (
sales
.filter(F.col("sale_date") >= "2026-10-01")
.select(
"merchant_id",
"amount",
)
)
merchants_small = merchants.select(
"merchant_id",
"region",
)
joined = sales_small.join(
merchants_small,
"merchant_id",
"left",
)
Filtering and projecting first can materially reduce shuffle work.
11. Check for skewed join/group keys
Start with:
(
df.groupBy("merchant_id")
.count()
.orderBy(F.desc("count"))
.show(50, truncate=False)
)
A very hot key can send a disproportionate amount of data to one partition.
Common examples:
- one default/unknown key
- null replacement such as
0 - one very large customer
- one dominant merchant
- malformed data collapsing many records onto one key
Do not immediately add random salting.
First identify the skewed key and confirm the business reason.
12. Avoid collect() for large results
Risky:
rows = df.collect()
This sends the full result to the driver.
Safer for inspection:
df.show(20, truncate=False)
or:
rows = df.limit(100).collect()
Use full collection only when the result is deliberately small.
13. Avoid unnecessary toPandas()
Risky:
pdf = df.toPandas()
The resulting pandas DataFrame must fit in driver memory.
For inspection:
pdf = (
df.limit(1000)
.toPandas()
)
For large transformations, keep the work distributed in Spark.
14. Avoid Python loops over Spark rows
Bad pattern:
for row in df.collect():
process(row)
Prefer Spark transformations:
result = (
df.withColumn(
"net_amount",
F.col("amount") - F.col("discount"),
)
)
Moving data to Python row-by-row removes Spark’s distributed execution advantage.
15. Prefer built-in Spark expressions over Python UDFs
Prefer:
df = df.withColumn(
"normalized_email",
F.lower(
F.trim(F.col("email"))
),
)
over a Python UDF for operations already available in Spark SQL functions.
Built-in expressions give Spark more opportunities to optimize execution and avoid Python serialization overhead.
Use a UDF only when the required logic cannot be expressed with built-in Spark functions or another more efficient supported mechanism.
16. Cache only reused DataFrames
If the same expensive result is reused multiple times:
filtered = (
df.filter(F.col("sale_date") >= "2026-10-01")
.select(
"merchant_id",
"amount",
)
.cache()
)
Materialize it:
filtered.count()
Reuse it:
by_merchant = (
filtered.groupBy("merchant_id")
.sum("amount")
)
total = filtered.agg(
F.sum("amount")
)
When finished:
filtered.unpersist()
Do not cache every DataFrame.
Caching data that is used once usually adds work rather than removing it.
17. Check cache state
print(filtered.storageLevel)
Use this when debugging whether a DataFrame is actually persisted.
Remember that cache() is lazy until an action materializes it.
18. Avoid unnecessary repeated actions
This triggers multiple jobs:
print(df.count())
print(df.filter(F.col("status") == "failed").count())
print(df.agg(F.sum("amount")).collect())
That may be acceptable, but each action can recompute upstream lineage if the input is not cached or otherwise materialized.
If several actions reuse an expensive common prefix, consider materializing that prefix once.
19. Inspect Spark configuration relevant to the session
for key, value in sorted(
spark.sparkContext.getConf().getAll()
):
print(f"{key}={value}")
Use this when:
- performance differs between environments
- a notebook behaves differently from a pipeline
- you suspect session configuration differences
Do not change configuration simply because a value looks unfamiliar.
20. Inspect important adaptive execution settings
keys = [
"spark.sql.adaptive.enabled",
"spark.sql.shuffle.partitions",
"spark.sql.autoBroadcastJoinThreshold",
]
for key in keys:
try:
print(
key,
spark.conf.get(key),
)
except Exception:
print(
key,
"<not available>",
)
Adaptive Query Execution can change the physical execution strategy based on runtime information.
Treat manually tuning spark.sql.shuffle.partitions as workload-specific, not as a universal fix.
21. Repartition when you need a real redistribution
repartitioned = df.repartition(
200,
"merchant_id",
)
Use repartition() when you intentionally need to redistribute data, for example before a repeated operation using the same partitioning key.
This creates a shuffle.
Do not repartition blindly.
22. Coalesce when reducing partitions without a full reshuffle
smaller = df.coalesce(20)
This is useful when reducing output partition count after a workload has already become small.
coalesce() generally avoids a full shuffle when reducing partitions.
It can produce uneven partitions, so validate the resulting workload.
23. Avoid repartition(1) for large production outputs
Risky:
df.repartition(1).write.parquet(
"Files/export"
)
This funnels output through one partition and can create a severe bottleneck.
If a downstream consumer requires one file, treat that as an export requirement and use a dedicated controlled process after the distributed transformation.
24. Check whether small files are part of the problem
A table containing many tiny files can increase planning and file-open overhead.
Inspect the source layout from the Lakehouse/OneLake side when a scan is unexpectedly expensive.
For Delta tables, use appropriate maintenance and optimization capabilities supported by Fabric rather than manually merging files in notebook code.
Do not optimize a table solely because file count is high; confirm the workload is affected.
25. Write only the columns you need
output = result.select(
"merchant_id",
"sale_date",
"amount",
)
(
output.write
.format("delta")
.mode("append")
.save("Tables/curated_sales")
)
Avoid carrying temporary/debug columns into production tables.
26. Be deliberate with overwrite
This is destructive to the target data covered by the operation:
(
result.write
.format("delta")
.mode("overwrite")
.save("Tables/curated_sales")
)
Do not use overwrite merely to make reruns easy.
Choose between:
- append
- merge/upsert
- partition replacement
- full rebuild
based on the pipeline’s data contract.
27. Stop comparing notebook runs that use different input volumes
Record basic execution context:
input_rows = df.count()
print(
"Input rows:",
input_rows,
)
Row count is not enough by itself, but performance comparisons are weak when runs process substantially different data volumes.
Also track when useful:
- source date range
- number of files
- partitions
- output rows
- join cardinality
28. Separate startup latency from Spark execution
Notebook wall-clock duration can include:
session acquisition
+ Spark startup
+ environment initialization
+ library initialization
+ actual Spark work
If a notebook’s Spark stages run quickly but the notebook still takes a long time end-to-end, optimize session reuse/startup rather than rewriting DataFrame code.
29. Use Fabric starter pools for faster session startup where appropriate
Starter pools are designed to reduce Spark session startup latency by using Microsoft-managed pre-warmed compute when available.
They are most useful when:
- notebooks are short
- startup is a meaningful percentage of runtime
- interactive development needs fast session acquisition
Starter-pool availability is best-effort; a session can still fall back to standard provisioning when pre-warmed capacity is not available.
Do not confuse startup optimization with query-execution optimization.
30. Use High Concurrency when compatible notebooks can share a session
Fabric High Concurrency mode allows compatible notebook workloads to share a running Spark application rather than creating a separate Spark session for each workload.
This is useful for:
- groups of notebook activities
- short notebooks where startup dominates
- parallel notebook orchestration
- improving Spark resource utilization
Session sharing requires compatible workload context, including the same user boundary, default Lakehouse, and Spark compute configuration.
Do not assume every notebook will share the same session.
31. Use a session tag for pipeline notebook reuse when High Concurrency is enabled
For Fabric pipeline Notebook activities, a session tag can direct compatible activities to reuse an existing Spark session.
Use a consistent tag for notebooks that are intended to share compute.
This is primarily a session startup / reuse optimization.
It does not make a slow Spark query itself faster.
32. Do not create excessive parallel notebook activity
Parallel notebook execution still shares finite Spark resources.
If many child notebooks run simultaneously, they can compete for:
- CPU
- memory
- executors
- shuffle resources
More concurrency does not automatically mean more throughput.
When using notebook orchestration such as parallel notebook execution, tune concurrency against the available Spark compute instead of maximizing it blindly.
33. Use autoscale and dynamic executor allocation for variable workloads when appropriate
Fabric Spark pools can use autoscaling and dynamic executor allocation.
This can help when workload size varies significantly.
Still monitor:
- scaling behavior
- queueing
- runtime
- capacity consumption
Autoscaling cannot compensate for pathological code such as accidental Cartesian joins or collecting massive results to the driver.
34. Consider Native Execution Engine only where supported and validated
Fabric provides a Native Execution Engine capability for supported Spark workloads.
It can improve supported operations, but should be treated as an execution-engine option rather than a substitute for:
- good data layout
- correct joins
- partition pruning
- avoiding skew
- right-sized compute
Validate compatibility and measure your actual workload before standardizing on it.
35. Use Spark monitoring to find the stage that dominates runtime
When a notebook is slow, inspect Fabric Spark monitoring / Spark UI information for:
- stage duration
- task duration
- input size
- shuffle read/write
- failed or retried tasks
- executor utilization
- skewed tasks
The notebook cell tells you where in the notebook.
Spark stage/task information tells you where in the distributed execution.
Use both.
What the result means
| Signal | What it usually tells you |
|---|---|
| Long delay before first Spark work | Session acquisition/startup/environment initialization |
| Large scan | Too much input data or insufficient pruning |
Exchange in plan |
Data redistribution/shuffle |
| One task much slower than peers | Possible skew or uneven partition size |
| High shuffle read/write | Join/aggregation/redistribution pressure |
| Driver memory pressure | Too much data moved to driver or large local objects |
| Repeated identical stages | Recomputed lineage or repeated actions |
| Large row growth after join | Join cardinality problem |
| Many tiny output files | Over-partitioned write pattern |
| Notebook fast interactively but slow in pipeline | Session/startup/config/concurrency differences may matter |
| Several short notebooks each start Spark | Session reuse / High Concurrency opportunity |
What to look for
Startup dominates total runtime
If Spark execution itself is short, investigate:
- starter pools
- High Concurrency
- pipeline session tags
- unnecessary separate notebook activities
Do not optimize a 10-second Spark query to fix a 2-minute startup problem.
A stage has one or a few straggling tasks
Likely causes include:
- key skew
- uneven partitions
- one unusually large file/partition
- executor pressure
Inspect the partition/key distribution.
Shuffle dominates runtime
Investigate:
- join strategy
- group keys
- unnecessary repartitioning
- data volume before the shuffle
- whether a small lookup can be broadcast
- skew
Driver fails or becomes slow
Look for:
collect()toPandas()- large Python lists/dictionaries
- row-by-row Python processing
- large results displayed in the notebook
Keep large data distributed.
Runtime changes dramatically for similar code
Compare:
- input volume
- file count
- plan
- Spark configuration
- pool/session mode
- concurrent workload
- skew
Do not compare wall-clock times without matching workload context.
Output creates many small files
Look for excessive partition counts at the write stage.
Reduce partitions deliberately when the output is already small, but avoid forcing one partition for large outputs.
Production notes
Fabric notebook performance has two different layers
Treat these separately:
Fabric session/platform overhead
Spark workload execution
Platform/session optimizations include:
- starter pools
- High Concurrency
- session reuse
- pool sizing
Spark optimizations include:
- pruning
- join strategy
- partitioning
- skew reduction
- avoiding driver collection
- file layout
Mixing these layers leads to poor diagnosis.
High Concurrency has compatibility rules
Compatible workloads can share a Spark application, but session sharing is not universal.
Differences in Lakehouse or Spark compute configuration can result in separate sessions.
Design notebook orchestration with that constraint in mind.
Current High Concurrency density is configurable
Fabric supports a configurable shared-notebook limit through the Environment Spark property:
spark.highConcurrency.max
The default is lower than the platform maximum.
Increasing concurrency can improve session density, but it also increases competition for the resources inside the shared Spark session.
Tune it from measured workload behavior rather than setting the maximum automatically.
Spark DMV-style snapshots do not exist in the same way as SQL Server
Performance investigation relies heavily on:
- physical plans
- Spark stages/tasks
- executor metrics
- Fabric monitoring
- workload/run history
Capture run metadata if you need before/after comparisons.
Capacity and pool sizing matter after code-level issues are understood
A larger pool can make a workload faster.
It can also make inefficient code more expensive.
Fix avoidable scans, row multiplication, skew, and driver collection before using compute size as the main tuning strategy.
Queueing can be a capacity problem rather than a notebook-code problem
If jobs spend time waiting to acquire Spark resources, investigate:
- workspace/capacity concurrency
- current Spark consumption
- parallel jobs
- High Concurrency suitability
- pool configuration
Do not interpret queue time as Spark transformation time.
Autotune is version- and feature-dependent
Fabric has Spark autotuning capabilities, but availability and compatibility depend on runtime and feature configuration.
Do not build a production performance strategy around autotune without verifying the currently supported runtime and workload mode.
Measure with representative production-scale data
A notebook that is fast on:
100,000 rows
may behave very differently on:
500 million rows
because of:
- shuffle
- skew
- partition count
- file layout
- memory pressure
Performance testing should resemble the real workload.
Common mistakes
1. Increasing Spark compute before finding the bottleneck
More compute does not fix:
- bad joins
- row explosion
collect()- skew
- repeated scans
Measure first.
2. Timing only DataFrame transformations
Spark is lazy.
Without an action, you may be timing plan construction rather than execution.
3. Using repartition(1) to get one output file
This creates a single-partition bottleneck.
Treat single-file export as a separate requirement.
4. Caching every DataFrame
Caching has materialization and memory cost.
Cache only expensive data that will be reused.
5. Broadcasting a table without checking its size
A broadcast that is too large can create memory pressure.
Validate the small side first.
6. Using Python UDFs for operations Spark already supports
Built-in Spark expressions usually give the optimizer a better execution path.
7. Ignoring data skew
Average partition size can look fine while one hot key controls stage duration.
Inspect distribution.
8. Increasing notebook concurrency blindly
Parallel notebooks can compete for the same finite resources.
Tune for throughput, not maximum parallel activity.
9. Treating session startup as query runtime
A notebook can be slow before the Spark transformation even begins.
Separate startup and execution measurements.
10. Optimizing code while ignoring file layout
Many small files or poor partition layout can make a good DataFrame transformation slow.
Inspect the storage side when scan planning dominates.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.