Browse cheat sheets · Microsoft Fabric

Technology: Microsoft FabricCategory: PerformanceDifficulty: IntermediateUpdated

Fabric Notebook Performance

Troubleshoot and improve Microsoft Fabric notebook performance by separating Spark session startup, data scan, shuffle, skew, driver pressure, concurrency, and write-path bottlenecks.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Measure the Spark action, not only DataFrame construction
  5. 2. Inspect the physical plan
  6. 3. Check the current number of partitions
  7. 4. Inspect partition row distribution
  8. 5. Filter before expensive transformations
  9. 6. Avoid select("*") when only a few columns are needed
  10. 7. Inspect join strategy
  11. 8. Broadcast a genuinely small lookup table
  12. 9. Check for duplicate join keys before blaming Spark
  13. 10. Reduce data before a large join
  14. 11. Check for skewed join/group keys
  15. 12. Avoid collect() for large results
  16. 13. Avoid unnecessary toPandas()
  17. 14. Avoid Python loops over Spark rows
  18. 15. Prefer built-in Spark expressions over Python UDFs
  19. 16. Cache only reused DataFrames
  20. 17. Check cache state
  21. 18. Avoid unnecessary repeated actions
  22. 19. Inspect Spark configuration relevant to the session
  23. 20. Inspect important adaptive execution settings
  24. 21. Repartition when you need a real redistribution
  25. 22. Coalesce when reducing partitions without a full reshuffle
  26. 23. Avoid repartition(1) for large production outputs
  27. 24. Check whether small files are part of the problem
  28. 25. Write only the columns you need
  29. 26. Be deliberate with overwrite
  30. 27. Stop comparing notebook runs that use different input volumes
  31. 28. Separate startup latency from Spark execution
  32. 29. Use Fabric starter pools for faster session startup where appropriate
  33. 30. Use High Concurrency when compatible notebooks can share a session
  34. 31. Use a session tag for pipeline notebook reuse when High Concurrency is enabled
  35. 32. Do not create excessive parallel notebook activity
  36. 33. Use autoscale and dynamic executor allocation for variable workloads when appropriate
  37. 34. Consider Native Execution Engine only where supported and validated
  38. 35. Use Spark monitoring to find the stage that dominates runtime
  39. What the result means
  40. What to look for
  41. Startup dominates total runtime
  42. A stage has one or a few straggling tasks
  43. Shuffle dominates runtime
  44. Driver fails or becomes slow
  45. Runtime changes dramatically for similar code
  46. Output creates many small files
  47. Production notes
  48. Fabric notebook performance has two different layers
  49. High Concurrency has compatibility rules
  50. Current High Concurrency density is configurable
  51. Spark DMV-style snapshots do not exist in the same way as SQL Server
  52. Capacity and pool sizing matter after code-level issues are understood
  53. Queueing can be a capacity problem rather than a notebook-code problem
  54. Autotune is version- and feature-dependent
  55. Measure with representative production-scale data
  56. Common mistakes
  57. 1. Increasing Spark compute before finding the bottleneck
  58. 2. Timing only DataFrame transformations
  59. 3. Using repartition(1) to get one output file
  60. 4. Caching every DataFrame
  61. 5. Broadcasting a table without checking its size
  62. 6. Using Python UDFs for operations Spark already supports
  63. 7. Ignoring data skew
  64. 8. Increasing notebook concurrency blindly
  65. 9. Treating session startup as query runtime
  66. 10. Optimizing code while ignoring file layout
  67. Next step
  68. Related deep dives

Problem

A Microsoft Fabric notebook is slow, inconsistent, or consumes more Spark capacity than expected.

The delay may come from very different places:

  • Spark session startup
  • reading too much data
  • too many small files
  • poor partitioning
  • shuffle-heavy joins or aggregations
  • data skew
  • Python code running on the driver
  • unnecessary collect() / toPandas()
  • repeated recomputation
  • excessive notebook concurrency
  • undersized or oversized Spark compute
  • slow Delta writes
  • several notebooks starting separate Spark sessions unnecessarily

The goal is to identify where the time is going before changing Spark settings.

Quick answer

Use this flow:

1. Separate session startup time from notebook execution time
2. Identify the slow Spark action/stage
3. Inspect the physical plan
4. Check how much data is being scanned
5. Check partitions and shuffle
6. Check for skew
7. Check for driver-side collection / Python loops
8. Check repeated recomputation
9. Check file layout and Delta write pattern
10. Check pool sizing, autoscale, concurrency, and session reuse
11. Change one thing and measure again

Start with the operation you believe is slow:

from time import perf_counter

start = perf_counter()

result = (
    spark.read.format("delta")
    .load("Tables/sales")
    .filter("sale_date >= '2026-10-01'")
    .groupBy("merchant_id")
    .sum("amount")
)

result.count()  # force execution

elapsed = perf_counter() - start

print(f"Elapsed: {elapsed:.2f} seconds")

Then inspect the plan:

result.explain("formatted")

Do not tune Spark configuration until you know whether the bottleneck is scan, shuffle, skew, compute, startup, or the driver.

Code / commands

1. Measure the Spark action, not only DataFrame construction

Spark DataFrames are lazy.

This:

start = perf_counter()

result = (
    df.filter("amount > 100")
    .groupBy("merchant_id")
    .sum("amount")
)

print(perf_counter() - start)

mostly measures logical-plan construction.

Force an action:

from time import perf_counter

start = perf_counter()

result = (
    df.filter("amount > 100")
    .groupBy("merchant_id")
    .sum("amount")
)

row_count = result.count()

elapsed = perf_counter() - start

print("Rows:", row_count)
print(f"Elapsed: {elapsed:.2f} seconds")

Use a representative action.

Do not add a large collect() merely to benchmark.


2. Inspect the physical plan

result.explain("formatted")

Use this before changing partition counts or compute size.

Look for:

  • scans
  • filters
  • joins
  • exchanges
  • sorts
  • aggregations
  • broadcast joins
  • repeated scans

An Exchange usually means Spark must redistribute data across partitions.

That is not automatically bad, but large or repeated exchanges deserve attention.


3. Check the current number of partitions

print(df.rdd.getNumPartitions())

After a shuffle:

result = (
    df.groupBy("merchant_id")
    .sum("amount")
)

print(result.rdd.getNumPartitions())

Partition count alone does not tell you whether the workload is healthy.

You need to compare it with:

  • data volume
  • partition-size distribution
  • executor count
  • task duration
  • skew

4. Inspect partition row distribution

For targeted troubleshooting:

from pyspark.sql import functions as F

partition_rows = (
    df.withColumn(
        "_partition_id",
        F.spark_partition_id(),
    )
    .groupBy("_partition_id")
    .count()
    .orderBy(F.desc("count"))
)

partition_rows.show(50, truncate=False)

Use this when you suspect skew.

If a small number of partitions contain far more rows than the rest, those tasks can dominate stage duration.

Row count is only one signal: variable-width rows can have different byte sizes.


5. Filter before expensive transformations

Prefer:

filtered = (
    spark.read.format("delta")
    .load("Tables/sales")
    .filter(F.col("sale_date") >= "2026-10-01")
    .select(
        "merchant_id",
        "sale_date",
        "amount",
    )
)

over loading wide data and carrying every column through joins and aggregations.

Push down:

  • filters
  • column selection

as early as practical.

This reduces the amount of data processed downstream.


6. Avoid select("*") when only a few columns are needed

Instead of:

df = (
    spark.read.format("delta")
    .load("Tables/sales")
    .select("*")
)

prefer:

df = (
    spark.read.format("delta")
    .load("Tables/sales")
    .select(
        "sale_id",
        "merchant_id",
        "sale_date",
        "amount",
    )
)

Column pruning matters most on wide analytical tables.


7. Inspect join strategy

joined = sales.join(
    merchants,
    on="merchant_id",
    how="left",
)

joined.explain("formatted")

Look for whether Spark chooses:

  • broadcast hash join
  • sort-merge join
  • another strategy

Do not force a broadcast until you know the smaller side is appropriate for broadcasting.


8. Broadcast a genuinely small lookup table

When you have verified that the lookup is small enough:

from pyspark.sql import functions as F

joined = sales.join(
    F.broadcast(merchants),
    on="merchant_id",
    how="left",
)

Then inspect again:

joined.explain("formatted")

Broadcasting avoids shuffling the large side for some join patterns.

Do not broadcast a large table merely because a join is slow.


9. Check for duplicate join keys before blaming Spark

A join may be slow because it multiplies rows.

Check key cardinality:

duplicate_keys = (
    merchants.groupBy("merchant_id")
    .count()
    .filter(F.col("count") > 1)
)

duplicate_keys.show(50, truncate=False)

If a supposed dimension table is not unique, a join can:

  • multiply rows
  • increase shuffle volume
  • increase memory pressure
  • inflate downstream metrics

Fix the data relationship before tuning the cluster.


10. Reduce data before a large join

Prefer:

sales_small = (
    sales
    .filter(F.col("sale_date") >= "2026-10-01")
    .select(
        "merchant_id",
        "amount",
    )
)

merchants_small = merchants.select(
    "merchant_id",
    "region",
)

joined = sales_small.join(
    merchants_small,
    "merchant_id",
    "left",
)

Filtering and projecting first can materially reduce shuffle work.


11. Check for skewed join/group keys

Start with:

(
    df.groupBy("merchant_id")
    .count()
    .orderBy(F.desc("count"))
    .show(50, truncate=False)
)

A very hot key can send a disproportionate amount of data to one partition.

Common examples:

  • one default/unknown key
  • null replacement such as 0
  • one very large customer
  • one dominant merchant
  • malformed data collapsing many records onto one key

Do not immediately add random salting.

First identify the skewed key and confirm the business reason.


12. Avoid collect() for large results

Risky:

rows = df.collect()

This sends the full result to the driver.

Safer for inspection:

df.show(20, truncate=False)

or:

rows = df.limit(100).collect()

Use full collection only when the result is deliberately small.


13. Avoid unnecessary toPandas()

Risky:

pdf = df.toPandas()

The resulting pandas DataFrame must fit in driver memory.

For inspection:

pdf = (
    df.limit(1000)
    .toPandas()
)

For large transformations, keep the work distributed in Spark.


14. Avoid Python loops over Spark rows

Bad pattern:

for row in df.collect():
    process(row)

Prefer Spark transformations:

result = (
    df.withColumn(
        "net_amount",
        F.col("amount") - F.col("discount"),
    )
)

Moving data to Python row-by-row removes Spark’s distributed execution advantage.


15. Prefer built-in Spark expressions over Python UDFs

Prefer:

df = df.withColumn(
    "normalized_email",
    F.lower(
        F.trim(F.col("email"))
    ),
)

over a Python UDF for operations already available in Spark SQL functions.

Built-in expressions give Spark more opportunities to optimize execution and avoid Python serialization overhead.

Use a UDF only when the required logic cannot be expressed with built-in Spark functions or another more efficient supported mechanism.


16. Cache only reused DataFrames

If the same expensive result is reused multiple times:

filtered = (
    df.filter(F.col("sale_date") >= "2026-10-01")
    .select(
        "merchant_id",
        "amount",
    )
    .cache()
)

Materialize it:

filtered.count()

Reuse it:

by_merchant = (
    filtered.groupBy("merchant_id")
    .sum("amount")
)

total = filtered.agg(
    F.sum("amount")
)

When finished:

filtered.unpersist()

Do not cache every DataFrame.

Caching data that is used once usually adds work rather than removing it.


17. Check cache state

print(filtered.storageLevel)

Use this when debugging whether a DataFrame is actually persisted.

Remember that cache() is lazy until an action materializes it.


18. Avoid unnecessary repeated actions

This triggers multiple jobs:

print(df.count())
print(df.filter(F.col("status") == "failed").count())
print(df.agg(F.sum("amount")).collect())

That may be acceptable, but each action can recompute upstream lineage if the input is not cached or otherwise materialized.

If several actions reuse an expensive common prefix, consider materializing that prefix once.


19. Inspect Spark configuration relevant to the session

for key, value in sorted(
    spark.sparkContext.getConf().getAll()
):
    print(f"{key}={value}")

Use this when:

  • performance differs between environments
  • a notebook behaves differently from a pipeline
  • you suspect session configuration differences

Do not change configuration simply because a value looks unfamiliar.


20. Inspect important adaptive execution settings

keys = [
    "spark.sql.adaptive.enabled",
    "spark.sql.shuffle.partitions",
    "spark.sql.autoBroadcastJoinThreshold",
]

for key in keys:
    try:
        print(
            key,
            spark.conf.get(key),
        )
    except Exception:
        print(
            key,
            "<not available>",
        )

Adaptive Query Execution can change the physical execution strategy based on runtime information.

Treat manually tuning spark.sql.shuffle.partitions as workload-specific, not as a universal fix.


21. Repartition when you need a real redistribution

repartitioned = df.repartition(
    200,
    "merchant_id",
)

Use repartition() when you intentionally need to redistribute data, for example before a repeated operation using the same partitioning key.

This creates a shuffle.

Do not repartition blindly.


22. Coalesce when reducing partitions without a full reshuffle

smaller = df.coalesce(20)

This is useful when reducing output partition count after a workload has already become small.

coalesce() generally avoids a full shuffle when reducing partitions.

It can produce uneven partitions, so validate the resulting workload.


23. Avoid repartition(1) for large production outputs

Risky:

df.repartition(1).write.parquet(
    "Files/export"
)

This funnels output through one partition and can create a severe bottleneck.

If a downstream consumer requires one file, treat that as an export requirement and use a dedicated controlled process after the distributed transformation.


24. Check whether small files are part of the problem

A table containing many tiny files can increase planning and file-open overhead.

Inspect the source layout from the Lakehouse/OneLake side when a scan is unexpectedly expensive.

For Delta tables, use appropriate maintenance and optimization capabilities supported by Fabric rather than manually merging files in notebook code.

Do not optimize a table solely because file count is high; confirm the workload is affected.


25. Write only the columns you need

output = result.select(
    "merchant_id",
    "sale_date",
    "amount",
)

(
    output.write
    .format("delta")
    .mode("append")
    .save("Tables/curated_sales")
)

Avoid carrying temporary/debug columns into production tables.


26. Be deliberate with overwrite

This is destructive to the target data covered by the operation:

(
    result.write
    .format("delta")
    .mode("overwrite")
    .save("Tables/curated_sales")
)

Do not use overwrite merely to make reruns easy.

Choose between:

  • append
  • merge/upsert
  • partition replacement
  • full rebuild

based on the pipeline’s data contract.


27. Stop comparing notebook runs that use different input volumes

Record basic execution context:

input_rows = df.count()

print(
    "Input rows:",
    input_rows,
)

Row count is not enough by itself, but performance comparisons are weak when runs process substantially different data volumes.

Also track when useful:

  • source date range
  • number of files
  • partitions
  • output rows
  • join cardinality

28. Separate startup latency from Spark execution

Notebook wall-clock duration can include:

session acquisition
+ Spark startup
+ environment initialization
+ library initialization
+ actual Spark work

If a notebook’s Spark stages run quickly but the notebook still takes a long time end-to-end, optimize session reuse/startup rather than rewriting DataFrame code.


29. Use Fabric starter pools for faster session startup where appropriate

Starter pools are designed to reduce Spark session startup latency by using Microsoft-managed pre-warmed compute when available.

They are most useful when:

  • notebooks are short
  • startup is a meaningful percentage of runtime
  • interactive development needs fast session acquisition

Starter-pool availability is best-effort; a session can still fall back to standard provisioning when pre-warmed capacity is not available.

Do not confuse startup optimization with query-execution optimization.


30. Use High Concurrency when compatible notebooks can share a session

Fabric High Concurrency mode allows compatible notebook workloads to share a running Spark application rather than creating a separate Spark session for each workload.

This is useful for:

  • groups of notebook activities
  • short notebooks where startup dominates
  • parallel notebook orchestration
  • improving Spark resource utilization

Session sharing requires compatible workload context, including the same user boundary, default Lakehouse, and Spark compute configuration.

Do not assume every notebook will share the same session.


31. Use a session tag for pipeline notebook reuse when High Concurrency is enabled

For Fabric pipeline Notebook activities, a session tag can direct compatible activities to reuse an existing Spark session.

Use a consistent tag for notebooks that are intended to share compute.

This is primarily a session startup / reuse optimization.

It does not make a slow Spark query itself faster.


32. Do not create excessive parallel notebook activity

Parallel notebook execution still shares finite Spark resources.

If many child notebooks run simultaneously, they can compete for:

  • CPU
  • memory
  • executors
  • shuffle resources

More concurrency does not automatically mean more throughput.

When using notebook orchestration such as parallel notebook execution, tune concurrency against the available Spark compute instead of maximizing it blindly.


33. Use autoscale and dynamic executor allocation for variable workloads when appropriate

Fabric Spark pools can use autoscaling and dynamic executor allocation.

This can help when workload size varies significantly.

Still monitor:

  • scaling behavior
  • queueing
  • runtime
  • capacity consumption

Autoscaling cannot compensate for pathological code such as accidental Cartesian joins or collecting massive results to the driver.


34. Consider Native Execution Engine only where supported and validated

Fabric provides a Native Execution Engine capability for supported Spark workloads.

It can improve supported operations, but should be treated as an execution-engine option rather than a substitute for:

  • good data layout
  • correct joins
  • partition pruning
  • avoiding skew
  • right-sized compute

Validate compatibility and measure your actual workload before standardizing on it.


35. Use Spark monitoring to find the stage that dominates runtime

When a notebook is slow, inspect Fabric Spark monitoring / Spark UI information for:

  • stage duration
  • task duration
  • input size
  • shuffle read/write
  • failed or retried tasks
  • executor utilization
  • skewed tasks

The notebook cell tells you where in the notebook.

Spark stage/task information tells you where in the distributed execution.

Use both.

What the result means

Signal What it usually tells you
Long delay before first Spark work Session acquisition/startup/environment initialization
Large scan Too much input data or insufficient pruning
Exchange in plan Data redistribution/shuffle
One task much slower than peers Possible skew or uneven partition size
High shuffle read/write Join/aggregation/redistribution pressure
Driver memory pressure Too much data moved to driver or large local objects
Repeated identical stages Recomputed lineage or repeated actions
Large row growth after join Join cardinality problem
Many tiny output files Over-partitioned write pattern
Notebook fast interactively but slow in pipeline Session/startup/config/concurrency differences may matter
Several short notebooks each start Spark Session reuse / High Concurrency opportunity

What to look for

Startup dominates total runtime

If Spark execution itself is short, investigate:

  • starter pools
  • High Concurrency
  • pipeline session tags
  • unnecessary separate notebook activities

Do not optimize a 10-second Spark query to fix a 2-minute startup problem.

A stage has one or a few straggling tasks

Likely causes include:

  • key skew
  • uneven partitions
  • one unusually large file/partition
  • executor pressure

Inspect the partition/key distribution.

Shuffle dominates runtime

Investigate:

  • join strategy
  • group keys
  • unnecessary repartitioning
  • data volume before the shuffle
  • whether a small lookup can be broadcast
  • skew

Driver fails or becomes slow

Look for:

  • collect()
  • toPandas()
  • large Python lists/dictionaries
  • row-by-row Python processing
  • large results displayed in the notebook

Keep large data distributed.

Runtime changes dramatically for similar code

Compare:

  • input volume
  • file count
  • plan
  • Spark configuration
  • pool/session mode
  • concurrent workload
  • skew

Do not compare wall-clock times without matching workload context.

Output creates many small files

Look for excessive partition counts at the write stage.

Reduce partitions deliberately when the output is already small, but avoid forcing one partition for large outputs.

Production notes

Fabric notebook performance has two different layers

Treat these separately:

Fabric session/platform overhead
Spark workload execution

Platform/session optimizations include:

  • starter pools
  • High Concurrency
  • session reuse
  • pool sizing

Spark optimizations include:

  • pruning
  • join strategy
  • partitioning
  • skew reduction
  • avoiding driver collection
  • file layout

Mixing these layers leads to poor diagnosis.

High Concurrency has compatibility rules

Compatible workloads can share a Spark application, but session sharing is not universal.

Differences in Lakehouse or Spark compute configuration can result in separate sessions.

Design notebook orchestration with that constraint in mind.

Current High Concurrency density is configurable

Fabric supports a configurable shared-notebook limit through the Environment Spark property:

spark.highConcurrency.max

The default is lower than the platform maximum.

Increasing concurrency can improve session density, but it also increases competition for the resources inside the shared Spark session.

Tune it from measured workload behavior rather than setting the maximum automatically.

Spark DMV-style snapshots do not exist in the same way as SQL Server

Performance investigation relies heavily on:

  • physical plans
  • Spark stages/tasks
  • executor metrics
  • Fabric monitoring
  • workload/run history

Capture run metadata if you need before/after comparisons.

Capacity and pool sizing matter after code-level issues are understood

A larger pool can make a workload faster.

It can also make inefficient code more expensive.

Fix avoidable scans, row multiplication, skew, and driver collection before using compute size as the main tuning strategy.

Queueing can be a capacity problem rather than a notebook-code problem

If jobs spend time waiting to acquire Spark resources, investigate:

  • workspace/capacity concurrency
  • current Spark consumption
  • parallel jobs
  • High Concurrency suitability
  • pool configuration

Do not interpret queue time as Spark transformation time.

Autotune is version- and feature-dependent

Fabric has Spark autotuning capabilities, but availability and compatibility depend on runtime and feature configuration.

Do not build a production performance strategy around autotune without verifying the currently supported runtime and workload mode.

Measure with representative production-scale data

A notebook that is fast on:

100,000 rows

may behave very differently on:

500 million rows

because of:

  • shuffle
  • skew
  • partition count
  • file layout
  • memory pressure

Performance testing should resemble the real workload.

Common mistakes

1. Increasing Spark compute before finding the bottleneck

More compute does not fix:

  • bad joins
  • row explosion
  • collect()
  • skew
  • repeated scans

Measure first.

2. Timing only DataFrame transformations

Spark is lazy.

Without an action, you may be timing plan construction rather than execution.

3. Using repartition(1) to get one output file

This creates a single-partition bottleneck.

Treat single-file export as a separate requirement.

4. Caching every DataFrame

Caching has materialization and memory cost.

Cache only expensive data that will be reused.

5. Broadcasting a table without checking its size

A broadcast that is too large can create memory pressure.

Validate the small side first.

6. Using Python UDFs for operations Spark already supports

Built-in Spark expressions usually give the optimizer a better execution path.

7. Ignoring data skew

Average partition size can look fine while one hot key controls stage duration.

Inspect distribution.

8. Increasing notebook concurrency blindly

Parallel notebooks can compete for the same finite resources.

Tune for throughput, not maximum parallel activity.

9. Treating session startup as query runtime

A notebook can be slow before the Spark transformation even begins.

Separate startup and execution measurements.

10. Optimizing code while ignoring file layout

Many small files or poor partition layout can make a good DataFrame transformation slow.

Inspect the storage side when scan planning dominates.

Next step

Use the observed bottleneck to choose the next investigation.

  • Startup dominates → evaluate starter pools, High Concurrency, and session reuse.
  • Scan dominates → reduce columns/rows early and inspect Delta/file layout.
  • Shuffle dominates → inspect joins, grouping keys, partitioning, and unnecessary exchanges.
  • One task dominates → investigate skew and partition distribution.
  • Driver pressure appears → remove large collect(), toPandas(), and Python-side processing.
  • Repeated stages appear → check repeated actions and cache only reusable expensive intermediates.
  • Join output explodes → validate key uniqueness and relationship cardinality.
  • Write is slow / creates many files → inspect final partition count and Delta maintenance strategy.
  • Jobs queue before running → investigate Fabric Spark capacity/concurrency rather than query code.
  • Code is already efficient but runtime remains high → evaluate pool size, autoscale, dynamic allocation, and supported execution-engine options.

← All cheat sheets