Microsoft Fabric
How to Make Microsoft Fabric Notebooks Faster
A practical workflow for reducing Fabric notebook startup, read, shuffle, transformation and Delta write time without guessing.
On this page
Short answer: a slow Fabric notebook is rarely slow everywhere. Its run time splits into separate phases: waiting in a queue for capacity, starting a Spark session, reading data, transforming and shuffling it, writing the result, and waiting on whatever runs after it. Each phase has its own evidence and its own fixes. Find the phase that grew before changing any setting: a faster shuffle does nothing for a run that spent most of its time waiting for capacity.
- QueuePipeline or scheduled job waiting for Spark capacity
- Session startupStarter pool or on-demand cluster, environment libraries
- ReadBytes and files scanned, pruning, small files
- TransformPlan shape, repeated actions, driver-side collects
- ShuffleShuffle read and write, skewed tasks, spill
- WriteOutput files and partitions, write mode, compaction
After the write, the pipeline may still wait on a downstream activity; that time belongs to orchestration, not Spark.
Problem
A production pipeline calls a notebook that sometimes completes quickly and sometimes takes much longer. The notebook reads a broad history, performs joins and aggregations, and overwrites a Delta target. Before changing capacity or adding cache calls, separate orchestration delay, session startup, Spark execution, and output writing. (If the whole capacity is under pressure, start with where your capacity goes instead.)
Record at least the pipeline run, notebook run, input window, input rows or files, output rows, and duration of major stages. Compare a slow run with a normal run that processed similar data. Different volumes are different experiments.
Why it happens
Spark is lazy: DataFrame transformations build a plan, while actions such as count, display, collection, and writes execute it. Several debugging actions can therefore run the same expensive lineage repeatedly. Wide operations—including joins, groupings, distinct operations, and repartitioning—can move data across executors through a shuffle. A small number of very large partitions can leave most tasks finished while a few stragglers dominate elapsed time.
Input design matters too. Reading an entire table and filtering afterward may prevent useful partition pruning or data skipping. Selecting every column increases I/O and serialization. Thousands of small files add listing and scheduling overhead (diagnosing and compacting small files). On the output side, forced repartitioning and repeated tiny appends can exchange one problem for another.
Fabric adds a session layer. Compatible workloads may reuse high-concurrency sessions; other configurations require a new session. Starter pools can reduce acquisition time when their constraints fit, but they are a best-effort fast path rather than a guarantee. Custom configuration, libraries, environments, and networking can change startup behavior. Microsoft’s Spark monitoring guidance and session detail views help separate those causes.
A diagnosis sequence
- Establish a baseline. Pick a normal run with similar input and record its total duration and the measures below. Without a baseline, “slow” is an opinion.
- Separate startup from execution. Compare the activity start, the session-ready time and the first Spark job. Time before the first job is queue or session startup; see reducing notebook startup time.
- Check queueing and concurrency. Notebook jobs started by pipelines or the scheduler wait in a first-in, first-out queue when the capacity’s Spark cores are in use; interactive runs are rejected instead. If runs are slow only at certain times of day, look at what else shares the capacity before tuning code: the Fabric capacity saturation runbook covers that investigation.
- Check the read. Bytes and files read by the scan stages, whether filters reached the scan, and whether the table has many small files.
- Check shuffle and skew. Shuffle read and write per stage, spill, and the spread of task durations: one task far longer than the median means skew.
- Check transformation logic. Repeated actions over the same lineage, joins on unfiltered data,
collect()ortoPandas()on large results, Python UDFs in hot paths. - Check the write. Number and size of output files, partition columns, append versus overwrite versus merge.
- Change one thing and rerun with comparable input.
- Compare with the baseline on duration and on correctness: row counts, duplicates, rejected rows.
What to measure
| Measure | Where to find it | What it tells you |
|---|---|---|
| Total duration | Pipeline run, Monitoring hub | Whether the run is really slower than its baseline |
| Queue and session startup time | Monitoring hub, session details | Whether time is lost before your code runs |
| Duration per stage | Spark UI (Stages), Spark application detail page | Which phase dominates |
| Task duration, median vs max | Stage summary metrics in the Spark UI | Skew: one task holding a whole stage |
| Input bytes and files read | Stage input metrics, the scan node in the plan | Over-reading, missing pruning, small files |
| Shuffle read and write, spill | Stage metrics | Expensive wide operations, memory pressure |
| Number of tasks per stage | Stage detail | Too few (idle cores) or too many (scheduling overhead) |
| Output file count and size | DESCRIBE DETAIL, table history |
Writes that leave small files for every later reader |
Small files
Many tiny files make a Delta table slow to read and to maintain. Each file has to be listed and opened, adds an entry the Delta log must track, and usually becomes its own task, so a read that needs a handful of tasks schedules thousands of short ones. Merges and deletes also have more files to rewrite. The usual cause is on the write side: frequent small appends, streaming micro-batches, or an output partitioned on a high-cardinality column.
Check the average file size before choosing a fix:
detail = spark.sql("DESCRIBE DETAIL SalesLakehouse.silver.orders").first()
avg_mb = detail.sizeInBytes / max(detail.numFiles, 1) / 1024**2
print({"files": detail.numFiles, "avg_file_mb": round(avg_mb, 1)})An average of a few megabytes on a large table is a warning sign. Compact with OPTIMIZE, consider Optimize Write for frequent small writes, and batch appends where the pipeline allows it. Delta table maintenance: OPTIMIZE and VACUUM covers when and how to run them safely.
Practical recommendations
Begin with a timing breakdown. If most time occurs before the first Spark job, investigate session acquisition and libraries. If a stage dominates, inspect its jobs, tasks, input, shuffle, spill, skew, and output. If the write dominates, inspect partition counts, file sizes, write mode, and downstream table design.
Read only the required columns and time range. Put selective filters next to the read and use predicates aligned with the table’s physical design. Confirm in the executed plan that pruning or skipping occurred; source syntax alone does not prove it.
Reduce data before wide operations. Aggregate at the required grain before joining when that preserves correctness. Broadcast only genuinely small reference data after measuring it. Avoid collect() and toPandas() for large results because they move data to the driver. Use limited samples for inspection.
Cache only a DataFrame that is expensive to recompute and reused by multiple actions. Materialize it deliberately, verify storage pressure, and unpersist it when finished. Caching a one-use DataFrame adds work and memory pressure without saving recomputation.
For Delta output, choose append, overwrite, or merge from the data contract. Avoid repartitioning merely to produce a preferred file count: repartition causes a shuffle, while coalesce reduces partitions without a full redistribution but can create imbalance. Evaluate Optimize Write, compaction, V-Order, and Z-Order according to write frequency and consumers. Microsoft notes that V-Order can help cross-engine reads while adding Spark write cost, so it is not a universal notebook speed switch.
Example pattern
The example keeps projection and filtering close to the source, performs one reusable aggregation, and validates output without collecting full data.
from pyspark.sql import functions as F
from pyspark.storagelevel import StorageLevel
run_date = "2026-10-02"
orders = (
spark.read.table("SalesLakehouse.silver.orders")
.select("order_id", "customer_id", "order_date", "status", "amount")
.where(F.col("order_date") == F.lit(run_date))
.where(F.col("status") == F.lit("Completed"))
)
daily_customer = (
orders.groupBy("customer_id")
.agg(
F.countDistinct("order_id").alias("order_count"),
F.sum("amount").alias("revenue"),
)
.persist(StorageLevel.MEMORY_AND_DISK)
)
output_rows = daily_customer.count()
(
daily_customer.write
.format("delta")
.mode("append")
.saveAsTable("SalesLakehouse.gold.daily_customer_sales")
)
print({"run_date": run_date, "output_rows": output_rows})
daily_customer.unpersist()This is a pattern, not a prescription. If daily_customer is used only by the write, remove persistence and the separate count or replace validation with a metric produced elsewhere. If reruns are possible, a blind append may duplicate the date; the idempotent write design belongs to the pipeline contract.
Use explain("formatted") before and after an important change. Look for filters near the scan, unexpected exchanges, join strategies, and repeated branches. Then confirm with executed Spark jobs because estimates and plans do not show every runtime condition.
Common mistakes
- Changing executor or shuffle settings before identifying the slow stage.
- Calling
count, display, and write on the same uncached expensive lineage. - Reading a complete history for an incremental run.
- Selecting every column “just in case.”
- Broadcasting a lookup based on row count without considering bytes.
- Caching everything and creating eviction or spill pressure.
- Repartitioning every output to one file, which destroys parallelism.
- Treating session reuse as isolation-free; shared compute still needs compatible configuration and workload planning.
- Comparing runs with different input volumes and calling the difference a regression.
Symptom, likely cause, first check
| Symptom | Likely cause | First check |
|---|---|---|
| Long time before the first Spark job | Queueing, or a slow session start (on-demand cluster, environment libraries) | Session details: pool source and delay reason; queue time in monitoring |
| Fast startup, slow execution | Read volume, wide operations or the write | Stage durations in the Spark UI |
| One stage dominates | A large scan, join or aggregation | That stage’s input bytes, shuffle and spill |
| Thousands of very short tasks | Small files or too many partitions | numFiles from DESCRIBE DETAIL; tasks per stage |
| One task runs far longer than the others | A skewed key in a join or aggregation | Max vs median task duration; row counts per key |
| The write is slow | Many output files or partitions; a merge into a large target | Output file count; the merge condition and target size |
| Duration varies with the time of day | Capacity contention or queueing | Capacity utilization and queue time at those hours |
| Spark finishes quickly but the pipeline is late | A downstream dependency or orchestration wait | The pipeline activity timeline, not the Spark UI |
Change one understood variable, rerun comparable input, and keep correctness controls alongside performance measures. A faster notebook that loses unmatched rows, duplicates a partition, or changes business grain is a failed optimization.
Related
- How to reduce Fabric notebook startup time and using session tags in Fabric pipelines, when time is lost before the code runs
- Increase the high-concurrency notebook session limit
- Troubleshooting Microsoft Fabric performance in production, for problems beyond one notebook
- Monitoring Fabric pipelines and notebooks in production, to keep the baseline you compare against
- pandas vs Polars vs DuckDB vs PySpark in Microsoft Fabric, when the question is whether the notebook should run on Spark at all
Sources
- Microsoft Learn: Spark monitoring best practices, Spark application detail monitoring, Job queueing for Spark in Fabric, Delta table optimization and V-Order
- Apache Spark: SQL performance tuning (adaptive query execution, skew handling), Web UI
Fabric behavior checked against Microsoft Learn in November 2026.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetFabric Notebook PerformanceTroubleshoot and improve Microsoft Fabric notebook performance by separating Spark session startup, data scan, shuffle, skew, driver pressure, concurrency, and write-path bottlenecks.Open cheat sheet
- RoadmapMicrosoft Fabric Notebook PerformanceA practical workflow for reducing Fabric notebook startup, read, shuffle, transformation and Delta write time without guessing.View roadmap
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.