Microsoft Fabric

How to Make Microsoft Fabric Notebooks Faster

A practical workflow for reducing Fabric notebook startup, read, shuffle, transformation and Delta write time without guessing.

By JaviPublished Updated 10 min readIntermediate

Notebook flowing through a reusable session and Spark processing into Delta output

Short answer: a slow Fabric notebook is rarely slow everywhere. Its run time splits into separate phases: waiting in a queue for capacity, starting a Spark session, reading data, transforming and shuffling it, writing the result, and waiting on whatever runs after it. Each phase has its own evidence and its own fixes. Find the phase that grew before changing any setting: a faster shuffle does nothing for a run that spent most of its time waiting for capacity.

  1. QueuePipeline or scheduled job waiting for Spark capacity
  2. Session startupStarter pool or on-demand cluster, environment libraries
  3. ReadBytes and files scanned, pruning, small files
  4. TransformPlan shape, repeated actions, driver-side collects
  5. ShuffleShuffle read and write, skewed tasks, spill
  6. WriteOutput files and partitions, write mode, compaction
Where a notebook run spends its time. Measure each phase separately; the slow one decides what to optimize. Queue and session startup happen before your code runs.

After the write, the pipeline may still wait on a downstream activity; that time belongs to orchestration, not Spark.

Problem

A production pipeline calls a notebook that sometimes completes quickly and sometimes takes much longer. The notebook reads a broad history, performs joins and aggregations, and overwrites a Delta target. Before changing capacity or adding cache calls, separate orchestration delay, session startup, Spark execution, and output writing. (If the whole capacity is under pressure, start with where your capacity goes instead.)

Record at least the pipeline run, notebook run, input window, input rows or files, output rows, and duration of major stages. Compare a slow run with a normal run that processed similar data. Different volumes are different experiments.

Why it happens

Spark is lazy: DataFrame transformations build a plan, while actions such as count, display, collection, and writes execute it. Several debugging actions can therefore run the same expensive lineage repeatedly. Wide operations—including joins, groupings, distinct operations, and repartitioning—can move data across executors through a shuffle. A small number of very large partitions can leave most tasks finished while a few stragglers dominate elapsed time.

Input design matters too. Reading an entire table and filtering afterward may prevent useful partition pruning or data skipping. Selecting every column increases I/O and serialization. Thousands of small files add listing and scheduling overhead (diagnosing and compacting small files). On the output side, forced repartitioning and repeated tiny appends can exchange one problem for another.

Fabric adds a session layer. Compatible workloads may reuse high-concurrency sessions; other configurations require a new session. Starter pools can reduce acquisition time when their constraints fit, but they are a best-effort fast path rather than a guarantee. Custom configuration, libraries, environments, and networking can change startup behavior. Microsoft’s Spark monitoring guidance and session detail views help separate those causes.

A diagnosis sequence

  1. Establish a baseline. Pick a normal run with similar input and record its total duration and the measures below. Without a baseline, “slow” is an opinion.
  2. Separate startup from execution. Compare the activity start, the session-ready time and the first Spark job. Time before the first job is queue or session startup; see reducing notebook startup time.
  3. Check queueing and concurrency. Notebook jobs started by pipelines or the scheduler wait in a first-in, first-out queue when the capacity’s Spark cores are in use; interactive runs are rejected instead. If runs are slow only at certain times of day, look at what else shares the capacity before tuning code: the Fabric capacity saturation runbook covers that investigation.
  4. Check the read. Bytes and files read by the scan stages, whether filters reached the scan, and whether the table has many small files.
  5. Check shuffle and skew. Shuffle read and write per stage, spill, and the spread of task durations: one task far longer than the median means skew.
  6. Check transformation logic. Repeated actions over the same lineage, joins on unfiltered data, collect() or toPandas() on large results, Python UDFs in hot paths.
  7. Check the write. Number and size of output files, partition columns, append versus overwrite versus merge.
  8. Change one thing and rerun with comparable input.
  9. Compare with the baseline on duration and on correctness: row counts, duplicates, rejected rows.

What to measure

Measure Where to find it What it tells you
Total duration Pipeline run, Monitoring hub Whether the run is really slower than its baseline
Queue and session startup time Monitoring hub, session details Whether time is lost before your code runs
Duration per stage Spark UI (Stages), Spark application detail page Which phase dominates
Task duration, median vs max Stage summary metrics in the Spark UI Skew: one task holding a whole stage
Input bytes and files read Stage input metrics, the scan node in the plan Over-reading, missing pruning, small files
Shuffle read and write, spill Stage metrics Expensive wide operations, memory pressure
Number of tasks per stage Stage detail Too few (idle cores) or too many (scheduling overhead)
Output file count and size DESCRIBE DETAIL, table history Writes that leave small files for every later reader

Small files

Many tiny files make a Delta table slow to read and to maintain. Each file has to be listed and opened, adds an entry the Delta log must track, and usually becomes its own task, so a read that needs a handful of tasks schedules thousands of short ones. Merges and deletes also have more files to rewrite. The usual cause is on the write side: frequent small appends, streaming micro-batches, or an output partitioned on a high-cardinality column.

Check the average file size before choosing a fix:

PySpark
detail = spark.sql("DESCRIBE DETAIL SalesLakehouse.silver.orders").first()
avg_mb = detail.sizeInBytes / max(detail.numFiles, 1) / 1024**2
print({"files": detail.numFiles, "avg_file_mb": round(avg_mb, 1)})
PySpark3 linesRead-only example cell. Nothing on this page runs code.

An average of a few megabytes on a large table is a warning sign. Compact with OPTIMIZE, consider Optimize Write for frequent small writes, and batch appends where the pipeline allows it. Delta table maintenance: OPTIMIZE and VACUUM covers when and how to run them safely.

Practical recommendations

Begin with a timing breakdown. If most time occurs before the first Spark job, investigate session acquisition and libraries. If a stage dominates, inspect its jobs, tasks, input, shuffle, spill, skew, and output. If the write dominates, inspect partition counts, file sizes, write mode, and downstream table design.

Read only the required columns and time range. Put selective filters next to the read and use predicates aligned with the table’s physical design. Confirm in the executed plan that pruning or skipping occurred; source syntax alone does not prove it.

Reduce data before wide operations. Aggregate at the required grain before joining when that preserves correctness. Broadcast only genuinely small reference data after measuring it. Avoid collect() and toPandas() for large results because they move data to the driver. Use limited samples for inspection.

Cache only a DataFrame that is expensive to recompute and reused by multiple actions. Materialize it deliberately, verify storage pressure, and unpersist it when finished. Caching a one-use DataFrame adds work and memory pressure without saving recomputation.

For Delta output, choose append, overwrite, or merge from the data contract. Avoid repartitioning merely to produce a preferred file count: repartition causes a shuffle, while coalesce reduces partitions without a full redistribution but can create imbalance. Evaluate Optimize Write, compaction, V-Order, and Z-Order according to write frequency and consumers. Microsoft notes that V-Order can help cross-engine reads while adding Spark write cost, so it is not a universal notebook speed switch.

Example pattern

The example keeps projection and filtering close to the source, performs one reusable aggregation, and validates output without collecting full data.

PySpark
from pyspark.sql import functions as F
from pyspark.storagelevel import StorageLevel

run_date = "2026-10-02"

orders = (
    spark.read.table("SalesLakehouse.silver.orders")
    .select("order_id", "customer_id", "order_date", "status", "amount")
    .where(F.col("order_date") == F.lit(run_date))
    .where(F.col("status") == F.lit("Completed"))
)

daily_customer = (
    orders.groupBy("customer_id")
    .agg(
        F.countDistinct("order_id").alias("order_count"),
        F.sum("amount").alias("revenue"),
    )
    .persist(StorageLevel.MEMORY_AND_DISK)
)

output_rows = daily_customer.count()

(
    daily_customer.write
    .format("delta")
    .mode("append")
    .saveAsTable("SalesLakehouse.gold.daily_customer_sales")
)

print({"run_date": run_date, "output_rows": output_rows})
daily_customer.unpersist()
PySpark32 linesRead-only example cell. Nothing on this page runs code.

This is a pattern, not a prescription. If daily_customer is used only by the write, remove persistence and the separate count or replace validation with a metric produced elsewhere. If reruns are possible, a blind append may duplicate the date; the idempotent write design belongs to the pipeline contract.

Use explain("formatted") before and after an important change. Look for filters near the scan, unexpected exchanges, join strategies, and repeated branches. Then confirm with executed Spark jobs because estimates and plans do not show every runtime condition.

Common mistakes

  • Changing executor or shuffle settings before identifying the slow stage.
  • Calling count, display, and write on the same uncached expensive lineage.
  • Reading a complete history for an incremental run.
  • Selecting every column “just in case.”
  • Broadcasting a lookup based on row count without considering bytes.
  • Caching everything and creating eviction or spill pressure.
  • Repartitioning every output to one file, which destroys parallelism.
  • Treating session reuse as isolation-free; shared compute still needs compatible configuration and workload planning.
  • Comparing runs with different input volumes and calling the difference a regression.

Symptom, likely cause, first check

Symptom Likely cause First check
Long time before the first Spark job Queueing, or a slow session start (on-demand cluster, environment libraries) Session details: pool source and delay reason; queue time in monitoring
Fast startup, slow execution Read volume, wide operations or the write Stage durations in the Spark UI
One stage dominates A large scan, join or aggregation That stage’s input bytes, shuffle and spill
Thousands of very short tasks Small files or too many partitions numFiles from DESCRIBE DETAIL; tasks per stage
One task runs far longer than the others A skewed key in a join or aggregation Max vs median task duration; row counts per key
The write is slow Many output files or partitions; a merge into a large target Output file count; the merge condition and target size
Duration varies with the time of day Capacity contention or queueing Capacity utilization and queue time at those hours
Spark finishes quickly but the pipeline is late A downstream dependency or orchestration wait The pipeline activity timeline, not the Spark UI

Change one understood variable, rerun comparable input, and keep correctness controls alongside performance measures. A faster notebook that loses unmatched rows, duplicates a partition, or changes business grain is a failed optimization.

Sources

Fabric behavior checked against Microsoft Learn in November 2026.

Tags

  • Microsoft Fabric
  • Fabric Notebook
  • Performance
  • PySpark
  • Delta Lake
  • Troubleshooting