Microsoft Fabric

How to Reduce Fabric Notebook Startup Time

Diagnose Microsoft Fabric Spark session acquisition and reduce avoidable notebook startup overhead through compatible pools, environments and reuse.

By JaviPublished Updated 9 min readIntermediate

Startup clock progressing into an active Spark session and running notebook

Short answer: the time before your notebook code runs has two different causes, and a third phase is often confused with them. Queue time is waiting for capacity. Session startup is Fabric creating or attaching a Spark session, including environment libraries. Code execution starts with your first cell. Measure which one grew before changing anything: only session startup is fixed by the pool, environment and reuse settings in this article.

  1. Activity startsPipeline, schedule or interactive run
  2. CapacityQueue timePipeline and scheduled jobs wait while Spark cores are in use
  3. This articleSession startupStarter pool or on-demand cluster, environment, libraries
  4. Your notebookCode executionFirst cell to last cell
Queue, startup and execution are different clocks. Only the middle phase is session startup. Queue time is a capacity problem; slow code execution is a data or code problem.

Problem

A notebook’s Spark code completes in under a minute, but the pipeline activity takes much longer. Some runs acquire a session quickly while others wait. Increasing executors will not fix time spent before Spark jobs begin, so first isolate session acquisition from execution.

Queue time, session startup and code execution

Queue time

When the capacity’s Spark cores are fully used, notebook jobs started by pipelines or the scheduler, and Spark job definitions, are queued first in, first out and retried automatically. Queue entries expire 24 hours after admission. Interactive notebook runs and notebook jobs submitted through the public API are not queued: they fail with TooManyRequestsForCapacity (HTTP 430). Pool and library settings cannot shorten queue time; capacity, scheduling and competing workloads can. If queueing is routine, follow the Fabric capacity saturation runbook and size from measured utilization with the Fabric Capacity Planner.

Session startup

  • Starter pools are pre-warmed and managed by Microsoft. Without extra libraries or custom Spark properties, a session typically starts in 5 to 10 seconds, but this is best effort: pre-warmed capacity is not guaranteed, and high regional demand can fall back to an on-demand cluster.
  • Custom compute (node size, custom pools, some networking configurations) can prevent use of the pre-warmed path and start an on-demand cluster instead.
  • Environment libraries depend on the publishing mode: in Full mode the library snapshot is deployed at session start, typically adding 1 to 3 minutes; in Quick mode libraries install when the first code cell runs, so the delay shows up as slow “code”.
  • Custom live pools keep dedicated clusters warm during a schedule you define, with environment libraries preinstalled, so sessions start in about 5 seconds during that window. Warm compute consumes capacity; reserve it for latency-sensitive jobs.

Session start insights in the session details shows whether a session came from a starter pool or an on-demand cluster, and why the fast path was not used.

Code execution

Execution starts at the first cell. Inline %pip install time lands here, as does a Quick-mode library install. Anything after that (reads, shuffles, writes) belongs to making the notebook itself faster.

Warm sessions, expiry and reuse

An interactive session stays warm until it expires; the default expiry is 20 minutes, adjustable in the workspace Spark settings. The next run after expiry pays startup again.

Pipelines can reuse a running session instead of starting a new one per notebook. With high concurrency enabled for pipelines, compatible notebook activities that share a session tag join one session; by default up to five notebooks share a session, and the limit can be raised to 50.

Watch for unnecessary notebook chaining. A pipeline that calls ten short notebooks as ten separate activities can pay queue and startup time ten times. Group compatible activities with one session tag, merge notebooks that are always run together, or call them from one orchestrating notebook: Microsoft documents that a notebook started with notebook.run() (NotebookUtils, formerly MSSparkUtils) runs on the Spark pool of the calling notebook. runMultiple() runs several in parallel and is marked Preview in that documentation.

Practical recommendations

Measure startup separately. Record the pipeline activity start, session-ready time when available, first Spark job start, and notebook completion. Use the session details or monitoring experience to identify the source and delay reason. Without that split, a slow first data read may be mislabeled as startup.

For small data, also question whether the work needs Spark: a pure Python notebook with Polars or DuckDB may be the better fit, and a single-node Spark session from the starter pool starts about as fast. See pandas vs Polars vs DuckDB vs PySpark in Microsoft Fabric.

Prefer the default starter-pool-compatible configuration when it meets the workload. Do not customize node size or Spark properties preemptively. A custom configuration can be correct for a sustained heavy job, but it may exchange a faster execution stage for longer acquisition. Evaluate total duration and reliability.

Move stable, shared dependencies into a governed Fabric Environment rather than running %pip install in every notebook. Choose environment publishing behavior according to development speed, reproducibility, and session needs. Avoid importing heavy packages that the notebook never uses. Package work is part of execution cost even when it is not visible as a Spark stage.

Example measurement pattern

Notebook code cannot measure time before its own process begins, but it can mark the earliest executable point and each subsequent stage. Pair these records with pipeline and Fabric session timestamps.

PySpark
from datetime import datetime, timezone
from time import perf_counter

notebook_started_at = datetime.now(timezone.utc).isoformat()
clock = perf_counter()

print({"event": "notebook_code_started", "at": notebook_started_at})

orders = (
    spark.read.table("SalesLakehouse.silver.orders")
    .select("order_id", "order_date", "amount")
    .where("order_date >= '2026-10-01'")
)

row_count = orders.count()
print({
    "event": "input_materialized",
    "elapsed_seconds": round(perf_counter() - clock, 3),
    "rows": row_count,
})
PySpark20 linesRead-only example cell. Nothing on this page runs code.

If pipeline-to-code time is high but code-to-first-action time is normal, investigate acquisition and environment initialization. If the notebook starts quickly but the first action is slow, inspect table access, file listing, plan creation, and the Spark job instead.

Create a small comparison table across several representative runs:

Run Session source Acquisition First action Remaining work
Baseline Starter or reused measured measured measured
Slow run On-demand or customized measured measured measured

Do not invent target numbers. Establish a baseline for the workspace, capacity, configuration, and time window. Regional demand and concurrent capacity use can change results.

Common mistakes

  • Tuning Spark transformations when the delay occurs before the first job.
  • Assuming starter-pool startup is guaranteed on every run.
  • Adding custom compute or properties without measuring their acquisition cost.
  • Installing the same packages inline in every notebook.
  • Treating a live pool as free latency reduction without considering capacity use.
  • Using unique session tags that prevent related pipeline activities from reusing a session.
  • Forcing incompatible workloads together and losing reliability or isolation.
  • Measuring one run and presenting it as a stable benchmark.

Startup checklist

  • Was the notebook queued? Check queue time before anything else.
  • Was a new session created, or did it attach to a warm or shared one?
  • Did the session come from a starter pool or an on-demand cluster, and what reason does session start insights give?
  • Did the environment change or get republished, and which library publishing mode does it use?
  • Are custom libraries installed inline with %pip in every run?
  • Is session reuse available: high concurrency enabled, compatible settings, a shared session tag?
  • Are many short notebooks invoked as separate activities?
  • Is concurrency on the capacity the real bottleneck?

If acquisition is normal, move the investigation into the first Spark action and the executed plan. Startup tuning ends when code begins; it should not become a convenient explanation for slow reads, shuffles or Delta writes.

Sources

Timings are Microsoft’s documented typical values, checked in November 2026; measure your own baseline.

Tags

  • Microsoft Fabric
  • Fabric Notebook
  • Performance
  • PySpark
  • Troubleshooting