DevOps & Observability

Monitoring Microsoft Fabric Pipelines and Notebooks in Production

How to monitor Fabric pipelines and notebooks well enough to catch failures, slowdowns, retries, capacity pressure and bad data before users do, with run correlation, execution logs, baselines and alerts.

By JaviPublished Updated 18 min readIntermediate

Fabric pipeline and notebook execution flow into operational monitoring

A green pipeline is the most misleading status in a data platform. It tells you that every activity returned without an error. It does not tell you that the right rows arrived, that the watermark moved, that the run took a normal amount of time, or that the 06:00 report will show yesterday’s numbers instead of last week’s.

Most Fabric teams start with the Monitoring hub and a failure email. That catches the loud problems: a broken connection, a notebook exception, a timeout. The expensive problems are quiet. A notebook that succeeds with zero rows because the source path changed. A load that takes 40 minutes instead of 10 because it now shares the capacity with a new workload. A retry that quietly appends the same window twice. Users find those, usually in a meeting.

This guide is about monitoring Fabric pipelines and notebooks well enough to find those problems first. It is organized around four questions every production run should answer:

  1. Did the workload run? Was it triggered, did it start, did it start on time?
  2. Did it finish successfully? Every activity, including retries, child pipelines and the notebooks they call.
  3. Did it produce the expected data? Row counts, keys, totals, freshness, watermarks.
  4. Did it consume an acceptable amount of time and capacity? Compared with its own history, not with a number someone guessed.

Fabric answers the first two well out of the box. The last two need a small amount of deliberate engineering: parameters, an execution log, validation checks and a baseline. That engineering is what this article covers.

Product behavior was checked against Microsoft Learn in October 2026. Several monitoring features, including the new Monitoring hub, were in preview at the time and are marked as such.

No single screen answers every question

Monitoring in Fabric comes in layers, and each answers different questions:

Layer What it gives you What it cannot tell you
Monitoring hub and pipeline run history Run status, duration, activity runs with inputs, outputs and errors, upstream and downstream run relationships, reruns Whether the data is correct; history beyond the platform’s retention; your business context
Spark application details (from a notebook run) Session, stages, executors, logs for one notebook run Which pipeline run and business window the notebook belonged to, unless you pass it in
Workspace monitoring Job and activity logs stored in an Eventhouse and queryable with KQL across the workspace Row counts, watermarks and validation results your code produces
Your execution log Business-level evidence: windows, row counts, watermarks, validation results, correlation across runs Platform internals; anything you did not log
Capacity Metrics app Which operations consumed capacity, smoothing, throttling Why a specific job did what it did
Alerting (pipeline alerts, Activator, Teams or Outlook activities, external tools) Notification when something crosses a line Anything you did not define a rule for

A production incident usually touches three of these. The point of the rest of this article is to make them connect.

What to monitor

Signal Why it matters Where to get it What should alert
Pipeline run status The first sign something broke Monitoring hub, workspace monitoring, pipeline alerts Failure of a production pipeline
Missed or late start A schedule that silently did not fire looks like “no failures” Expected-run list vs actual runs; execution log Expected run absent past its window
Pipeline duration Slowdowns precede failures and missed SLAs Run history, workspace monitoring, execution log Duration far outside the workload’s own baseline
Failed activity and error Tells you where to look first Activity runs in the run detail view Included in the failure alert
Retry count Hidden instability; retries can also duplicate work Activity runs; retry_count in the execution log Repeated retries, or retries growing over time
Parameters (date, window, environment) Most “wrong data” incidents are wrong inputs Activity input; execution log Parameters outside expected values
Upstream and downstream runs A child failure surfaces as a vague parent error Upstream/downstream run columns in the Monitoring hub Part of the failure investigation
Notebook exit status and exception The notebook knows why it failed; the pipeline only knows that it did Notebook run, Spark logs, execution log Notebook failure in a production run
Rows read, written and rejected Zero or unusual counts are the most common silent failure Execution log (from Delta operation metrics) Zero rows where data is expected; counts far from baseline
Source window and watermark Incremental correctness depends on it Execution log, watermark table Data written but watermark not advanced, or the reverse
Duplicate keys and nulls Silent data corruption Validation step Any duplicate business key; null rate above an agreed limit
Reconciliation totals Proves source and target agree Validation step Difference above an agreed tolerance
Freshness Users care when data was last correct, not when a job last ran Max business timestamp per table; execution log Data older than the agreed freshness
Files discovered, processed, failed File ingestion fails file by file Processed-file table; execution log Failed or unprocessed files past their window
Startup or queue delay Spark capacity or session issues, not code Spark application details; startup vs processing split in the log Startup consistently far above normal
Capacity pressure Throttling slows or rejects everything on the capacity Capacity Metrics app; capacity events Throttling affecting a critical workload

There are deliberately no thresholds in the table. A “normal” duration for a nightly load and for a five-minute micro-batch differ by orders of magnitude. Thresholds come from each workload’s own history, as described in the section on duration baselines.

Built-in pipeline monitoring

Fabric Data Factory gives you a solid baseline without any custom work.

Monitoring hub. At the time of writing there are two experiences: the established Monitoring hub, which Microsoft now calls the legacy experience, and a new Monitoring hub in preview that adds cross-workspace pipeline health views and alert management. Both let you:

  • Find pipeline runs by name, status, start time, submitter and location.
  • Open a run’s detail view: the pipeline diagram, run ID, status, errors and every activity run.
  • Open an activity’s input and output to see the parameters it received and what it returned, and its performance details (including a duration breakdown for Copy activities).
  • See upstream and downstream runs, meaning which run triggered this one and which runs it triggered. Turn on those columns to follow an Invoke pipeline chain.
  • Use the Gantt view to compare durations across runs and spot overlaps.
  • Rerun a pipeline: the whole run, from the failed activity, or from a selected activity.
  • Export activity runs to CSV.

Workspace monitoring. When enabled for a workspace, Fabric writes job execution logs into a monitoring Eventhouse, including an ItemJobEventLogs table with item name, job status, start and end times and duration. That turns run history into something you can query, chart and alert on across every pipeline in the workspace:

// Pipeline duration per day over two weeks, from workspace monitoring
ItemJobEventLogs
| where ItemKind == "Data Pipeline" and JobStatus == "Completed"
| where JobStartTime > ago(14d)
| summarize runs = count(),
            p50_minutes = percentile(DurationMs, 50) / 60000.0,
            p90_minutes = percentile(DurationMs, 90) / 60000.0
          by ItemName, bin(JobStartTime, 1d)
| order by ItemName asc, JobStartTime asc

Check the exact column values in your own monitoring Eventhouse before building on them; the schema is documented by Microsoft, but item kinds and statuses are easy to mistype.

What the built-in layer does not do is know your data. It can tell you that nb_silver_orders succeeded in four minutes. It cannot tell you that it merged zero rows because the window parameter was a day early.

Notebook monitoring: capture more than “succeeded”

A notebook activity’s status is binary. The questions you will ask during an incident are not:

  • Which run was this? Notebook run ID, plus the pipeline run that called it.
  • What was it asked to do? Business date, source window, environment, target table.
  • What did it actually do? Rows read, written, rejected; tables touched; output location.
  • How long did each part take? Waiting and startup vs processing vs the final write.
  • If it failed, why? The exception type and message, sanitized.

Fabric can tell you some of this. Spark application details show stages, executors and logs. notebookutils.runtime.context reports the notebook name, workspace, default Lakehouse, whether the run is for a pipeline, and the Livy job ID of the current activity. What the notebook does not know by itself is which pipeline run started it: the runtime context only carries run IDs for notebook reference runs. So pass the pipeline’s identity in.

In the Notebook activity’s Base parameters:

Parameter Value
pipeline_run_id @pipeline().RunId
parent_pipeline_run_id @pipeline()?.TriggeredByPipelineRunId (null unless invoked by a parent pipeline)
pipeline_name @pipeline().PipelineName
trigger_time @pipeline().TriggerTime
window_start, window_end the processing window computed by the pipeline

The notebook declares them in its parameter cell with safe defaults for interactive runs. Notebook parameters covers the pattern, including validation, and Invoke Pipeline covers passing values from parent to child pipelines.

Use the execution log for what Fabric cannot see

The execution log is the table your code writes: one row per attempt, written at start and finalized at the end. It is where monitoring answers questions three and four. This article uses that model’s field names rather than inventing new ones:

Field Monitoring use
execution_id One per attempt; a retry gets a new one
parent_execution_id The calling pipeline run, so a notebook row can be tied to its pipeline
correlation_id The business operation across services, such as one daily load
pipeline_name, notebook_name, environment What ran, and where
start_time, end_time, duration_ms Duration and baselines
status started, succeeded, failed, cancelled, timed_out
rows_read, rows_written Volume and silent zero-row runs
retry_count, trigger_type Retries, and scheduled vs manual reruns
error_code, error_message Sanitized failure reason

Two extensions are worth adding for monitoring: window_start and window_end for incremental workloads, and rows_rejected for validation. The Incremental Loads article adds rows_inserted, rows_updated and rows_deleted from the Delta MERGE metrics.

A compact notebook wrapper that writes it:

PySpark
import time, uuid
from datetime import datetime, timezone

ctx = notebookutils.runtime.context
execution_id = str(uuid.uuid4())   # one per execution, even if the notebook runs twice in a pipeline run
started = time.time()

def log(status, **metrics):
    row = {
        "execution_id": execution_id,
        "parent_execution_id": pipeline_run_id,          # notebook parameter from @pipeline().RunId
        "correlation_id": correlation_id,                 # notebook parameter, e.g. "sales-daily-2026-10-24"
        "pipeline_name": pipeline_name,
        "notebook_name": ctx["currentNotebookName"],
        "environment": environment,
        "status": status,
        "start_time": datetime.fromtimestamp(started, timezone.utc),
        "end_time": datetime.now(timezone.utc) if status != "started" else None,
        "duration_ms": int((time.time() - started) * 1000) if status != "started" else None,
        "window_start": window_start,
        "window_end": window_end,
        **metrics,
    }
    spark.createDataFrame([row]).write.format("delta").mode("append").option("mergeSchema", "true").saveAsTable("execution_log")

log("started")
try:
    rows_read, rows_written = run_load(window_start, window_end)      # your transformation
    log("succeeded", rows_read=rows_read, rows_written=rows_written)
except Exception as e:
    log("failed", error_code=type(e).__name__, error_message=str(e)[:500])
    raise
PySpark32 linesRead-only example cell. Nothing on this page runs code.

Re-raise the exception after logging. A notebook that logs a failure and then exits cleanly turns a red activity into a green one.

Whatever stores it (a Lakehouse table as above, a Warehouse table or a SQL database), keep the log independent of the platform’s run-history retention, so you can compare this month with last quarter.

Correlate parent and child runs

Real workloads are trees, not single runs:

pl_daily_sales                     pipeline run  9f1c…   (correlation_id = sales-daily-2026-10-24)
├── Invoke pl_load_orders          child run     47ab…   parent = 9f1c…
│   ├── Copy job activity          copy run
│   └── nb_silver_orders           notebook run          parent_execution_id = 47ab…
└── nb_validate_sales              notebook run          parent_execution_id = 9f1c…

When nb_silver_orders fails, the parent pipeline only reports that the invoked pipeline failed. Without correlation, an operator opens the parent, clicks into the child, finds the notebook run, opens its Spark logs, and then guesses which window it was processing. With correlation, one query returns the whole tree:

SQL query: Correlate parent and child runsSpark SQL
-- Everything that happened for one business operation
SELECT pipeline_name, notebook_name, status, start_time, duration_ms,
       rows_read, rows_written, retry_count, error_message
FROM execution_log
WHERE correlation_id = 'sales-daily-2026-10-24'
ORDER BY start_time;
Spark SQL6 linesRead-only query display. It is not connected to a database.

The rules are simple: every pipeline passes @pipeline().RunId (and the parent’s run ID when invoked) down to what it calls; every notebook logs both; and one correlation_id identifies the business operation across reruns. The Monitoring hub’s upstream and downstream columns give the same tree for the platform’s side, so you can move between the two.

Monitoring incremental loads

Incremental workloads add state, and state adds failure modes that run status cannot see. The Incremental Loads article covers the design. For monitoring, log these per run:

  • Previous watermark and the upper bound captured for this run.
  • Rows extracted from the window.
  • Rows inserted, updated, deleted by the MERGE.
  • Watermark advanced? Did last_successful_watermark move to the upper bound?
  • Reconciliation status of the most recent check.

The two conditions worth an alert:

  • Data written, watermark not advanced. The run committed Silver or Gold but failed before persisting the watermark. With an idempotent design, the next run replays the window safely, but you should know, and repeated occurrences mean something is failing at the end of every run.
  • Watermark advanced, downstream write failed or missing. The dangerous one. Changes in that window may be permanently skipped. It should be impossible by design (the watermark moves last), so if it happens, treat it as an incident.

A simple check compares the watermark table with the execution log:

SQL query: Monitoring incremental loadsSpark SQL
-- Watermark advanced without a succeeded run for that window
SELECT w.pipeline_name, w.last_successful_watermark, w.last_execution_id
FROM etl_watermark AS w
LEFT JOIN execution_log AS l
  ON l.execution_id = w.last_execution_id AND l.status = 'succeeded'
WHERE l.execution_id IS NULL;
Spark SQL6 linesRead-only query display. It is not connected to a database.

Also watch watermark lag: how far last_successful_watermark trails the current time. A watermark that stops moving while every run reports success usually means an empty or wrong window.

Monitoring retries and reruns

Retries hide instability. An activity configured with retries can fail twice, succeed on the third attempt, and leave the pipeline green. That is fine once. It is a warning when it happens every night, and an incident when retries start duplicating work.

Distinguish four cases:

  • Transient retry: one retry after a network blip or a brief source timeout. Log it, do not alert.
  • Retry storm: many retries across activities or runs, often after a source outage. Each retry can restart heavy work and add capacity load. Alert on growth, not on single occurrences.
  • Successful rerun after partial failure: an operator or schedule reruns a failed window. Correct when writes are idempotent, but make it visible: trigger_type and a new execution_id show it was a rerun, and correlation_id ties it to the original attempt.
  • Rerun from the failed activity: Fabric can resume from the failed activity. That is useful, but only safe if the activities before it are not expected to run again for the same window.

Keep failed attempts in the log; never overwrite them when a retry succeeds. “Succeeded after three retries” and “succeeded” are different health states, and only the log can tell them apart afterwards.

Duration baselines

Fixed duration thresholds (“alert if over 30 minutes”) are either too tight for heavy days or too loose for light ones. Compare each run with its own history instead:

  • The same workload, not a platform-wide average.
  • Recent history: the last two to four weeks, so the baseline follows gradual growth.
  • Similar input volume where it varies a lot. Monday’s catch-up window is not comparable with Tuesday’s.
  • Percentiles (median and a high percentile) rather than averages, which a single bad night distorts.

For example (illustrative only), a load whose recent runs take 8 to 12 minutes and which today takes 35 is worth investigating, even though 35 minutes would be unremarkable for another workload. The workspace monitoring query earlier gives you the platform view; the same calculation over execution_log.duration_ms gives you per-notebook and per-window baselines.

Look at trends, too. A run that gets a little slower every week is often a table accumulating small files or a source growing faster than expected. Delta table maintenance covers the file-layout side. Re-check baselines after deployments, because a CI/CD release is the most common reason a stable duration changes overnight.

Capacity-aware monitoring

A workload can slow down without any change to its code or data, because it shares capacity with everything else. Common causes:

  • Overlapping workloads at the same hour.
  • Smoothing: background work from earlier in the day still weighing on the capacity.
  • Throttling: interactive delays or rejections once smoothed usage exceeds the limits.
  • Spark concurrency limits: jobs queued (pipeline-triggered) or rejected (interactive) when Spark cores are in use.

The diagnostic path is short:

  1. Inspect the workload’s history. Is this run an outlier against its baseline? Did startup or processing time grow?
  2. Inspect the capacity window in the Capacity Metrics app for the same period. Was there throttling? Which operations dominated?
  3. Correlate with other operations. Which other pipelines, notebooks, refreshes or queries ran at the same time?

If the capacity was under pressure and the workload itself did not change, rescheduling or isolating it is usually more effective than tuning it. Microsoft Fabric Capacity Explained covers smoothing, throttling stages and how to read the Capacity Metrics app; this article does not repeat them.

Separate startup from processing time

When a notebook’s total duration grows but its logic did not change, split the time into three parts:

  • Waiting and startup: queueing, Spark session acquisition, environment and library setup.
  • Processing: reading, transforming, shuffling.
  • Write and commit: writing files and committing the Delta transaction, plus any maintenance triggered after the write.

Each points somewhere different. Growing startup time points to session configuration, environments, libraries or capacity, which Notebook startup time and session tags address. Growing processing time points to data volume, skew, scans or joins; see Make Fabric notebooks faster. Growing write time points to file layout, MERGE scope or compaction.

A cheap way to get the split is to log timestamps inside the notebook: when the first cell runs (end of startup), before the final write, and after the commit. The difference between the pipeline activity’s start and the first cell is waiting and startup; the rest is yours.

Compare like with like: a cold session and a reused session answer different questions, and so do different input windows.

Data-quality monitoring

Infrastructure success is not data success. A pipeline can be green and still deliver bad data, so validation has to be part of the run, not a separate report someone looks at later.

Practical checks, run as a validation step after the load:

SQL query: Data-quality monitoringSpark SQL
-- Duplicates on the business key
SELECT COUNT(*) - COUNT(DISTINCT sales_order_id) AS duplicate_rows FROM silver_sales_orders;

-- Null rate on a required column
SELECT AVG(CASE WHEN customer_id IS NULL THEN 1.0 ELSE 0.0 END) AS null_rate FROM silver_sales_orders;

-- Freshness: newest business timestamp
SELECT MAX(order_timestamp) AS newest_order FROM silver_sales_orders;
Spark SQL8 linesRead-only query display. It is not connected to a database.

And the checks that need context:

  • Row-count deltas: today’s rows against the recent baseline for the same weekday.
  • Missing keys: keys present in the source but not in the target, or the reverse, for a recent range.
  • Totals and reconciliation: sums per date in source and target.
  • Expected partitions or dates: every expected date present in Gold.
  • Late files: expected inputs that have not arrived.
  • Stale data: the newest timestamp older than the agreed freshness, even if the job ran.

Decide what each failed check does. Some should fail the run (duplicate keys in a current-state table), so the pipeline’s failure path and alerts fire. Others should warn (a row count 20% below baseline on a holiday). Record every result in the execution log, so trends are visible.

Monitoring file ingestion

File-based loads fail file by file, so monitor them at that grain:

  • Files discovered in the landing area this run.
  • Files processed, skipped and failed, with the reason for skips and failures.
  • Duplicate files: same name or checksum already processed.
  • Processing lag: the oldest unprocessed file’s age.

A processed-file table, as described in the incremental loads guide, gives you these counts for free. Alert on failed files and on lag, not on individual skips.

Alerts

Alert on what someone must act on; trend-track the rest.

Immediate alerts:

  • Failure of a production pipeline.
  • Repeated notebook failure for the same workload.
  • A data validation failure that blocks downstream use.
  • Watermark advanced without a successful downstream write.
  • Capacity throttling affecting a critical workload.
  • An expected run that did not start.

Warnings and trends:

  • Duration degrading against baseline.
  • Growing retry counts.
  • Growing small-file counts on key tables.
  • Gradual row-count drift.
  • Rising startup times.

What Fabric provides natively, as of October 2026:

  • Activity-level alerts: add an Outlook or Teams activity on an activity’s failure path.
  • Pipeline-level alerts in the new Monitoring hub (preview), and failure notifications on scheduled pipeline runs.
  • Activator rules on pipeline job events, for success or failure notifications by email or Teams.
  • Workspace-level alerts: a KQL query over workspace monitoring’s ItemJobEventLogs, with an Activator rule on top.
  • Capacity events in the Real-Time hub, for capacity state changes.

What usually needs your own work: alerts on execution-log data (row counts, watermarks, validation), duration-vs-baseline rules, and routing to an on-call system outside Microsoft 365. A validation notebook that fails the pipeline is the simplest bridge: it turns a data problem into a failure the native alerts already catch.

Every alert needs a receiver and a first action. An alert that nobody owns trains people to ignore the channel.

A production troubleshooting workflow

  1. 1Alert or user complaintWhat is wrong, for which data, since when?
  2. 2Find the affected runMonitoring hub, or correlation_id in the execution log
  3. 3Failure or slowdown?Red run, green-but-wrong, or green-but-slow
  4. 4Check activity run detailsFailed activity, input parameters, error, child runs
  5. 5Check the execution logWindow, rows, retries, watermark
  6. 6Check data validationDuplicates, totals, freshness
  7. 7Check the capacity windowThrottling, overlapping workloads
  8. 8Compare with duration baselineStartup vs processing vs write
  9. 9Identify the cause
    • Source
    • Code
    • Capacity
    • Data
  10. 10Rerun safelySame window, idempotent writes
  11. 11Verify downstream stateValidation passes, watermark correct, reports fresh
Investigating a Fabric pipeline or notebook incident. Evidence in a fixed order. Most incidents are explained by step 5, 6 or 7, but checking capacity and the baseline prevents fixing the wrong thing.

Two steps are skipped most often. Checking capacity before tuning code, and verifying downstream state after the rerun. A rerun that succeeds but leaves Gold or the semantic model stale has not closed the incident. Debugging Microsoft Fabric Pipelines and Notebooks in Production works through each step for failed, slow and wrong-result runs.

The Fabric monitoring cheat sheet and the pipeline debugging cheat sheet collect the commands and places to look for each step.

Illustrative example: the daily load that took 42 minutes

This is an illustrative troubleshooting walk-through, not a measured incident.

The daily sales pipeline normally completes in about 11 minutes. One morning it takes 42. It is green, the report is late, and someone asks whether the notebook needs optimizing.

  1. Find the run. The Monitoring hub shows the run succeeded; the Gantt view shows it started on time and ran long.
  2. Check activity details. The Copy job activity took its usual time. The Silver notebook activity accounts for almost all of the extra half hour.
  3. Check the execution log. Same window size as usual, normal row counts, no retries, validation passed. The data is fine; only the time is wrong.
  4. Split startup from processing. Startup was normal. Processing took about four times its baseline.
  5. Check capacity. The Capacity Metrics app shows the capacity running hot during the same window, with a newly scheduled heavy backfill notebook in another workspace on the same capacity overlapping the load.
  6. Check for changes. No deployment, no code change, no data growth.
  7. Decide. The cause is contention, not the notebook. Moving the backfill to a quieter window restores the normal duration the next day.

Without the execution log and the capacity check, the likely response would have been to rewrite a notebook that was not the problem.

Common monitoring mistakes

  • Alerting only on failure. Slow, empty and wrong runs are green.
  • Not storing run IDs. Without them, notebook logs cannot be tied to pipeline runs.
  • No parent-child correlation. Every child failure becomes a manual click-through.
  • No row-count logging. Zero-row runs go unnoticed for days.
  • No reconciliation. Missed deletes and skipped windows surface only when someone compares totals by hand.
  • Treating a green pipeline as delivered data. Status is not correctness.
  • No duration baseline. Without history, “slow” is an opinion.
  • Ignoring retries. They hide instability and can duplicate work.
  • Tuning notebooks without checking capacity. Contention looks exactly like slow code from inside the notebook.
  • Logging everything and using none of it. Logs without queries, dashboards or alerts are storage cost, not monitoring.

Production checklist

  • Run status monitored, with alerts on production pipeline failures.
  • Missed runs detected: expected schedules compared with actual runs.
  • Duration compared with each workload’s own baseline, not fixed thresholds.
  • Retries logged and trended; failed attempts kept.
  • Run IDs passed from pipelines to notebooks and stored.
  • Parent-child correlation through parent run IDs and a correlation_id.
  • Parameters (date, window, environment) recorded per run.
  • Watermarks logged, with alerts on mismatches between watermark and writes.
  • Rows read, written and rejected logged for every load.
  • Validation (duplicates, nulls, totals, freshness) part of the run, with clear fail vs warn rules.
  • Capacity checked during investigations, with capacity events for critical workloads.
  • Alerting owned: every alert has a receiver and a first action.
  • Rerun safety confirmed: idempotent writes, and reruns visible in the log.

For a wider checklist across APIs, SQL and data SLAs, see the data platform monitoring checklist. To decide which tool should own each step in the first place, and how that affects where evidence lives, see Pipelines vs Notebooks vs Dataflow Gen2 vs Copy Job.

Tags

  • Microsoft Fabric
  • Fabric Notebook
  • Data Pipelines
  • Observability
  • Monitoring
  • Performance