DevOps & Observability
Monitoring Microsoft Fabric Pipelines and Notebooks in Production
How to monitor Fabric pipelines and notebooks well enough to catch failures, slowdowns, retries, capacity pressure and bad data before users do, with run correlation, execution logs, baselines and alerts.
On this page
A green pipeline is the most misleading status in a data platform. It tells you that every activity returned without an error. It does not tell you that the right rows arrived, that the watermark moved, that the run took a normal amount of time, or that the 06:00 report will show yesterday’s numbers instead of last week’s.
Most Fabric teams start with the Monitoring hub and a failure email. That catches the loud problems: a broken connection, a notebook exception, a timeout. The expensive problems are quiet. A notebook that succeeds with zero rows because the source path changed. A load that takes 40 minutes instead of 10 because it now shares the capacity with a new workload. A retry that quietly appends the same window twice. Users find those, usually in a meeting.
This guide is about monitoring Fabric pipelines and notebooks well enough to find those problems first. It is organized around four questions every production run should answer:
- Did the workload run? Was it triggered, did it start, did it start on time?
- Did it finish successfully? Every activity, including retries, child pipelines and the notebooks they call.
- Did it produce the expected data? Row counts, keys, totals, freshness, watermarks.
- Did it consume an acceptable amount of time and capacity? Compared with its own history, not with a number someone guessed.
Fabric answers the first two well out of the box. The last two need a small amount of deliberate engineering: parameters, an execution log, validation checks and a baseline. That engineering is what this article covers.
Product behavior was checked against Microsoft Learn in October 2026. Several monitoring features, including the new Monitoring hub, were in preview at the time and are marked as such.
No single screen answers every question
Monitoring in Fabric comes in layers, and each answers different questions:
| Layer | What it gives you | What it cannot tell you |
|---|---|---|
| Monitoring hub and pipeline run history | Run status, duration, activity runs with inputs, outputs and errors, upstream and downstream run relationships, reruns | Whether the data is correct; history beyond the platform’s retention; your business context |
| Spark application details (from a notebook run) | Session, stages, executors, logs for one notebook run | Which pipeline run and business window the notebook belonged to, unless you pass it in |
| Workspace monitoring | Job and activity logs stored in an Eventhouse and queryable with KQL across the workspace | Row counts, watermarks and validation results your code produces |
| Your execution log | Business-level evidence: windows, row counts, watermarks, validation results, correlation across runs | Platform internals; anything you did not log |
| Capacity Metrics app | Which operations consumed capacity, smoothing, throttling | Why a specific job did what it did |
| Alerting (pipeline alerts, Activator, Teams or Outlook activities, external tools) | Notification when something crosses a line | Anything you did not define a rule for |
A production incident usually touches three of these. The point of the rest of this article is to make them connect.
What to monitor
| Signal | Why it matters | Where to get it | What should alert |
|---|---|---|---|
| Pipeline run status | The first sign something broke | Monitoring hub, workspace monitoring, pipeline alerts | Failure of a production pipeline |
| Missed or late start | A schedule that silently did not fire looks like “no failures” | Expected-run list vs actual runs; execution log | Expected run absent past its window |
| Pipeline duration | Slowdowns precede failures and missed SLAs | Run history, workspace monitoring, execution log | Duration far outside the workload’s own baseline |
| Failed activity and error | Tells you where to look first | Activity runs in the run detail view | Included in the failure alert |
| Retry count | Hidden instability; retries can also duplicate work | Activity runs; retry_count in the execution log |
Repeated retries, or retries growing over time |
| Parameters (date, window, environment) | Most “wrong data” incidents are wrong inputs | Activity input; execution log | Parameters outside expected values |
| Upstream and downstream runs | A child failure surfaces as a vague parent error | Upstream/downstream run columns in the Monitoring hub | Part of the failure investigation |
| Notebook exit status and exception | The notebook knows why it failed; the pipeline only knows that it did | Notebook run, Spark logs, execution log | Notebook failure in a production run |
| Rows read, written and rejected | Zero or unusual counts are the most common silent failure | Execution log (from Delta operation metrics) | Zero rows where data is expected; counts far from baseline |
| Source window and watermark | Incremental correctness depends on it | Execution log, watermark table | Data written but watermark not advanced, or the reverse |
| Duplicate keys and nulls | Silent data corruption | Validation step | Any duplicate business key; null rate above an agreed limit |
| Reconciliation totals | Proves source and target agree | Validation step | Difference above an agreed tolerance |
| Freshness | Users care when data was last correct, not when a job last ran | Max business timestamp per table; execution log | Data older than the agreed freshness |
| Files discovered, processed, failed | File ingestion fails file by file | Processed-file table; execution log | Failed or unprocessed files past their window |
| Startup or queue delay | Spark capacity or session issues, not code | Spark application details; startup vs processing split in the log | Startup consistently far above normal |
| Capacity pressure | Throttling slows or rejects everything on the capacity | Capacity Metrics app; capacity events | Throttling affecting a critical workload |
There are deliberately no thresholds in the table. A “normal” duration for a nightly load and for a five-minute micro-batch differ by orders of magnitude. Thresholds come from each workload’s own history, as described in the section on duration baselines.
Built-in pipeline monitoring
Fabric Data Factory gives you a solid baseline without any custom work.
Monitoring hub. At the time of writing there are two experiences: the established Monitoring hub, which Microsoft now calls the legacy experience, and a new Monitoring hub in preview that adds cross-workspace pipeline health views and alert management. Both let you:
- Find pipeline runs by name, status, start time, submitter and location.
- Open a run’s detail view: the pipeline diagram, run ID, status, errors and every activity run.
- Open an activity’s input and output to see the parameters it received and what it returned, and its performance details (including a duration breakdown for Copy activities).
- See upstream and downstream runs, meaning which run triggered this one and which runs it triggered. Turn on those columns to follow an Invoke pipeline chain.
- Use the Gantt view to compare durations across runs and spot overlaps.
- Rerun a pipeline: the whole run, from the failed activity, or from a selected activity.
- Export activity runs to CSV.
Workspace monitoring. When enabled for a workspace, Fabric writes job execution logs into a monitoring Eventhouse, including an ItemJobEventLogs table with item name, job status, start and end times and duration. That turns run history into something you can query, chart and alert on across every pipeline in the workspace:
// Pipeline duration per day over two weeks, from workspace monitoring
ItemJobEventLogs
| where ItemKind == "Data Pipeline" and JobStatus == "Completed"
| where JobStartTime > ago(14d)
| summarize runs = count(),
p50_minutes = percentile(DurationMs, 50) / 60000.0,
p90_minutes = percentile(DurationMs, 90) / 60000.0
by ItemName, bin(JobStartTime, 1d)
| order by ItemName asc, JobStartTime asc
Check the exact column values in your own monitoring Eventhouse before building on them; the schema is documented by Microsoft, but item kinds and statuses are easy to mistype.
What the built-in layer does not do is know your data. It can tell you that nb_silver_orders succeeded in four minutes. It cannot tell you that it merged zero rows because the window parameter was a day early.
Notebook monitoring: capture more than “succeeded”
A notebook activity’s status is binary. The questions you will ask during an incident are not:
- Which run was this? Notebook run ID, plus the pipeline run that called it.
- What was it asked to do? Business date, source window, environment, target table.
- What did it actually do? Rows read, written, rejected; tables touched; output location.
- How long did each part take? Waiting and startup vs processing vs the final write.
- If it failed, why? The exception type and message, sanitized.
Fabric can tell you some of this. Spark application details show stages, executors and logs. notebookutils.runtime.context reports the notebook name, workspace, default Lakehouse, whether the run is for a pipeline, and the Livy job ID of the current activity. What the notebook does not know by itself is which pipeline run started it: the runtime context only carries run IDs for notebook reference runs. So pass the pipeline’s identity in.
In the Notebook activity’s Base parameters:
| Parameter | Value |
|---|---|
pipeline_run_id |
@pipeline().RunId |
parent_pipeline_run_id |
@pipeline()?.TriggeredByPipelineRunId (null unless invoked by a parent pipeline) |
pipeline_name |
@pipeline().PipelineName |
trigger_time |
@pipeline().TriggerTime |
window_start, window_end |
the processing window computed by the pipeline |
The notebook declares them in its parameter cell with safe defaults for interactive runs. Notebook parameters covers the pattern, including validation, and Invoke Pipeline covers passing values from parent to child pipelines.
Use the execution log for what Fabric cannot see
The execution log is the table your code writes: one row per attempt, written at start and finalized at the end. It is where monitoring answers questions three and four. This article uses that model’s field names rather than inventing new ones:
| Field | Monitoring use |
|---|---|
execution_id |
One per attempt; a retry gets a new one |
parent_execution_id |
The calling pipeline run, so a notebook row can be tied to its pipeline |
correlation_id |
The business operation across services, such as one daily load |
pipeline_name, notebook_name, environment |
What ran, and where |
start_time, end_time, duration_ms |
Duration and baselines |
status |
started, succeeded, failed, cancelled, timed_out |
rows_read, rows_written |
Volume and silent zero-row runs |
retry_count, trigger_type |
Retries, and scheduled vs manual reruns |
error_code, error_message |
Sanitized failure reason |
Two extensions are worth adding for monitoring: window_start and window_end for incremental workloads, and rows_rejected for validation. The Incremental Loads article adds rows_inserted, rows_updated and rows_deleted from the Delta MERGE metrics.
A compact notebook wrapper that writes it:
import time, uuid
from datetime import datetime, timezone
ctx = notebookutils.runtime.context
execution_id = str(uuid.uuid4()) # one per execution, even if the notebook runs twice in a pipeline run
started = time.time()
def log(status, **metrics):
row = {
"execution_id": execution_id,
"parent_execution_id": pipeline_run_id, # notebook parameter from @pipeline().RunId
"correlation_id": correlation_id, # notebook parameter, e.g. "sales-daily-2026-10-24"
"pipeline_name": pipeline_name,
"notebook_name": ctx["currentNotebookName"],
"environment": environment,
"status": status,
"start_time": datetime.fromtimestamp(started, timezone.utc),
"end_time": datetime.now(timezone.utc) if status != "started" else None,
"duration_ms": int((time.time() - started) * 1000) if status != "started" else None,
"window_start": window_start,
"window_end": window_end,
**metrics,
}
spark.createDataFrame([row]).write.format("delta").mode("append").option("mergeSchema", "true").saveAsTable("execution_log")
log("started")
try:
rows_read, rows_written = run_load(window_start, window_end) # your transformation
log("succeeded", rows_read=rows_read, rows_written=rows_written)
except Exception as e:
log("failed", error_code=type(e).__name__, error_message=str(e)[:500])
raiseRe-raise the exception after logging. A notebook that logs a failure and then exits cleanly turns a red activity into a green one.
Whatever stores it (a Lakehouse table as above, a Warehouse table or a SQL database), keep the log independent of the platform’s run-history retention, so you can compare this month with last quarter.
Correlate parent and child runs
Real workloads are trees, not single runs:
pl_daily_sales pipeline run 9f1c… (correlation_id = sales-daily-2026-10-24)
├── Invoke pl_load_orders child run 47ab… parent = 9f1c…
│ ├── Copy job activity copy run
│ └── nb_silver_orders notebook run parent_execution_id = 47ab…
└── nb_validate_sales notebook run parent_execution_id = 9f1c…
When nb_silver_orders fails, the parent pipeline only reports that the invoked pipeline failed. Without correlation, an operator opens the parent, clicks into the child, finds the notebook run, opens its Spark logs, and then guesses which window it was processing. With correlation, one query returns the whole tree:
-- Everything that happened for one business operation
SELECT pipeline_name, notebook_name, status, start_time, duration_ms,
rows_read, rows_written, retry_count, error_message
FROM execution_log
WHERE correlation_id = 'sales-daily-2026-10-24'
ORDER BY start_time;The rules are simple: every pipeline passes @pipeline().RunId (and the parent’s run ID when invoked) down to what it calls; every notebook logs both; and one correlation_id identifies the business operation across reruns. The Monitoring hub’s upstream and downstream columns give the same tree for the platform’s side, so you can move between the two.
Monitoring incremental loads
Incremental workloads add state, and state adds failure modes that run status cannot see. The Incremental Loads article covers the design. For monitoring, log these per run:
- Previous watermark and the upper bound captured for this run.
- Rows extracted from the window.
- Rows inserted, updated, deleted by the MERGE.
- Watermark advanced? Did
last_successful_watermarkmove to the upper bound? - Reconciliation status of the most recent check.
The two conditions worth an alert:
- Data written, watermark not advanced. The run committed Silver or Gold but failed before persisting the watermark. With an idempotent design, the next run replays the window safely, but you should know, and repeated occurrences mean something is failing at the end of every run.
- Watermark advanced, downstream write failed or missing. The dangerous one. Changes in that window may be permanently skipped. It should be impossible by design (the watermark moves last), so if it happens, treat it as an incident.
A simple check compares the watermark table with the execution log:
-- Watermark advanced without a succeeded run for that window
SELECT w.pipeline_name, w.last_successful_watermark, w.last_execution_id
FROM etl_watermark AS w
LEFT JOIN execution_log AS l
ON l.execution_id = w.last_execution_id AND l.status = 'succeeded'
WHERE l.execution_id IS NULL;Also watch watermark lag: how far last_successful_watermark trails the current time. A watermark that stops moving while every run reports success usually means an empty or wrong window.
Monitoring retries and reruns
Retries hide instability. An activity configured with retries can fail twice, succeed on the third attempt, and leave the pipeline green. That is fine once. It is a warning when it happens every night, and an incident when retries start duplicating work.
Distinguish four cases:
- Transient retry: one retry after a network blip or a brief source timeout. Log it, do not alert.
- Retry storm: many retries across activities or runs, often after a source outage. Each retry can restart heavy work and add capacity load. Alert on growth, not on single occurrences.
- Successful rerun after partial failure: an operator or schedule reruns a failed window. Correct when writes are idempotent, but make it visible:
trigger_typeand a newexecution_idshow it was a rerun, andcorrelation_idties it to the original attempt. - Rerun from the failed activity: Fabric can resume from the failed activity. That is useful, but only safe if the activities before it are not expected to run again for the same window.
Keep failed attempts in the log; never overwrite them when a retry succeeds. “Succeeded after three retries” and “succeeded” are different health states, and only the log can tell them apart afterwards.
Duration baselines
Fixed duration thresholds (“alert if over 30 minutes”) are either too tight for heavy days or too loose for light ones. Compare each run with its own history instead:
- The same workload, not a platform-wide average.
- Recent history: the last two to four weeks, so the baseline follows gradual growth.
- Similar input volume where it varies a lot. Monday’s catch-up window is not comparable with Tuesday’s.
- Percentiles (median and a high percentile) rather than averages, which a single bad night distorts.
For example (illustrative only), a load whose recent runs take 8 to 12 minutes and which today takes 35 is worth investigating, even though 35 minutes would be unremarkable for another workload. The workspace monitoring query earlier gives you the platform view; the same calculation over execution_log.duration_ms gives you per-notebook and per-window baselines.
Look at trends, too. A run that gets a little slower every week is often a table accumulating small files or a source growing faster than expected. Delta table maintenance covers the file-layout side. Re-check baselines after deployments, because a CI/CD release is the most common reason a stable duration changes overnight.
Capacity-aware monitoring
A workload can slow down without any change to its code or data, because it shares capacity with everything else. Common causes:
- Overlapping workloads at the same hour.
- Smoothing: background work from earlier in the day still weighing on the capacity.
- Throttling: interactive delays or rejections once smoothed usage exceeds the limits.
- Spark concurrency limits: jobs queued (pipeline-triggered) or rejected (interactive) when Spark cores are in use.
The diagnostic path is short:
- Inspect the workload’s history. Is this run an outlier against its baseline? Did startup or processing time grow?
- Inspect the capacity window in the Capacity Metrics app for the same period. Was there throttling? Which operations dominated?
- Correlate with other operations. Which other pipelines, notebooks, refreshes or queries ran at the same time?
If the capacity was under pressure and the workload itself did not change, rescheduling or isolating it is usually more effective than tuning it. Microsoft Fabric Capacity Explained covers smoothing, throttling stages and how to read the Capacity Metrics app; this article does not repeat them.
Separate startup from processing time
When a notebook’s total duration grows but its logic did not change, split the time into three parts:
- Waiting and startup: queueing, Spark session acquisition, environment and library setup.
- Processing: reading, transforming, shuffling.
- Write and commit: writing files and committing the Delta transaction, plus any maintenance triggered after the write.
Each points somewhere different. Growing startup time points to session configuration, environments, libraries or capacity, which Notebook startup time and session tags address. Growing processing time points to data volume, skew, scans or joins; see Make Fabric notebooks faster. Growing write time points to file layout, MERGE scope or compaction.
A cheap way to get the split is to log timestamps inside the notebook: when the first cell runs (end of startup), before the final write, and after the commit. The difference between the pipeline activity’s start and the first cell is waiting and startup; the rest is yours.
Compare like with like: a cold session and a reused session answer different questions, and so do different input windows.
Data-quality monitoring
Infrastructure success is not data success. A pipeline can be green and still deliver bad data, so validation has to be part of the run, not a separate report someone looks at later.
Practical checks, run as a validation step after the load:
-- Duplicates on the business key
SELECT COUNT(*) - COUNT(DISTINCT sales_order_id) AS duplicate_rows FROM silver_sales_orders;
-- Null rate on a required column
SELECT AVG(CASE WHEN customer_id IS NULL THEN 1.0 ELSE 0.0 END) AS null_rate FROM silver_sales_orders;
-- Freshness: newest business timestamp
SELECT MAX(order_timestamp) AS newest_order FROM silver_sales_orders;And the checks that need context:
- Row-count deltas: today’s rows against the recent baseline for the same weekday.
- Missing keys: keys present in the source but not in the target, or the reverse, for a recent range.
- Totals and reconciliation: sums per date in source and target.
- Expected partitions or dates: every expected date present in Gold.
- Late files: expected inputs that have not arrived.
- Stale data: the newest timestamp older than the agreed freshness, even if the job ran.
Decide what each failed check does. Some should fail the run (duplicate keys in a current-state table), so the pipeline’s failure path and alerts fire. Others should warn (a row count 20% below baseline on a holiday). Record every result in the execution log, so trends are visible.
Monitoring file ingestion
File-based loads fail file by file, so monitor them at that grain:
- Files discovered in the landing area this run.
- Files processed, skipped and failed, with the reason for skips and failures.
- Duplicate files: same name or checksum already processed.
- Processing lag: the oldest unprocessed file’s age.
A processed-file table, as described in the incremental loads guide, gives you these counts for free. Alert on failed files and on lag, not on individual skips.
Alerts
Alert on what someone must act on; trend-track the rest.
Immediate alerts:
- Failure of a production pipeline.
- Repeated notebook failure for the same workload.
- A data validation failure that blocks downstream use.
- Watermark advanced without a successful downstream write.
- Capacity throttling affecting a critical workload.
- An expected run that did not start.
Warnings and trends:
- Duration degrading against baseline.
- Growing retry counts.
- Growing small-file counts on key tables.
- Gradual row-count drift.
- Rising startup times.
What Fabric provides natively, as of October 2026:
- Activity-level alerts: add an Outlook or Teams activity on an activity’s failure path.
- Pipeline-level alerts in the new Monitoring hub (preview), and failure notifications on scheduled pipeline runs.
- Activator rules on pipeline job events, for success or failure notifications by email or Teams.
- Workspace-level alerts: a KQL query over workspace monitoring’s
ItemJobEventLogs, with an Activator rule on top. - Capacity events in the Real-Time hub, for capacity state changes.
What usually needs your own work: alerts on execution-log data (row counts, watermarks, validation), duration-vs-baseline rules, and routing to an on-call system outside Microsoft 365. A validation notebook that fails the pipeline is the simplest bridge: it turns a data problem into a failure the native alerts already catch.
Every alert needs a receiver and a first action. An alert that nobody owns trains people to ignore the channel.
A production troubleshooting workflow
- 1Alert or user complaintWhat is wrong, for which data, since when?
- 2Find the affected runMonitoring hub, or correlation_id in the execution log
- 3Failure or slowdown?Red run, green-but-wrong, or green-but-slow
- 4Check activity run detailsFailed activity, input parameters, error, child runs
- 5Check the execution logWindow, rows, retries, watermark
- 6Check data validationDuplicates, totals, freshness
- 7Check the capacity windowThrottling, overlapping workloads
- 8Compare with duration baselineStartup vs processing vs write
- 9Identify the cause
- Source
- Code
- Capacity
- Data
- 10Rerun safelySame window, idempotent writes
- 11Verify downstream stateValidation passes, watermark correct, reports fresh
Two steps are skipped most often. Checking capacity before tuning code, and verifying downstream state after the rerun. A rerun that succeeds but leaves Gold or the semantic model stale has not closed the incident. Debugging Microsoft Fabric Pipelines and Notebooks in Production works through each step for failed, slow and wrong-result runs.
The Fabric monitoring cheat sheet and the pipeline debugging cheat sheet collect the commands and places to look for each step.
Illustrative example: the daily load that took 42 minutes
This is an illustrative troubleshooting walk-through, not a measured incident.
The daily sales pipeline normally completes in about 11 minutes. One morning it takes 42. It is green, the report is late, and someone asks whether the notebook needs optimizing.
- Find the run. The Monitoring hub shows the run succeeded; the Gantt view shows it started on time and ran long.
- Check activity details. The Copy job activity took its usual time. The Silver notebook activity accounts for almost all of the extra half hour.
- Check the execution log. Same window size as usual, normal row counts, no retries, validation passed. The data is fine; only the time is wrong.
- Split startup from processing. Startup was normal. Processing took about four times its baseline.
- Check capacity. The Capacity Metrics app shows the capacity running hot during the same window, with a newly scheduled heavy backfill notebook in another workspace on the same capacity overlapping the load.
- Check for changes. No deployment, no code change, no data growth.
- Decide. The cause is contention, not the notebook. Moving the backfill to a quieter window restores the normal duration the next day.
Without the execution log and the capacity check, the likely response would have been to rewrite a notebook that was not the problem.
Common monitoring mistakes
- Alerting only on failure. Slow, empty and wrong runs are green.
- Not storing run IDs. Without them, notebook logs cannot be tied to pipeline runs.
- No parent-child correlation. Every child failure becomes a manual click-through.
- No row-count logging. Zero-row runs go unnoticed for days.
- No reconciliation. Missed deletes and skipped windows surface only when someone compares totals by hand.
- Treating a green pipeline as delivered data. Status is not correctness.
- No duration baseline. Without history, “slow” is an opinion.
- Ignoring retries. They hide instability and can duplicate work.
- Tuning notebooks without checking capacity. Contention looks exactly like slow code from inside the notebook.
- Logging everything and using none of it. Logs without queries, dashboards or alerts are storage cost, not monitoring.
Production checklist
- Run status monitored, with alerts on production pipeline failures.
- Missed runs detected: expected schedules compared with actual runs.
- Duration compared with each workload’s own baseline, not fixed thresholds.
- Retries logged and trended; failed attempts kept.
- Run IDs passed from pipelines to notebooks and stored.
- Parent-child correlation through parent run IDs and a
correlation_id. - Parameters (date, window, environment) recorded per run.
- Watermarks logged, with alerts on mismatches between watermark and writes.
- Rows read, written and rejected logged for every load.
- Validation (duplicates, nulls, totals, freshness) part of the run, with clear fail vs warn rules.
- Capacity checked during investigations, with capacity events for critical workloads.
- Alerting owned: every alert has a receiver and a first action.
- Rerun safety confirmed: idempotent writes, and reruns visible in the log.
For a wider checklist across APIs, SQL and data SLAs, see the data platform monitoring checklist. To decide which tool should own each step in the first place, and how that affects where evidence lives, see Pipelines vs Notebooks vs Dataflow Gen2 vs Copy Job.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetMonitor Fabric Pipelines and NotebooksMonitor Microsoft Fabric pipelines and notebooks using Monitoring hub, Workspace Monitoring, Spark monitoring, KQL, alerts, and capacity telemetry so failures and performance regressions become observable.Open cheat sheet
- ArticleMonitoring a Data Platform: What Should You Actually Measure?A practical monitoring model connecting clients, applications, APIs, report engines, data platforms, compute and storage.Read article
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.