DevOps & Observability
Testing Microsoft Fabric Pipelines and Notebooks: A Practical Guide
How to test Microsoft Fabric pipelines and notebooks before production: testable design, parameter and expression tests, controlled failures, retries, idempotent reruns, partial-failure recovery, exit-value contracts, data assertions, CI-triggered runs, environment rebinding and post-deployment smoke tests.
On this page
Clicking Run in a test workspace and watching every activity turn green proves one thing: nothing threw an error. It doesn’t prove that the pipeline read the right file, wrote the right table, rejected the bad rows, will survive a retry, can be rerun tomorrow without duplicating orders, or will still point at the right Lakehouse after it is promoted. This guide answers one question:
How do I prove that a Fabric pipeline or notebook behaves correctly, fails safely, reruns safely, and produces trustworthy evidence before production deployment?
The principle that runs through all of it:
“Succeeded” is not the same thing as “correct.”
A pipeline run status describes the orchestration. Correctness is a property of the data the run produced, the data it refused, the state it left behind, and the evidence it recorded. Every test below inspects at least one of those, not just the status.
This guide connects pieces that already have their own pages, without repeating them: transformation logic is unit-tested as in PySpark DataFrame Unit Testing with pytest; the rules that decide whether a batch is published are in Data Quality Checks for Data Pipelines; parameter mechanics are in Microsoft Fabric Notebook Parameters; promotion between workspaces is the Git, deployment pipelines and database projects lab; access tests are the Fabric Security Lab. Where these pieces sit in the overall strategy is the subject of Testing a Production Data Platform.
The testing levels
Fabric testing happens at five levels, and most problems come from expecting one level to prove what only another can.
- Deployment validationAfter every promotion, in the target workspace
- Bindings point to this stage
- Smoke run
- Evidence recorded
- End-to-endSource to published table, in a test workspace
- Landing file to Silver
- Quality gate decides
- Pipeline integrationThe orchestration, with real activities
- Parameters and expressions
- Branches and failure paths
- Retries and reruns
- Notebook integrationThe notebook in a Fabric session
- Parameters cell
- Lakehouse reads and writes
- Exit value
- UnitLogic outside the orchestration, no tenant
- Transformations
- Parameter parsing
- Exit contract
- CI gate logic
| Level | What it proves | Needs |
|---|---|---|
| Unit | Transformation logic, parameter parsing, exit contract and CI decision logic are correct | Python, pytest, local Spark |
| Notebook integration | Parameters arrive as expected, the Fabric session can read and write the Lakehouse, failures fail and the exit value is correct | A test workspace and test Lakehouse |
| Pipeline integration | Expressions resolve, branches route correctly, dependencies and failure paths work, retries and reruns behave | A test workspace, test inputs |
| End-to-end | Landing file → pipeline → notebook → Silver → quality checks → publish works as one flow | Representative test data |
| Deployment validation | After promotion, everything points at this stage’s resources and still works | The target workspace, right after deployment |
The scenario
One pipeline, pl_orders_ingest, used throughout:
- A partner drops
orders.csvintoFiles/landing/<env>/orders/YYYY/MM/DD/in the Lakehouse. - Get Metadata checks that the file exists; if not, the pipeline takes its missing-file path.
- The notebook
nb_orders_loadvalidates its parameters, parses and stages the file, runs the data-quality checks, and publishes tosilver.orders(or holds the batch). - The pipeline advances the watermark only if the notebook reports a published batch.
- Every run leaves rows in
dq.check_result,dq.orders_quarantineand the execution log, tagged with its run ID.
Parameters:
| Parameter | Type | Example | Meaning |
|---|---|---|---|
run_date |
string (ISO date) | 2026-11-09 |
Business date of the batch |
source_path |
string | Files/landing/test/orders/2026/11/09/orders.csv |
The file to load |
target_table |
string | silver.orders |
Logical target; resolved per environment |
full_reload |
bool | false |
Rebuild the table instead of merging the day |
run_id |
string | the pipeline’s @pipeline().RunId |
Correlates every record the run writes |
What makes a pipeline testable
Testability is an architecture property, not something added after the pipeline is built.
| Design choice | What it makes testable |
|---|---|
| Explicit parameters, no hidden defaults that point at production | A test can run the same pipeline against test inputs |
Deterministic inputs: the run date is a parameter, not utcNow() deep inside an activity |
A test can replay a specific day |
| Transformation logic in functions, not inline in notebook cells | Unit tests without a tenant |
| A run ID on every record the run writes | Assertions can find exactly what this run did |
| A clear output contract: target table, schema, quarantine table, exit value | Assertions know what to check |
Explicit failure behaviour: Fail activities, raised exceptions, a held batch |
Failure tests have a defined expected outcome |
| Restartable steps: state (watermark, processed files) advanced only after success | Rerun tests have a defined expected outcome |
| Observable execution records | Evidence survives the run |
A pipeline built without these can still be tested, but only by clicking Run and looking at it. That’s the anti-pattern this guide replaces.
Notebook structure for testing
- Unit + integrationParameters cellParsed and validated by a function
- Unit testTransformation functionsDataFrame in, DataFrame out
- Integration testRead and write boundariesLakehouse, Delta, permissions
- Contract testExit valueSmall versioned JSON
The notebook itself becomes thin: a parameters cell, one call to a parsing function, calls to transformation and quality functions imported from a library (packaged as a wheel and attached through an environment, or kept in the notebook’s resources), the writes, and the exit. Everything with logic in it can be tested without Fabric.
Parameter testing
Parameters are input, and every input has valid, invalid, missing and boundary values. Write the matrix before writing the tests:
| Parameter | Valid | Invalid | Missing | Boundary | Environment-specific |
|---|---|---|---|---|---|
run_date |
2026-11-09 |
2026-13-01, a future date |
empty string | 2026-01-01, 2025-12-31, 2024-02-29 |
— |
source_path |
under this environment’s landing root, matching run_date |
another environment’s path; a folder for a different date | empty | first file of a month | Test must not read Prod’s landing folder |
full_reload |
true, false |
"no", null |
default | the string "false" |
— |
target_table |
silver.orders |
gold.revenue |
empty | — | Outside prod, resolves to silver.orders_<env> |
run_id |
the pipeline run ID | empty | interactive run with no ID | — | — |
Test four things for each parameter: the default (an interactive run with defaults must fail validation or do nothing harmful), valid values, invalid and missing values (rejected before any data is read, with a message an on-call engineer can act on), and environment-specific values (a value that is valid in one environment must be rejected in another).
Notebook parameter testing
Fabric notebooks receive parameters through a parameters cell: the cell toggled as the parameter cell holds defaults, and a run from a pipeline’s Notebook activity or the job API injects the supplied values after it. The Notebook activity can auto-populate its base parameters from that cell, with names, types and defaults (Notebook activity). Microsoft Fabric Notebook Parameters covers the mechanics; here, the point is how to test them.
Separate parsing from business logic. The parameters cell only assigns values; the next cell calls one function that validates all of them and returns a typed object. That function is plain Python, so every row of the matrix above becomes a fast unit test.
# orders_ingest/params.py
"""Parse and validate the orders notebook's parameters. Plain Python: testable without Fabric or Spark."""
import re
from dataclasses import dataclass
from datetime import date
LANDING_ROOT = {
"dev": "Files/landing/dev/orders/",
"test": "Files/landing/test/orders/",
"prod": "Files/landing/prod/orders/",
}
ALLOWED_TARGETS = {"silver.orders"} # outside prod, the notebook writes to silver.orders_<env>
RUN_ID = re.compile(r"^[A-Za-z0-9-]{8,64}$")
@dataclass(frozen=True)
class OrdersRun:
run_date: date
source_path: str
target_table: str
full_reload: bool
run_id: str
environment: str
def parse_bool(value) -> bool:
"""Accept real booleans and the strings 'true'/'false'. Reject anything else: bool('false') is True."""
if isinstance(value, bool):
return value
if isinstance(value, str) and value.strip().lower() in {"true", "false"}:
return value.strip().lower() == "true"
raise ValueError(f"full_reload must be true or false, got {value!r}")
def parse_params(run_date, source_path, target_table, full_reload, run_id, environment, today: date) -> OrdersRun:
"""Validate everything before any data is read. `today` is passed in, never read from the clock."""
errors = []
if environment not in LANDING_ROOT:
errors.append(f"environment must be one of {sorted(LANDING_ROOT)}")
try:
day = date.fromisoformat(run_date or "")
if day > today:
errors.append("run_date is in the future")
except ValueError:
day = None
errors.append("run_date must be an ISO date such as 2026-11-09")
root = LANDING_ROOT.get(environment)
if not source_path:
errors.append("source_path is required")
elif root and not source_path.startswith(root):
errors.append(f"source_path must be under {root} in {environment}")
elif day and day.strftime("%Y/%m/%d") not in source_path:
errors.append("source_path does not match run_date")
expected_target = target_table if environment == "prod" else f"{target_table}_{environment}"
if target_table not in ALLOWED_TARGETS:
errors.append(f"target_table must be one of {sorted(ALLOWED_TARGETS)}")
try:
reload_flag = parse_bool(full_reload)
except ValueError as error:
errors.append(str(error))
reload_flag = False
if not RUN_ID.fullmatch(run_id or ""):
errors.append("run_id is required (the pipeline run ID)")
if errors:
raise ValueError("Invalid parameters: " + "; ".join(errors))
return OrdersRun(day, source_path, expected_target, reload_flag, run_id, environment)
The tests turn the matrix into code:
@pytest.mark.parametrize(("field", "value", "message"), [
("run_date", "2026-13-01", "ISO date"),
("run_date", "", "ISO date"),
("run_date", "2026-11-11", "future"),
("source_path", "", "required"),
("source_path", "Files/landing/prod/orders/2026/11/09/orders.csv", "must be under"),
("source_path", "Files/landing/test/orders/2026/11/08/orders.csv", "does not match run_date"),
("target_table", "gold.revenue", "target_table"),
("full_reload", "no", "full_reload"),
("full_reload", None, "full_reload"),
("run_id", "", "run_id"),
("environment", "staging", "environment"),
])
def test_invalid_parameters_are_rejected_with_a_clear_reason(field, value, message):
with pytest.raises(ValueError, match=message):
params(**{field: value})
def test_the_truthy_string_trap():
assert bool("false") is True # why full_reload can't be checked with `if full_reload:`
assert parse_bool("false") is False
(params() builds a valid parameter set with one field overridden; TODAY is a fixed date.) Three details matter more than they look:
- All errors are reported at once. A run that fails on the first problem, then the second, then the third wastes three test cycles.
"false"is truthy. A parameter that arrives as a string (an interactive run, a hand-typed value, a changed type in the activity) turnsif full_reload:into a full rebuild of the table. The test pins that down.- The environment check is a parameter test. “Test reads Prod’s landing folder” is the most expensive parameter bug there is, and it’s one line of validation.
At notebook-integration level, run the notebook in the test workspace three ways: with valid values, with one invalid value (Expected result: the run fails in the validation cell, before any read, with the validation message in the error), and with no values at all (Expected result: the defaults fail validation; nothing is written).
Pipeline expression testing
A pipeline expression can be syntactically valid, evaluate without error, and send the run to the wrong place. That’s the failure mode to test for. Typical expressions in the scenario, using functions and system variables documented in the pipeline expression language:
Default run date @formatDateTime(addDays(pipeline().TriggerTime, -1), 'yyyy-MM-dd')
Dynamic file path @concat('Files/landing/', pipeline().libraryVariables.environment, '/orders/',
replace(pipeline().parameters.run_date, '-', '/'), '/orders.csv')
Target table @concat(pipeline().parameters.target_table,
if(equals(pipeline().libraryVariables.environment, 'prod'), '',
concat('_', pipeline().libraryVariables.environment)))
Branch @equals(pipeline().parameters.full_reload, true)
Run correlation @pipeline().RunId (passed to the notebook as run_id)
| Expression | What can go silently wrong |
|---|---|
| Default run date | TriggerTime is UTC. A run at 20:00 local time in a UTC−5 zone is already the next day in UTC, so “yesterday” is computed from the wrong day. A format string with dd and MM swapped produces a valid but wrong date |
| Dynamic file path | A missing slash, or the environment value from the wrong value set, gives a path that exists in another environment |
| Target table | Test writes to the production table name |
| Branch | A string "false" compared with a boolean routes to the wrong branch |
| Run correlation | Forgetting it leaves the run’s records unfindable |
pipeline().libraryVariables.environment stands for a variable library reference, added in the pipeline’s Library variables tab and inserted from the expression builder; insert it from the builder rather than typing it (variable libraries in pipelines).
How to test expressions:
- Make the resolved value visible. Pass computed paths and names into the notebook as parameters, and have the notebook log them to the execution record. A test then asserts on the resolved value, not on the expression text.
- Test the boundaries the expression handles: month-end, year-end, a run just after midnight, both values of every boolean.
- Assert the destination, not the status. After a test run, check that the file read and the table written are the expected ones for this environment. A run that wrote
silver.ordersfrom the Test workspace is green and wrong. - Treat every
If ConditionandSwitchas two or more tests, one per branch. Untested branches are where failure handling breaks.
Controlled failure testing
A pipeline’s failure paths are part of its design, and they need tests as much as the happy path does. Run each scenario deliberately, in the test workspace, with test inputs:
| Scenario | How to cause it | Expected result |
|---|---|---|
| Missing source file | Run with a run_date that has no file |
Get Metadata reports the file absent; the pipeline routes to a Fail activity with a specific error code and message, or to the designed no-input path; the watermark doesn’t move; the execution record says why |
| Invalid notebook parameter | full_reload = "no" |
The notebook fails in the validation cell, before reading anything; the activity fails; the error names the parameter |
| Schema mismatch | A test file with a renamed column | The transformation or the quality step rejects the batch; nothing is published; the evidence names the column |
| Permission problem | Run with an identity that lacks access to the test Lakehouse | The run fails with an authorization error in the activity output; no partial write |
| Downstream unavailable | Point a test connection at an unavailable endpoint | The configured retries run; then the activity fails; the failure path records it |
The Fail activity makes a designed failure explicit: it fails the run with a custom message and error code (activity overview), which a test (and an on-call engineer) can distinguish from an accident. Don’t assert on exact error text from Fabric itself; it changes. Assert on what your design controls: which activity failed, the error code you set, whether state moved, what the evidence says.
Retry testing
Retries exist for transient failures: throttling, a brief outage, a flapping dependency. They should never hide a deterministic defect.
Fabric activities have Retry (default 1 when enabled), Retry interval (default 30 seconds, fixed or increasing delay with exponential back-off) and Timeout (default 12 hours). Retry conditions (preview) restrict retries to failures whose error code, error message or failure type matches; without conditions, an activity retries on every failure (activity retries).
Two tests, two expected outcomes:
| Test | Cause | Expected result |
|---|---|---|
| Transient failure | The endpoint is briefly unavailable | First attempt fails, a retry succeeds; retryAttempt above zero is recorded |
| Deterministic defect | The file has an invalid schema | Every attempt fails the same way; with a retry condition, it fails once |
The second one is the important test. Retrying a bad file reads it again, parses it again and fails again, consuming capacity each time, and with a long retry interval it delays the alert by the interval multiplied by the retry count. Microsoft’s own example of a retry condition is retrying only on rate limiting (error code 429). Use retry conditions on Notebook and Copy activities so that deterministic failures (validation errors, schema errors, authorization failures) fail on the first attempt.
Two cautions from the documentation: the retry interval runs before the condition is evaluated, so a non-matching failure still waits one interval; and conditional retries are supported only for some activity types (Copy data, Notebook, Dataflow and Stored procedure among them). Test the behaviour on the activity types you use.
A notebook can help: raise a clearly distinguishable exception for validation and data errors, so a retry condition can exclude them. The activity runs API reports a retryAttempt per activity run; record it, because a run that succeeded on its third attempt isn’t as healthy as one that succeeded on its first (debugging retries).
Idempotency: run the same batch twice
This is the most valuable test a Fabric pipeline can have, because every pipeline will be retried, rerun after an incident, or replayed during recovery.
Run the same batch twice. The data must end up exactly as if it had run once.
Expected final state after the second run:
- No duplicate business rows:
silver.ordershas the same rows and the same row count as after the first run. - No duplicate side effects: the quarantine table doesn’t contain the same rejected rows twice; the file isn’t registered as processed twice; downstream notifications aren’t sent twice.
- Distinct execution records: two records, with different run IDs, both visible. The log mustn’t block a legitimate rerun.
- Correct state: the watermark is where it was after the first run.
What makes it pass is design, not luck: MERGE on the business key instead of append, deduplication inside the batch, processed-file tracking written after the load succeeds, quarantine rows keyed by run and source row, and watermark advancement only after the publish. Incremental Loads in Microsoft Fabric covers these patterns, and Recovery and Disaster Recovery Architecture for Microsoft Fabric explains why recovery depends on them. A Delta MERGE idempotency test at unit-test scale is in the PySpark testing guide; in Fabric, run it as a pipeline test. Testing Incremental Loads and Idempotency tests each of these guarantees in depth, with a tested PySpark suite.
Rerun from partial failure
The harder variant: the run fails halfway.
- Ingestion succeeds: the file is read and staged.
- Transformation and audit succeed: the batch is ready to publish.
- Publishing fails: for example, a permission change on the target, or a capacity interruption.
Then rerun. For each step, the design must say whether it reruns, skips, rebuilds or resumes:
- SucceededIngest file
- on rerunRebuild from the fileFile is unchanged and still in landing
- SucceededStage and audit
- on rerunRebuildDeterministic from the same input
- FailedPublish
- on rerunRerun, idempotentlyMERGE by business key
- Not advancedWatermark
- on rerunAdvance after publishOnly now, and only once
Two rules make partial reruns safe. State moves last: the watermark and the processed-file register are updated only after the publish succeeds, so a failure anywhere leaves them pointing at the start of the batch. Rebuild from inputs that haven’t moved: rebuilding staging from the same landing file is safe; rerunning only the failed activity, after a scheduled run has already advanced the watermark or replaced the file, is not. Fabric can rerun a pipeline from a failed activity, which is useful exactly when the inputs that activity depends on are unchanged (rerun debugging).
The test: force the publish step to fail once (for example, remove write permission on the test target for the first run), restore it, rerun the whole pipeline, and assert the idempotency outcomes above plus “the watermark moved exactly once”.
Notebook exit values: an explicit contract
A notebook can end with notebookutils.notebook.exit(value). The value is always a string; exit() ends the run as successful, so a failure must raise instead; and exit() doesn’t take effect inside a try/except (Microsoft Fabric Notebook Parameters, NotebookUtils notebook run). For runs started through the Job Scheduler API, the exit value is returned in the exitValue field of the job instance (notebook public APIs).
Orchestrators need an explicit signal, because “the notebook succeeded” covers both “published 10,000 rows” and “found an empty file and did nothing”. Make the exit value a small, versioned contract:
# orders_ingest/exit_contract.py
"""The notebook's exit value: a small, versioned JSON string the orchestrator can rely on."""
import json
STATUSES = {"published", "published_with_warnings", "held", "no_input"}
def build_exit(status: str, run_id: str, rows_read: int, rows_written: int, rows_quarantined: int) -> str:
if status not in STATUSES:
raise ValueError(f"unknown status {status!r}")
return json.dumps({
"contract": 1,
"status": status,
"run_id": run_id,
"rows_read": rows_read,
"rows_written": rows_written,
"rows_quarantined": rows_quarantined,
})
def parse_exit(value) -> dict:
"""Strict on what the orchestrator branches on, tolerant of fields it doesn't know yet."""
if not value:
raise ValueError("notebook returned no exit value")
payload = json.loads(value)
if payload.get("contract") != 1:
raise ValueError(f"unsupported exit contract {payload.get('contract')!r}")
if payload.get("status") not in STATUSES:
raise ValueError(f"unknown status {payload.get('status')!r}")
return payload
The last cell of the notebook calls notebookutils.notebook.exit(build_exit(...)), outside any try block. A held batch is a successful notebook run (the gate did its job) with an explicit status the pipeline can branch on.
Keep the contract boring: a few fields, a version number, no data, and a reader that ignores fields it doesn’t know (so the notebook can add one without breaking the pipeline). In the pipeline, read the exit value from the Notebook activity’s output and parse it with the json() expression function before branching on a field. Microsoft Learn doesn’t document the exact output path of the exit value in the activity output; copy it from the activity’s output JSON in a real run’s details rather than typing it from memory, and keep that one expression covered by a pipeline test.
Test data
The principles from the flagship’s test data strategy apply unchanged: small, deterministic, synthetic, privacy-safe, with edge cases included on purpose. For this scenario, a handful of versioned test files in the test workspace’s landing folder:
| File | Contents | Expected outcome |
|---|---|---|
2026/11/02/orders.csv |
Three normal orders | Published; 3 rows; quarantine empty |
2026/11/03/orders.csv |
The same order twice (identical rows) | Published; 1 row for that order |
2026/11/04/orders.csv |
One order with no customer | Published with warnings; 1 row quarantined with reason customer_id_present |
2026/11/05/orders.csv |
A malformed amount and a malformed date | Rows quarantined, or batch held if above the quarantine ratio |
2026/11/06/orders.csv |
An order dated 2026-11-01 (late-arriving) | Handled by the design: merged into the right business date, or quarantined, as specified |
2026/11/07/orders.csv |
Header row only | Exit status no_input; nothing published; watermark behaviour as designed |
2026/11/08/ |
No file | Missing-file path (see failure tests) |
Each date folder is one test case, so a test run is selected by run_date alone. Keep the files in Git next to the pipeline definition, and copy them into the test Lakehouse as part of test setup.
Assertions after a run
After each test run, assert on the data and the state, never only on the run status. Data Quality Checks for Data Pipelines defines the rule framework and the results table; the test assertions read what it wrote:
| Assertion | Example (illustrative, not executed) |
|---|---|
| The target exists and has the expected rows | SELECT COUNT(*) FROM silver.orders_test WHERE run_id = '<run_id>' equals the expected count for that test file |
| Critical keys are not null | SELECT COUNT(*) FROM silver.orders_test WHERE customer_id IS NULL is 0 |
| No duplicate business keys | SELECT order_id FROM silver.orders_test GROUP BY order_id HAVING COUNT(*) > 1 returns nothing |
| The run was recorded | execution_log has one row for the pipeline run ID, with status and row counts |
| Quality results were recorded | dq.check_result has a row per check for the run, and no failed error-level checks for a file that should pass |
| Rejected rows were captured | dq.orders_quarantine has exactly the expected rows and reasons for the test file |
| The watermark advanced correctly | It moved to run_date after a published batch, and didn’t move after a held batch or a failure |
Run these from the CI job through the test Lakehouse’s SQL analytics endpoint or a test notebook, and fail the CI step on the first mismatch.
Execution evidence
A test that leaves no evidence has to be rerun to be believed. Each test run should leave a record like this:
| Field | Meaning |
|---|---|
run_id |
The pipeline run ID (@pipeline().RunId) |
pipeline |
Pipeline name |
activity |
The failing or checked activity |
start_time, end_time |
Timing, for baselines |
status |
Outcome of the run, and of the batch (exit status) |
input |
Parameter summary (resolved paths and table names, no secrets) |
output |
Rows read, written, quarantined |
error |
Error code and message, when it failed |
environment |
dev, test or prod |
version |
The commit or release that was deployed |
The pipeline writes most of these into the execution log as part of its normal run, which is the point: the same evidence serves tests, monitoring and incident investigation. The CI job adds the version and the gate decision. For Fabric’s own view, the activity runs API returns each activity’s status, start and end, durationInMs, error and retryAttempt for a pipeline run (pipeline REST API).
The testing matrix
| What | Unit | Integration (test workspace) | Deployment validation | Production smoke |
|---|---|---|---|---|
| Transformation | pytest + local Spark | Output table on test files | — | Schema of the published table |
| Notebook parameters | Parsing function | Valid, invalid and empty runs | Defaults are harmless in this stage | — |
| Pipeline parameters | — | Each matrix row as a run | Values resolve for this stage | — |
| Expressions | — | Resolved paths and names asserted | Resolve to this stage’s resources | Resolved path logged |
| Storage access | — | Reads and writes on the test Lakehouse | Bindings point to this stage | Small read and write |
| Identity | — | Positive and negative runs | Workspace identity or connection works | Expected identity in the log |
| Data quality | Rule logic | Rules on test files | — | Checks run on the smoke batch |
| Idempotency | Batch deduplication | Same batch twice | — | — |
| Failure paths | Validation errors | Missing file, bad schema, permission | — | — |
| Environment configuration | Environment check in parsing | Variable library values | No Dev reference anywhere | — |
| Exit contract | Build and parse | Pipeline branches on it | — | — |
| Performance | — | Duration against baseline | — | Duration within baseline |
CI/CD integration
- Pull request
- Static validation
- Unit testsParsing, transforms, gate logic
- Deploy to Test
- Run Fabric test pipelineThrough the Job Scheduler API
- Data assertionsTables, quarantine, evidence
- Security checks
- Approval
- Deploy to Production
- Smoke testSmall, safe subset
The flagship covers the general gate model; here is the Fabric-specific part, the middle of the flow.
Automated Fabric execution
A CI job can start Fabric items with the Job Scheduler REST API and a service principal or managed identity that has a role on the test workspace:
- Deploy to the test workspace (deployment pipelines, Git-based deployment, or a tool such as fabric-cicd).
- Start the test run:
POST /v1/workspaces/{workspaceId}/items/{itemId}/jobs/{jobType}/instances(the job type isPipelinefor a pipeline,RunNotebookfor a notebook). The request is accepted with202and aLocationheader pointing at the job instance, plusRetry-After(Run on demand item job). Per-run parameters go in the request body; its format differs by item type, so take it from the item’s API reference. - Poll
GET …/jobs/instances/{jobInstanceId}until the status is terminal:Completed,Failed,CancelledorDeduped;NotStartedandInProgressmean keep waiting, and Microsoft notes that more statuses may be added (Get item job instance). For a notebook, the response includesexitValue. For a pipeline, query its activity runs to see each activity’s outcome. - Evaluate: run the data assertions and read the quality results for the run.
- Block or promote.
The polling and decision logic belongs in a small, unit-tested module, with the HTTP calls injected:
# orders_ingest/ci_gate.py
"""CI helpers: wait for a Fabric job instance, then decide whether the test run allows promotion.
The HTTP calls are injected (get_status), so this logic is unit-tested without a tenant.
"""
import time
TERMINAL = {"Completed", "Failed", "Cancelled", "Deduped"}
def wait_for_job(get_status, timeout_s: int, poll_s: int = 30, sleep=time.sleep, clock=time.monotonic) -> dict:
"""Poll a job instance until it reaches a terminal status or the timeout expires.
get_status() returns the Get Item Job Instance payload. Unknown statuses keep polling until the
timeout: the API documents that new statuses may be added.
"""
deadline = clock() + timeout_s
while True:
job = get_status()
if job.get("status") in TERMINAL:
return job
if clock() >= deadline:
raise TimeoutError(f"job still {job.get('status')!r} after {timeout_s}s")
sleep(poll_s)
def promotion_decision(job: dict, activity_runs: list[dict], dq_failed_errors: int, exit_payload: dict | None) -> tuple[bool, list[str]]:
"""Only a completed run, with every activity succeeded, no error-level data-quality failure and the
expected exit status, allows promotion. Everything else blocks, with reasons."""
reasons = []
if job.get("status") != "Completed":
reasons.append(f"job status is {job.get('status')!r}, not 'Completed'")
failed = [a["activityName"] for a in activity_runs if a.get("status") != "Succeeded"]
if failed:
reasons.append(f"activities not succeeded: {failed}")
if dq_failed_errors:
reasons.append(f"{dq_failed_errors} error-level data-quality checks failed")
if exit_payload is None or exit_payload.get("status") not in {"published", "published_with_warnings"}:
reasons.append("notebook did not report a published batch")
return (not reasons, reasons)
Three behaviours that the tests pin down, and that a hand-written polling loop usually gets wrong:
Dedupedis terminal, and it blocks. It means another instance of the same job was already running, so this test run was skipped. A gate that only checks “notFailed” would promote untested code.- The poll has a timeout. A job stuck in
InProgressfails the CI step instead of hanging it. - A green run can still block. The test
test_succeeded_is_not_correctgives the gate a completed job with every activity succeeded and an exit status ofheld, and expects promotion to be refused. Removing the data-quality and exit-status checks from the gate (I tried it, in a copy of the module) makes exactly that test and the data-quality test fail.
def test_succeeded_is_not_correct():
# Every activity green, but the notebook held the batch: nothing was published.
allowed, reasons = promotion_decision({"status": "Completed"}, ACTIVITIES, 0, {"contract": 1, "status": "held"})
assert not allowed and "published batch" in reasons[0]
Credentials: the CI job authenticates with a service principal or managed identity whose secret or federated credential lives in the CI system’s secret store, never in the repository, the pipeline definition or a notebook. Give that identity a role on the test workspace only; a separate identity, behind the approval, deploys to production.
Environment-specific testing
Promotion is where correct code turns into a wrong deployment. The usual mistakes:
- A notebook still bound to the Dev Lakehouse. Deployment pipelines auto-bind a notebook’s default Lakehouse and attached environment to the corresponding items in the target workspace when those items are in the same workspace as the notebook. A Lakehouse in another workspace keeps the original binding unless a deployment rule says otherwise (notebook source control and deployment).
- The wrong connection on a Copy or Notebook activity.
- The wrong active value set in the variable library, so Test reads Dev’s values.
- A hard-coded workspace or item ID in an expression or a notebook.
- A missing secret in the target environment’s Key Vault.
- A partial deployment: the deployment pipeline can continue when some items fail, leaving a stage that mixes versions.
Deployment rules and variable libraries
Two mechanisms handle environment differences, and both need tests:
- Deployment rules override values on the target stage: data source and parameter rules for some item types, and a default Lakehouse rule for notebooks, which takes priority over auto-binding. Rules are set on the target stage, not on Development, apply only on the next deployment, and stop working if the source item no longer has the referenced data source or parameter (create deployment rules).
- Variable libraries hold one value set per environment, with one value set active per workspace; pipelines, notebooks (through NotebookUtils), Dataflow Gen2 and other items read the active value at run time (variable library overview).
Test three things after every promotion: the expected value is the one used (the resolved path and table in the run’s evidence belong to this stage); a missing value fails loudly (a variable without a value for this stage must fail validation, not fall back to Dev’s); and there is no accidental Dev reference (search the deployed definitions and the run’s evidence for Dev workspace IDs, Dev Lakehouse names and Dev paths). The lifecycle lab walks through a notebook that still reads Dev after deployment, and fixes it with a deployment rule.
Environment rebinding checklist, after every promotion:
- Every notebook’s default Lakehouse belongs to this stage’s workspace
- Every notebook’s attached environment belongs to this stage
- Every connection on every activity points to this stage’s sources and targets
- The variable library’s active value set is this stage’s
- No Dev or Test workspace ID, item ID or path appears in deployed definitions or in the smoke run’s evidence
- Every secret the run needs resolves in this stage
- The deployment completed for every item; no item was left at the previous version
Security tests
The Fabric Security Lab covers how to prove access controls. For a pipeline release, a short test is enough:
- the intended identity (the workspace identity or the connection’s identity) can run the pipeline and read and write what it needs;
- an unintended identity can’t run it, and can’t read the target table’s restricted data;
- every secret resolves in the stage, with no fallback to a default;
- workspace and data permissions match the stage’s design (no Contributor left over from testing).
Performance smoke testing
A run that succeeds in four hours instead of twenty minutes is functionally successful and operationally broken. Capture pipeline duration, notebook duration and per-activity duration (durationInMs from the activity runs) for every test run, and compare them with a baseline from previous runs of the same test files on the same capacity. There’s no universal threshold; a large deviation from the baseline is the signal. Small regressions are usually a warning to investigate; a large one blocks or needs approval. Troubleshooting Microsoft Fabric Performance in Production covers diagnosing what you find.
Production smoke tests
After deploying to production, run a small, safe subset:
- a metadata check: the items exist, at the deployed version, bound to production resources;
- a small notebook run, or the pipeline on a dedicated smoke-test file that writes to a smoke table, not to
silver.orders; - the expected schema on the published table;
- the expected identity in the run’s evidence;
- the data-quality checks run and record results;
- the execution log has the smoke run’s record.
Smoke tests must be low risk: they mustn’t write business data, move a watermark or trigger downstream consumers. A smoke test that can corrupt production is worse than none. The full behaviour was proven in Test; production smoke tests prove the deployment.
What should block a deployment
Javi Builds guidance, not a Microsoft requirement:
| Failure | Recommendation |
|---|---|
| Notebook unit test fails | Block |
| Pipeline parameter or expression test fails | Block |
| Failure-path test behaves differently from design | Block |
| Rerun test produces duplicates or moves state twice | Block |
| Critical data-quality rule fails on a test file that should pass | Block |
| Test environment still references Dev | Block |
| Security test fails | Block |
Test run status Deduped, Cancelled or timed out |
Block (rerun the test; never treat it as a pass) |
| Small performance deviation from baseline | Investigate; policy-based |
| Large performance regression | Block or require explicit approval |
| Production smoke test fails | Stop the rollout; roll back or fix forward |
Common anti-patterns
| Anti-pattern | Instead |
|---|---|
| Testing only successful runs | A test per failure path: missing file, bad parameter, bad schema, permission |
| Clicking Run as validation | Triggered runs with assertions, from CI, with evidence |
| No test inputs | Versioned test files, one per case, selected by run_date |
| Pipelines tied to production resources | Parameters, variable libraries and rules; validation that rejects the wrong environment |
| One notebook with all the logic and all the I/O | Thin notebook; logic in tested functions |
| Retries hiding bad data | Retry conditions for transient errors only; deterministic failures fail once |
No run_id |
@pipeline().RunId on every record the run writes |
| Reruns create duplicates | MERGE, state moved last, the same-batch-twice test |
| No post-deployment validation | Rebinding checklist and a smoke run after every promotion |
| Tests that never inspect the data | Assertions on tables, quarantine, results and watermark |
| Comparing only activity status | “Succeeded” is not “correct”: gate on data, quality results and the exit contract |
Mini-lab: test the orders pipeline
Validation status: docs-based. This lab is written from current Microsoft Learn documentation and the patterns above. It hasn’t been run end to end in a Javi Builds Fabric test tenant; every outcome below is an Expected result, not an observed one.
Setup: a test workspace with a Lakehouse, the pl_orders_ingest pipeline and nb_orders_load notebook from your repository, the dq tables from Data Quality Checks for Data Pipelines (or their Delta equivalents), and the test files from the test data table copied into Files/landing/test/orders/.
- Create deterministic input. Copy the seven test folders into the landing area. Expected result: each date folder holds exactly the file described in the table; no other files.
- Run the valid case:
run_date = 2026-11-02. Expected result: the run completes; the exit value reportspublishedwith 3 rows written;silver.orders_testhas 3 rows with this run’srun_id; the quarantine table has none; the watermark is 2026-11-02. - Run an invalid parameter:
full_reload = "no". Expected result: the notebook activity fails in the validation cell; the error namesfull_reload; nothing is written; the watermark doesn’t move. - Introduce duplicate input:
run_date = 2026-11-03. Expected result: one row for the duplicated order; no duplicate keys insilver.orders_test. - Verify quarantine:
run_date = 2026-11-04. Expected result: exit statuspublished_with_warnings; one quarantined row with reasoncustomer_id_present; a failed row-level result indq.check_resultfor this run. - Rerun the same batch: run
2026-11-04again. Expected result:silver.orders_testand the quarantine table are unchanged; there are two execution records with different run IDs; the watermark hasn’t moved again. - Confirm idempotency with the assertions above: row count, duplicate keys, quarantine count and watermark, compared before and after the rerun.
- Inspect the evidence: the execution log, the quality results and the activity runs for each run ID. Expected result: for every run, you can say what it read, what it wrote, what it rejected, and why it succeeded or failed, without opening the notebook.
Then break it on purpose: remove write access on silver.orders_test for one run, rerun after restoring it, and check that the watermark moved exactly once.
Before promoting a Fabric pipeline
- Parameters tested (valid, invalid, missing, boundary, environment)
- Expressions tested, with resolved values asserted
- Empty input tested
- Bad schema tested
- Duplicate input tested
- Retry behaviour tested (transient recovers, deterministic fails once)
- Rerun and idempotency tested, including a partial failure
- Data-quality gates tested on files that must pass and files that must fail
- Identity and access tested
- Environment bindings verified after promotion
- Execution evidence recorded for every test run
- Performance compared with the baseline
- Post-deploy smoke test defined, and safe
What to read next
- Testing a Production Data Platform: the overall strategy and gate model.
- Data Quality Checks for Data Pipelines: the rules, quarantine and results these tests assert on.
- PySpark DataFrame Unit Testing with pytest: unit tests for the notebook’s transformations.
- Microsoft Fabric Notebook Parameters and Microsoft Fabric Invoke Pipeline Activity: parameter and orchestration mechanics.
- Deploy a Fabric Solution with Git, Deployment Pipelines and Database Projects and Microsoft Fabric CI/CD with fabric-cicd: the deployment side.
- Debugging Microsoft Fabric Pipelines and Notebooks in Production: what to do when a production run goes wrong anyway.
Sources
- Pipelines: Activity overview (general settings, timeout, Fail activity), Activity retries (retry interval types, retry conditions in preview), Expressions and functions (system variables such as
RunIdandTriggerTime), Notebook activity, Variable library integration with pipelines, REST API capabilities for pipelines (run, job instance, query activity runs) - Notebooks: Manage and execute notebooks with public APIs (
RunNotebook,exitValue), NotebookUtils notebook run, Notebook source control and deployment (auto-binding, default Lakehouse rule) - Job Scheduler: Run on demand item job, Get item job instance (status values)
- Lifecycle: Create deployment rules, Variable library overview
Fabric behaviour checked against Microsoft Learn in November 2026; several features referenced here (retry conditions, the new deployment pipeline interface) are in preview and change often. The Python modules shown were executed (36 tests, Python 3.12.10, pytest 9.2.0); the pipeline expressions, REST calls, assertion queries and mini-lab are docs-based. The testing levels, matrix, blocking table and checklists are Javi Builds guidance.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetDebug Fabric PipelinesTroubleshoot Microsoft Fabric pipeline failures by isolating the failed activity, separating transient from deterministic errors, validating parameters and identities, and using retries, logging, and controlled reruns safely.Open cheat sheet
- ResourceMicrosoft Fabric Certification Study PlannerAn adaptable 6–8 week example plan that connects official Fabric certification objectives with hands-on practice, review, and exam readiness.View resource
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.