DevOps et observabilité

Testing Microsoft Fabric Pipelines and Notebooks: A Practical Guide

How to test Microsoft Fabric pipelines and notebooks before production: testable design, parameter and expression tests, controlled failures, retries, idempotent reruns, partial-failure recovery, exit-value contracts, data assertions, CI-triggered runs, environment rebinding and post-deployment smoke tests.

Par JaviPublié 30 min de lectureAvancé

Clicking Run in a test workspace and watching every activity turn green proves one thing: nothing threw an error. It doesn’t prove that the pipeline read the right file, wrote the right table, rejected the bad rows, will survive a retry, can be rerun tomorrow without duplicating orders, or will still point at the right Lakehouse after it is promoted. This guide answers one question:

How do I prove that a Fabric pipeline or notebook behaves correctly, fails safely, reruns safely, and produces trustworthy evidence before production deployment?

The principle that runs through all of it:

“Succeeded” is not the same thing as “correct.”

A pipeline run status describes the orchestration. Correctness is a property of the data the run produced, the data it refused, the state it left behind, and the evidence it recorded. Every test below inspects at least one of those, not just the status.

This guide connects pieces that already have their own pages, without repeating them: transformation logic is unit-tested as in PySpark DataFrame Unit Testing with pytest; the rules that decide whether a batch is published are in Data Quality Checks for Data Pipelines; parameter mechanics are in Microsoft Fabric Notebook Parameters; promotion between workspaces is the Git, deployment pipelines and database projects lab; access tests are the Fabric Security Lab. Where these pieces sit in the overall strategy is the subject of Testing a Production Data Platform.

The testing levels

Fabric testing happens at five levels, and most problems come from expecting one level to prove what only another can.

  1. Deployment validationAfter every promotion, in the target workspace
    • Bindings point to this stage
    • Smoke run
    • Evidence recorded
  2. End-to-endSource to published table, in a test workspace
    • Landing file to Silver
    • Quality gate decides
  3. Pipeline integrationThe orchestration, with real activities
    • Parameters and expressions
    • Branches and failure paths
    • Retries and reruns
  4. Notebook integrationThe notebook in a Fabric session
    • Parameters cell
    • Lakehouse reads and writes
    • Exit value
  5. UnitLogic outside the orchestration, no tenant
    • Transformations
    • Parameter parsing
    • Exit contract
    • CI gate logic
Testing levels for Fabric pipelines and notebooks. Each level proves something the others can't. The lower levels run on every pull request without a tenant; the upper levels need a test workspace, a deployment, or production.
Level What it proves Needs
Unit Transformation logic, parameter parsing, exit contract and CI decision logic are correct Python, pytest, local Spark
Notebook integration Parameters arrive as expected, the Fabric session can read and write the Lakehouse, failures fail and the exit value is correct A test workspace and test Lakehouse
Pipeline integration Expressions resolve, branches route correctly, dependencies and failure paths work, retries and reruns behave A test workspace, test inputs
End-to-end Landing file → pipeline → notebook → Silver → quality checks → publish works as one flow Representative test data
Deployment validation After promotion, everything points at this stage’s resources and still works The target workspace, right after deployment

The scenario

One pipeline, pl_orders_ingest, used throughout:

  1. A partner drops orders.csv into Files/landing/<env>/orders/YYYY/MM/DD/ in the Lakehouse.
  2. Get Metadata checks that the file exists; if not, the pipeline takes its missing-file path.
  3. The notebook nb_orders_load validates its parameters, parses and stages the file, runs the data-quality checks, and publishes to silver.orders (or holds the batch).
  4. The pipeline advances the watermark only if the notebook reports a published batch.
  5. Every run leaves rows in dq.check_result, dq.orders_quarantine and the execution log, tagged with its run ID.

Parameters:

Parameter Type Example Meaning
run_date string (ISO date) 2026-11-09 Business date of the batch
source_path string Files/landing/test/orders/2026/11/09/orders.csv The file to load
target_table string silver.orders Logical target; resolved per environment
full_reload bool false Rebuild the table instead of merging the day
run_id string the pipeline’s @pipeline().RunId Correlates every record the run writes

What makes a pipeline testable

Testability is an architecture property, not something added after the pipeline is built.

Design choice What it makes testable
Explicit parameters, no hidden defaults that point at production A test can run the same pipeline against test inputs
Deterministic inputs: the run date is a parameter, not utcNow() deep inside an activity A test can replay a specific day
Transformation logic in functions, not inline in notebook cells Unit tests without a tenant
A run ID on every record the run writes Assertions can find exactly what this run did
A clear output contract: target table, schema, quarantine table, exit value Assertions know what to check
Explicit failure behaviour: Fail activities, raised exceptions, a held batch Failure tests have a defined expected outcome
Restartable steps: state (watermark, processed files) advanced only after success Rerun tests have a defined expected outcome
Observable execution records Evidence survives the run

A pipeline built without these can still be tested, but only by clicking Run and looking at it. That’s the anti-pattern this guide replaces.

Notebook structure for testing

  1. Unit + integrationParameters cellParsed and validated by a function
  2. Unit testTransformation functionsDataFrame in, DataFrame out
  3. Integration testRead and write boundariesLakehouse, Delta, permissions
  4. Contract testExit valueSmall versioned JSON
Where each part of a notebook is tested. Parameters and transformations are unit-tested outside Fabric; reads, writes and the session are integration-tested in a test workspace; the exit value is the contract the pipeline reads.

The notebook itself becomes thin: a parameters cell, one call to a parsing function, calls to transformation and quality functions imported from a library (packaged as a wheel and attached through an environment, or kept in the notebook’s resources), the writes, and the exit. Everything with logic in it can be tested without Fabric.

Parameter testing

Parameters are input, and every input has valid, invalid, missing and boundary values. Write the matrix before writing the tests:

Parameter Valid Invalid Missing Boundary Environment-specific
run_date 2026-11-09 2026-13-01, a future date empty string 2026-01-01, 2025-12-31, 2024-02-29 —
source_path under this environment’s landing root, matching run_date another environment’s path; a folder for a different date empty first file of a month Test must not read Prod’s landing folder
full_reload true, false "no", null default the string "false" —
target_table silver.orders gold.revenue empty — Outside prod, resolves to silver.orders_<env>
run_id the pipeline run ID empty interactive run with no ID — —

Test four things for each parameter: the default (an interactive run with defaults must fail validation or do nothing harmful), valid values, invalid and missing values (rejected before any data is read, with a message an on-call engineer can act on), and environment-specific values (a value that is valid in one environment must be rejected in another).

Notebook parameter testing

Fabric notebooks receive parameters through a parameters cell: the cell toggled as the parameter cell holds defaults, and a run from a pipeline’s Notebook activity or the job API injects the supplied values after it. The Notebook activity can auto-populate its base parameters from that cell, with names, types and defaults (Notebook activity). Microsoft Fabric Notebook Parameters covers the mechanics; here, the point is how to test them.

Separate parsing from business logic. The parameters cell only assigns values; the next cell calls one function that validates all of them and returns a typed object. That function is plain Python, so every row of the matrix above becomes a fast unit test.

# orders_ingest/params.py
"""Parse and validate the orders notebook's parameters. Plain Python: testable without Fabric or Spark."""
import re
from dataclasses import dataclass
from datetime import date

LANDING_ROOT = {
    "dev": "Files/landing/dev/orders/",
    "test": "Files/landing/test/orders/",
    "prod": "Files/landing/prod/orders/",
}
ALLOWED_TARGETS = {"silver.orders"}       # outside prod, the notebook writes to silver.orders_<env>
RUN_ID = re.compile(r"^[A-Za-z0-9-]{8,64}$")


@dataclass(frozen=True)
class OrdersRun:
    run_date: date
    source_path: str
    target_table: str
    full_reload: bool
    run_id: str
    environment: str


def parse_bool(value) -> bool:
    """Accept real booleans and the strings 'true'/'false'. Reject anything else: bool('false') is True."""
    if isinstance(value, bool):
        return value
    if isinstance(value, str) and value.strip().lower() in {"true", "false"}:
        return value.strip().lower() == "true"
    raise ValueError(f"full_reload must be true or false, got {value!r}")


def parse_params(run_date, source_path, target_table, full_reload, run_id, environment, today: date) -> OrdersRun:
    """Validate everything before any data is read. `today` is passed in, never read from the clock."""
    errors = []

    if environment not in LANDING_ROOT:
        errors.append(f"environment must be one of {sorted(LANDING_ROOT)}")

    try:
        day = date.fromisoformat(run_date or "")
        if day > today:
            errors.append("run_date is in the future")
    except ValueError:
        day = None
        errors.append("run_date must be an ISO date such as 2026-11-09")

    root = LANDING_ROOT.get(environment)
    if not source_path:
        errors.append("source_path is required")
    elif root and not source_path.startswith(root):
        errors.append(f"source_path must be under {root} in {environment}")
    elif day and day.strftime("%Y/%m/%d") not in source_path:
        errors.append("source_path does not match run_date")

    expected_target = target_table if environment == "prod" else f"{target_table}_{environment}"
    if target_table not in ALLOWED_TARGETS:
        errors.append(f"target_table must be one of {sorted(ALLOWED_TARGETS)}")

    try:
        reload_flag = parse_bool(full_reload)
    except ValueError as error:
        errors.append(str(error))
        reload_flag = False

    if not RUN_ID.fullmatch(run_id or ""):
        errors.append("run_id is required (the pipeline run ID)")

    if errors:
        raise ValueError("Invalid parameters: " + "; ".join(errors))

    return OrdersRun(day, source_path, expected_target, reload_flag, run_id, environment)

The tests turn the matrix into code:

@pytest.mark.parametrize(("field", "value", "message"), [
    ("run_date", "2026-13-01", "ISO date"),
    ("run_date", "", "ISO date"),
    ("run_date", "2026-11-11", "future"),
    ("source_path", "", "required"),
    ("source_path", "Files/landing/prod/orders/2026/11/09/orders.csv", "must be under"),
    ("source_path", "Files/landing/test/orders/2026/11/08/orders.csv", "does not match run_date"),
    ("target_table", "gold.revenue", "target_table"),
    ("full_reload", "no", "full_reload"),
    ("full_reload", None, "full_reload"),
    ("run_id", "", "run_id"),
    ("environment", "staging", "environment"),
])
def test_invalid_parameters_are_rejected_with_a_clear_reason(field, value, message):
    with pytest.raises(ValueError, match=message):
        params(**{field: value})


def test_the_truthy_string_trap():
    assert bool("false") is True          # why full_reload can't be checked with `if full_reload:`
    assert parse_bool("false") is False

(params() builds a valid parameter set with one field overridden; TODAY is a fixed date.) Three details matter more than they look:

  • All errors are reported at once. A run that fails on the first problem, then the second, then the third wastes three test cycles.
  • "false" is truthy. A parameter that arrives as a string (an interactive run, a hand-typed value, a changed type in the activity) turns if full_reload: into a full rebuild of the table. The test pins that down.
  • The environment check is a parameter test. “Test reads Prod’s landing folder” is the most expensive parameter bug there is, and it’s one line of validation.

At notebook-integration level, run the notebook in the test workspace three ways: with valid values, with one invalid value (Expected result: the run fails in the validation cell, before any read, with the validation message in the error), and with no values at all (Expected result: the defaults fail validation; nothing is written).

Pipeline expression testing

A pipeline expression can be syntactically valid, evaluate without error, and send the run to the wrong place. That’s the failure mode to test for. Typical expressions in the scenario, using functions and system variables documented in the pipeline expression language:

Default run date   @formatDateTime(addDays(pipeline().TriggerTime, -1), 'yyyy-MM-dd')

Dynamic file path  @concat('Files/landing/', pipeline().libraryVariables.environment, '/orders/',
                           replace(pipeline().parameters.run_date, '-', '/'), '/orders.csv')

Target table       @concat(pipeline().parameters.target_table,
                           if(equals(pipeline().libraryVariables.environment, 'prod'), '',
                              concat('_', pipeline().libraryVariables.environment)))

Branch             @equals(pipeline().parameters.full_reload, true)

Run correlation    @pipeline().RunId   (passed to the notebook as run_id)
Expression What can go silently wrong
Default run date TriggerTime is UTC. A run at 20:00 local time in a UTC−5 zone is already the next day in UTC, so “yesterday” is computed from the wrong day. A format string with dd and MM swapped produces a valid but wrong date
Dynamic file path A missing slash, or the environment value from the wrong value set, gives a path that exists in another environment
Target table Test writes to the production table name
Branch A string "false" compared with a boolean routes to the wrong branch
Run correlation Forgetting it leaves the run’s records unfindable

pipeline().libraryVariables.environment stands for a variable library reference, added in the pipeline’s Library variables tab and inserted from the expression builder; insert it from the builder rather than typing it (variable libraries in pipelines).

How to test expressions:

  • Make the resolved value visible. Pass computed paths and names into the notebook as parameters, and have the notebook log them to the execution record. A test then asserts on the resolved value, not on the expression text.
  • Test the boundaries the expression handles: month-end, year-end, a run just after midnight, both values of every boolean.
  • Assert the destination, not the status. After a test run, check that the file read and the table written are the expected ones for this environment. A run that wrote silver.orders from the Test workspace is green and wrong.
  • Treat every If Condition and Switch as two or more tests, one per branch. Untested branches are where failure handling breaks.

Controlled failure testing

A pipeline’s failure paths are part of its design, and they need tests as much as the happy path does. Run each scenario deliberately, in the test workspace, with test inputs:

Scenario How to cause it Expected result
Missing source file Run with a run_date that has no file Get Metadata reports the file absent; the pipeline routes to a Fail activity with a specific error code and message, or to the designed no-input path; the watermark doesn’t move; the execution record says why
Invalid notebook parameter full_reload = "no" The notebook fails in the validation cell, before reading anything; the activity fails; the error names the parameter
Schema mismatch A test file with a renamed column The transformation or the quality step rejects the batch; nothing is published; the evidence names the column
Permission problem Run with an identity that lacks access to the test Lakehouse The run fails with an authorization error in the activity output; no partial write
Downstream unavailable Point a test connection at an unavailable endpoint The configured retries run; then the activity fails; the failure path records it

The Fail activity makes a designed failure explicit: it fails the run with a custom message and error code (activity overview), which a test (and an on-call engineer) can distinguish from an accident. Don’t assert on exact error text from Fabric itself; it changes. Assert on what your design controls: which activity failed, the error code you set, whether state moved, what the evidence says.

Retry testing

Retries exist for transient failures: throttling, a brief outage, a flapping dependency. They should never hide a deterministic defect.

Fabric activities have Retry (default 1 when enabled), Retry interval (default 30 seconds, fixed or increasing delay with exponential back-off) and Timeout (default 12 hours). Retry conditions (preview) restrict retries to failures whose error code, error message or failure type matches; without conditions, an activity retries on every failure (activity retries).

Two tests, two expected outcomes:

Test Cause Expected result
Transient failure The endpoint is briefly unavailable First attempt fails, a retry succeeds; retryAttempt above zero is recorded
Deterministic defect The file has an invalid schema Every attempt fails the same way; with a retry condition, it fails once

The second one is the important test. Retrying a bad file reads it again, parses it again and fails again, consuming capacity each time, and with a long retry interval it delays the alert by the interval multiplied by the retry count. Microsoft’s own example of a retry condition is retrying only on rate limiting (error code 429). Use retry conditions on Notebook and Copy activities so that deterministic failures (validation errors, schema errors, authorization failures) fail on the first attempt.

Two cautions from the documentation: the retry interval runs before the condition is evaluated, so a non-matching failure still waits one interval; and conditional retries are supported only for some activity types (Copy data, Notebook, Dataflow and Stored procedure among them). Test the behaviour on the activity types you use.

A notebook can help: raise a clearly distinguishable exception for validation and data errors, so a retry condition can exclude them. The activity runs API reports a retryAttempt per activity run; record it, because a run that succeeded on its third attempt isn’t as healthy as one that succeeded on its first (debugging retries).

Idempotency: run the same batch twice

This is the most valuable test a Fabric pipeline can have, because every pipeline will be retried, rerun after an incident, or replayed during recovery.

Run the same batch twice. The data must end up exactly as if it had run once.

Expected final state after the second run:

  • No duplicate business rows: silver.orders has the same rows and the same row count as after the first run.
  • No duplicate side effects: the quarantine table doesn’t contain the same rejected rows twice; the file isn’t registered as processed twice; downstream notifications aren’t sent twice.
  • Distinct execution records: two records, with different run IDs, both visible. The log mustn’t block a legitimate rerun.
  • Correct state: the watermark is where it was after the first run.

What makes it pass is design, not luck: MERGE on the business key instead of append, deduplication inside the batch, processed-file tracking written after the load succeeds, quarantine rows keyed by run and source row, and watermark advancement only after the publish. Incremental Loads in Microsoft Fabric covers these patterns, and Recovery and Disaster Recovery Architecture for Microsoft Fabric explains why recovery depends on them. A Delta MERGE idempotency test at unit-test scale is in the PySpark testing guide; in Fabric, run it as a pipeline test. Testing Incremental Loads and Idempotency tests each of these guarantees in depth, with a tested PySpark suite.

Rerun from partial failure

The harder variant: the run fails halfway.

  1. Ingestion succeeds: the file is read and staged.
  2. Transformation and audit succeed: the batch is ready to publish.
  3. Publishing fails: for example, a permission change on the target, or a capacity interruption.

Then rerun. For each step, the design must say whether it reruns, skips, rebuilds or resumes:

    1. SucceededIngest file
    2. on rerun
      Rebuild from the fileFile is unchanged and still in landing
    1. SucceededStage and audit
    2. on rerun
      RebuildDeterministic from the same input
    1. FailedPublish
    2. on rerun
      Rerun, idempotentlyMERGE by business key
    1. Not advancedWatermark
    2. on rerun
      Advance after publishOnly now, and only once
What each step does on rerun. After a failure at publish, the rerun must neither skip the publish nor duplicate the earlier steps' effects. State that moved before the failure is the usual cause of a broken rerun.

Two rules make partial reruns safe. State moves last: the watermark and the processed-file register are updated only after the publish succeeds, so a failure anywhere leaves them pointing at the start of the batch. Rebuild from inputs that haven’t moved: rebuilding staging from the same landing file is safe; rerunning only the failed activity, after a scheduled run has already advanced the watermark or replaced the file, is not. Fabric can rerun a pipeline from a failed activity, which is useful exactly when the inputs that activity depends on are unchanged (rerun debugging).

The test: force the publish step to fail once (for example, remove write permission on the test target for the first run), restore it, rerun the whole pipeline, and assert the idempotency outcomes above plus “the watermark moved exactly once”.

Notebook exit values: an explicit contract

A notebook can end with notebookutils.notebook.exit(value). The value is always a string; exit() ends the run as successful, so a failure must raise instead; and exit() doesn’t take effect inside a try/except (Microsoft Fabric Notebook Parameters, NotebookUtils notebook run). For runs started through the Job Scheduler API, the exit value is returned in the exitValue field of the job instance (notebook public APIs).

Orchestrators need an explicit signal, because “the notebook succeeded” covers both “published 10,000 rows” and “found an empty file and did nothing”. Make the exit value a small, versioned contract:

# orders_ingest/exit_contract.py
"""The notebook's exit value: a small, versioned JSON string the orchestrator can rely on."""
import json

STATUSES = {"published", "published_with_warnings", "held", "no_input"}


def build_exit(status: str, run_id: str, rows_read: int, rows_written: int, rows_quarantined: int) -> str:
    if status not in STATUSES:
        raise ValueError(f"unknown status {status!r}")
    return json.dumps({
        "contract": 1,
        "status": status,
        "run_id": run_id,
        "rows_read": rows_read,
        "rows_written": rows_written,
        "rows_quarantined": rows_quarantined,
    })


def parse_exit(value) -> dict:
    """Strict on what the orchestrator branches on, tolerant of fields it doesn't know yet."""
    if not value:
        raise ValueError("notebook returned no exit value")
    payload = json.loads(value)
    if payload.get("contract") != 1:
        raise ValueError(f"unsupported exit contract {payload.get('contract')!r}")
    if payload.get("status") not in STATUSES:
        raise ValueError(f"unknown status {payload.get('status')!r}")
    return payload

The last cell of the notebook calls notebookutils.notebook.exit(build_exit(...)), outside any try block. A held batch is a successful notebook run (the gate did its job) with an explicit status the pipeline can branch on.

Keep the contract boring: a few fields, a version number, no data, and a reader that ignores fields it doesn’t know (so the notebook can add one without breaking the pipeline). In the pipeline, read the exit value from the Notebook activity’s output and parse it with the json() expression function before branching on a field. Microsoft Learn doesn’t document the exact output path of the exit value in the activity output; copy it from the activity’s output JSON in a real run’s details rather than typing it from memory, and keep that one expression covered by a pipeline test.

Test data

The principles from the flagship’s test data strategy apply unchanged: small, deterministic, synthetic, privacy-safe, with edge cases included on purpose. For this scenario, a handful of versioned test files in the test workspace’s landing folder:

File Contents Expected outcome
2026/11/02/orders.csv Three normal orders Published; 3 rows; quarantine empty
2026/11/03/orders.csv The same order twice (identical rows) Published; 1 row for that order
2026/11/04/orders.csv One order with no customer Published with warnings; 1 row quarantined with reason customer_id_present
2026/11/05/orders.csv A malformed amount and a malformed date Rows quarantined, or batch held if above the quarantine ratio
2026/11/06/orders.csv An order dated 2026-11-01 (late-arriving) Handled by the design: merged into the right business date, or quarantined, as specified
2026/11/07/orders.csv Header row only Exit status no_input; nothing published; watermark behaviour as designed
2026/11/08/ No file Missing-file path (see failure tests)

Each date folder is one test case, so a test run is selected by run_date alone. Keep the files in Git next to the pipeline definition, and copy them into the test Lakehouse as part of test setup.

Assertions after a run

After each test run, assert on the data and the state, never only on the run status. Data Quality Checks for Data Pipelines defines the rule framework and the results table; the test assertions read what it wrote:

Assertion Example (illustrative, not executed)
The target exists and has the expected rows SELECT COUNT(*) FROM silver.orders_test WHERE run_id = '<run_id>' equals the expected count for that test file
Critical keys are not null SELECT COUNT(*) FROM silver.orders_test WHERE customer_id IS NULL is 0
No duplicate business keys SELECT order_id FROM silver.orders_test GROUP BY order_id HAVING COUNT(*) > 1 returns nothing
The run was recorded execution_log has one row for the pipeline run ID, with status and row counts
Quality results were recorded dq.check_result has a row per check for the run, and no failed error-level checks for a file that should pass
Rejected rows were captured dq.orders_quarantine has exactly the expected rows and reasons for the test file
The watermark advanced correctly It moved to run_date after a published batch, and didn’t move after a held batch or a failure

Run these from the CI job through the test Lakehouse’s SQL analytics endpoint or a test notebook, and fail the CI step on the first mismatch.

Execution evidence

A test that leaves no evidence has to be rerun to be believed. Each test run should leave a record like this:

Field Meaning
run_id The pipeline run ID (@pipeline().RunId)
pipeline Pipeline name
activity The failing or checked activity
start_time, end_time Timing, for baselines
status Outcome of the run, and of the batch (exit status)
input Parameter summary (resolved paths and table names, no secrets)
output Rows read, written, quarantined
error Error code and message, when it failed
environment dev, test or prod
version The commit or release that was deployed

The pipeline writes most of these into the execution log as part of its normal run, which is the point: the same evidence serves tests, monitoring and incident investigation. The CI job adds the version and the gate decision. For Fabric’s own view, the activity runs API returns each activity’s status, start and end, durationInMs, error and retryAttempt for a pipeline run (pipeline REST API).

The testing matrix

What Unit Integration (test workspace) Deployment validation Production smoke
Transformation pytest + local Spark Output table on test files — Schema of the published table
Notebook parameters Parsing function Valid, invalid and empty runs Defaults are harmless in this stage —
Pipeline parameters — Each matrix row as a run Values resolve for this stage —
Expressions — Resolved paths and names asserted Resolve to this stage’s resources Resolved path logged
Storage access — Reads and writes on the test Lakehouse Bindings point to this stage Small read and write
Identity — Positive and negative runs Workspace identity or connection works Expected identity in the log
Data quality Rule logic Rules on test files — Checks run on the smoke batch
Idempotency Batch deduplication Same batch twice — —
Failure paths Validation errors Missing file, bad schema, permission — —
Environment configuration Environment check in parsing Variable library values No Dev reference anywhere —
Exit contract Build and parse Pipeline branches on it — —
Performance — Duration against baseline — Duration within baseline

CI/CD integration

  1. Pull request
  2. Static validation
  3. Unit testsParsing, transforms, gate logic
  4. Deploy to Test
  5. Run Fabric test pipelineThrough the Job Scheduler API
  6. Data assertionsTables, quarantine, evidence
  7. Security checks
  8. Approval
  9. Deploy to Production
  10. Smoke testSmall, safe subset
Fabric pipeline tests in a release. Unit tests need no tenant. Everything after deployment runs in a Fabric workspace, triggered and evaluated by the CI job, which blocks promotion when any step fails.

The flagship covers the general gate model; here is the Fabric-specific part, the middle of the flow.

Automated Fabric execution

A CI job can start Fabric items with the Job Scheduler REST API and a service principal or managed identity that has a role on the test workspace:

  1. Deploy to the test workspace (deployment pipelines, Git-based deployment, or a tool such as fabric-cicd).
  2. Start the test run: POST /v1/workspaces/{workspaceId}/items/{itemId}/jobs/{jobType}/instances (the job type is Pipeline for a pipeline, RunNotebook for a notebook). The request is accepted with 202 and a Location header pointing at the job instance, plus Retry-After (Run on demand item job). Per-run parameters go in the request body; its format differs by item type, so take it from the item’s API reference.
  3. Poll GET …/jobs/instances/{jobInstanceId} until the status is terminal: Completed, Failed, Cancelled or Deduped; NotStarted and InProgress mean keep waiting, and Microsoft notes that more statuses may be added (Get item job instance). For a notebook, the response includes exitValue. For a pipeline, query its activity runs to see each activity’s outcome.
  4. Evaluate: run the data assertions and read the quality results for the run.
  5. Block or promote.

The polling and decision logic belongs in a small, unit-tested module, with the HTTP calls injected:

# orders_ingest/ci_gate.py
"""CI helpers: wait for a Fabric job instance, then decide whether the test run allows promotion.

The HTTP calls are injected (get_status), so this logic is unit-tested without a tenant.
"""
import time

TERMINAL = {"Completed", "Failed", "Cancelled", "Deduped"}


def wait_for_job(get_status, timeout_s: int, poll_s: int = 30, sleep=time.sleep, clock=time.monotonic) -> dict:
    """Poll a job instance until it reaches a terminal status or the timeout expires.

    get_status() returns the Get Item Job Instance payload. Unknown statuses keep polling until the
    timeout: the API documents that new statuses may be added.
    """
    deadline = clock() + timeout_s
    while True:
        job = get_status()
        if job.get("status") in TERMINAL:
            return job
        if clock() >= deadline:
            raise TimeoutError(f"job still {job.get('status')!r} after {timeout_s}s")
        sleep(poll_s)


def promotion_decision(job: dict, activity_runs: list[dict], dq_failed_errors: int, exit_payload: dict | None) -> tuple[bool, list[str]]:
    """Only a completed run, with every activity succeeded, no error-level data-quality failure and the
    expected exit status, allows promotion. Everything else blocks, with reasons."""
    reasons = []
    if job.get("status") != "Completed":
        reasons.append(f"job status is {job.get('status')!r}, not 'Completed'")
    failed = [a["activityName"] for a in activity_runs if a.get("status") != "Succeeded"]
    if failed:
        reasons.append(f"activities not succeeded: {failed}")
    if dq_failed_errors:
        reasons.append(f"{dq_failed_errors} error-level data-quality checks failed")
    if exit_payload is None or exit_payload.get("status") not in {"published", "published_with_warnings"}:
        reasons.append("notebook did not report a published batch")
    return (not reasons, reasons)

Three behaviours that the tests pin down, and that a hand-written polling loop usually gets wrong:

  • Deduped is terminal, and it blocks. It means another instance of the same job was already running, so this test run was skipped. A gate that only checks “not Failed” would promote untested code.
  • The poll has a timeout. A job stuck in InProgress fails the CI step instead of hanging it.
  • A green run can still block. The test test_succeeded_is_not_correct gives the gate a completed job with every activity succeeded and an exit status of held, and expects promotion to be refused. Removing the data-quality and exit-status checks from the gate (I tried it, in a copy of the module) makes exactly that test and the data-quality test fail.
def test_succeeded_is_not_correct():
    # Every activity green, but the notebook held the batch: nothing was published.
    allowed, reasons = promotion_decision({"status": "Completed"}, ACTIVITIES, 0, {"contract": 1, "status": "held"})
    assert not allowed and "published batch" in reasons[0]

Credentials: the CI job authenticates with a service principal or managed identity whose secret or federated credential lives in the CI system’s secret store, never in the repository, the pipeline definition or a notebook. Give that identity a role on the test workspace only; a separate identity, behind the approval, deploys to production.

Environment-specific testing

Promotion is where correct code turns into a wrong deployment. The usual mistakes:

  • A notebook still bound to the Dev Lakehouse. Deployment pipelines auto-bind a notebook’s default Lakehouse and attached environment to the corresponding items in the target workspace when those items are in the same workspace as the notebook. A Lakehouse in another workspace keeps the original binding unless a deployment rule says otherwise (notebook source control and deployment).
  • The wrong connection on a Copy or Notebook activity.
  • The wrong active value set in the variable library, so Test reads Dev’s values.
  • A hard-coded workspace or item ID in an expression or a notebook.
  • A missing secret in the target environment’s Key Vault.
  • A partial deployment: the deployment pipeline can continue when some items fail, leaving a stage that mixes versions.

Deployment rules and variable libraries

Two mechanisms handle environment differences, and both need tests:

  • Deployment rules override values on the target stage: data source and parameter rules for some item types, and a default Lakehouse rule for notebooks, which takes priority over auto-binding. Rules are set on the target stage, not on Development, apply only on the next deployment, and stop working if the source item no longer has the referenced data source or parameter (create deployment rules).
  • Variable libraries hold one value set per environment, with one value set active per workspace; pipelines, notebooks (through NotebookUtils), Dataflow Gen2 and other items read the active value at run time (variable library overview).

Test three things after every promotion: the expected value is the one used (the resolved path and table in the run’s evidence belong to this stage); a missing value fails loudly (a variable without a value for this stage must fail validation, not fall back to Dev’s); and there is no accidental Dev reference (search the deployed definitions and the run’s evidence for Dev workspace IDs, Dev Lakehouse names and Dev paths). The lifecycle lab walks through a notebook that still reads Dev after deployment, and fixes it with a deployment rule.

Environment rebinding checklist, after every promotion:

  • Every notebook’s default Lakehouse belongs to this stage’s workspace
  • Every notebook’s attached environment belongs to this stage
  • Every connection on every activity points to this stage’s sources and targets
  • The variable library’s active value set is this stage’s
  • No Dev or Test workspace ID, item ID or path appears in deployed definitions or in the smoke run’s evidence
  • Every secret the run needs resolves in this stage
  • The deployment completed for every item; no item was left at the previous version

Security tests

The Fabric Security Lab covers how to prove access controls. For a pipeline release, a short test is enough:

  • the intended identity (the workspace identity or the connection’s identity) can run the pipeline and read and write what it needs;
  • an unintended identity can’t run it, and can’t read the target table’s restricted data;
  • every secret resolves in the stage, with no fallback to a default;
  • workspace and data permissions match the stage’s design (no Contributor left over from testing).

Performance smoke testing

A run that succeeds in four hours instead of twenty minutes is functionally successful and operationally broken. Capture pipeline duration, notebook duration and per-activity duration (durationInMs from the activity runs) for every test run, and compare them with a baseline from previous runs of the same test files on the same capacity. There’s no universal threshold; a large deviation from the baseline is the signal. Small regressions are usually a warning to investigate; a large one blocks or needs approval. Troubleshooting Microsoft Fabric Performance in Production covers diagnosing what you find.

Production smoke tests

After deploying to production, run a small, safe subset:

  • a metadata check: the items exist, at the deployed version, bound to production resources;
  • a small notebook run, or the pipeline on a dedicated smoke-test file that writes to a smoke table, not to silver.orders;
  • the expected schema on the published table;
  • the expected identity in the run’s evidence;
  • the data-quality checks run and record results;
  • the execution log has the smoke run’s record.

Smoke tests must be low risk: they mustn’t write business data, move a watermark or trigger downstream consumers. A smoke test that can corrupt production is worse than none. The full behaviour was proven in Test; production smoke tests prove the deployment.

What should block a deployment

Javi Builds guidance, not a Microsoft requirement:

Failure Recommendation
Notebook unit test fails Block
Pipeline parameter or expression test fails Block
Failure-path test behaves differently from design Block
Rerun test produces duplicates or moves state twice Block
Critical data-quality rule fails on a test file that should pass Block
Test environment still references Dev Block
Security test fails Block
Test run status Deduped, Cancelled or timed out Block (rerun the test; never treat it as a pass)
Small performance deviation from baseline Investigate; policy-based
Large performance regression Block or require explicit approval
Production smoke test fails Stop the rollout; roll back or fix forward

Common anti-patterns

Anti-pattern Instead
Testing only successful runs A test per failure path: missing file, bad parameter, bad schema, permission
Clicking Run as validation Triggered runs with assertions, from CI, with evidence
No test inputs Versioned test files, one per case, selected by run_date
Pipelines tied to production resources Parameters, variable libraries and rules; validation that rejects the wrong environment
One notebook with all the logic and all the I/O Thin notebook; logic in tested functions
Retries hiding bad data Retry conditions for transient errors only; deterministic failures fail once
No run_id @pipeline().RunId on every record the run writes
Reruns create duplicates MERGE, state moved last, the same-batch-twice test
No post-deployment validation Rebinding checklist and a smoke run after every promotion
Tests that never inspect the data Assertions on tables, quarantine, results and watermark
Comparing only activity status “Succeeded” is not “correct”: gate on data, quality results and the exit contract

Mini-lab: test the orders pipeline

Validation status: docs-based. This lab is written from current Microsoft Learn documentation and the patterns above. It hasn’t been run end to end in a Javi Builds Fabric test tenant; every outcome below is an Expected result, not an observed one.

Setup: a test workspace with a Lakehouse, the pl_orders_ingest pipeline and nb_orders_load notebook from your repository, the dq tables from Data Quality Checks for Data Pipelines (or their Delta equivalents), and the test files from the test data table copied into Files/landing/test/orders/.

  1. Create deterministic input. Copy the seven test folders into the landing area. Expected result: each date folder holds exactly the file described in the table; no other files.
  2. Run the valid case: run_date = 2026-11-02. Expected result: the run completes; the exit value reports published with 3 rows written; silver.orders_test has 3 rows with this run’s run_id; the quarantine table has none; the watermark is 2026-11-02.
  3. Run an invalid parameter: full_reload = "no". Expected result: the notebook activity fails in the validation cell; the error names full_reload; nothing is written; the watermark doesn’t move.
  4. Introduce duplicate input: run_date = 2026-11-03. Expected result: one row for the duplicated order; no duplicate keys in silver.orders_test.
  5. Verify quarantine: run_date = 2026-11-04. Expected result: exit status published_with_warnings; one quarantined row with reason customer_id_present; a failed row-level result in dq.check_result for this run.
  6. Rerun the same batch: run 2026-11-04 again. Expected result: silver.orders_test and the quarantine table are unchanged; there are two execution records with different run IDs; the watermark hasn’t moved again.
  7. Confirm idempotency with the assertions above: row count, duplicate keys, quarantine count and watermark, compared before and after the rerun.
  8. Inspect the evidence: the execution log, the quality results and the activity runs for each run ID. Expected result: for every run, you can say what it read, what it wrote, what it rejected, and why it succeeded or failed, without opening the notebook.

Then break it on purpose: remove write access on silver.orders_test for one run, rerun after restoring it, and check that the watermark moved exactly once.

Before promoting a Fabric pipeline

  • Parameters tested (valid, invalid, missing, boundary, environment)
  • Expressions tested, with resolved values asserted
  • Empty input tested
  • Bad schema tested
  • Duplicate input tested
  • Retry behaviour tested (transient recovers, deterministic fails once)
  • Rerun and idempotency tested, including a partial failure
  • Data-quality gates tested on files that must pass and files that must fail
  • Identity and access tested
  • Environment bindings verified after promotion
  • Execution evidence recorded for every test run
  • Performance compared with the baseline
  • Post-deploy smoke test defined, and safe

Sources

Fabric behaviour checked against Microsoft Learn in November 2026; several features referenced here (retry conditions, the new deployment pipeline interface) are in preview and change often. The Python modules shown were executed (36 tests, Python 3.12.10, pytest 9.2.0); the pipeline expressions, REST calls, assertion queries and mini-lab are docs-based. The testing levels, matrix, blocking table and checklists are Javi Builds guidance.

Étiquettes

  • Testing
  • Microsoft Fabric
  • CI/CD
  • Pipelines de données
  • Fabric Notebook