Testing & quality

Data Testing & Quality

Prove that a data platform works, before and after it reaches production.

Practical testing strategies for SQL Server, Microsoft Fabric, Python, PySpark and production data pipelines: unit and transformation tests, data-quality checks, integration tests and the CI/CD gates that decide whether a change should ship.

Covers data platform testing strategy, data pipeline testing, PySpark and SQL unit testing, data-quality checks, schema and data-contract tests, Microsoft Fabric pipeline and notebook testing, performance regression and post-deployment smoke tests.

For data engineers, analytics engineers, platform engineers, architects and technical leads who own production data.

Learning path

Start here

Three steps: the testing architecture and its gates, a real release with a deliberately blocked deployment, then the rerun and reconciliation tests that keep incremental loads safe.

  1. Step 1 · Testing strategy

  2. Step 2 · Release gates in practice

  3. Step 3 · Safe reruns and reconciliation

What this hub covers

Testing by concern

The areas this hub is organized around. Areas without a dedicated guide yet point to the Start here step that covers them today.

Unit & transformation testing

Small, deterministic tests for logic, and input-to-output tests for joins, aggregations and deduplication.

  • pytest
  • PySpark DataFrames
  • T-SQL procedures
  • Edge cases
  • Fixtures

Data quality

Rules on the data that actually arrived: completeness, uniqueness, validity, freshness and reconciliation.

  • Null and duplicate checks
  • Freshness
  • Referential integrity
  • Reconciliation
  • Distributions

Integration & pipeline testing

Components together, against real engines and storage: pipelines, notebooks, Delta writes and reruns.

  • Pipelines
  • Notebooks
  • External dependencies
  • Retry and replay
  • Idempotency

CI/CD quality gates

Which checks run where, from pull request to production, and which ones block a release.

  • Pull request
  • Build
  • Test environment
  • Approval
  • Smoke tests

Performance regression

Baselines, not universal thresholds: query, Spark and pipeline runtimes compared before and after a change.

  • Query runtime
  • Spark jobs
  • Pipeline duration
  • Throughput
  • Concurrency

Security testing

Prove that the right identities reach the right data, and that nobody else does.

  • Workspace roles
  • RLS and OLS
  • Service principals
  • Negative tests
  • Tenant isolation

Recovery & replay testing

Reruns, retries, replays and rebuilds that leave the data correct.

  • Reruns
  • Retries
  • Replay
  • Checkpoint restart
  • Restore and rebuild

Schema & data contracts

What producers promise consumers, and the checks that catch breaking changes before they ship.

  • Required columns
  • Types and nullability
  • Breaking changes
  • Schema evolution
  • Semantic changes

Covered today in Start here, step 1.

View all testing articles →

By technology

Technology tracks

Each track starts in the matching section of the flagship guide, then continues on the technology hub.

Hands-on

Checklists and quick references

Use these while building and releasing; the guides above explain the reasoning.

  • ChecklistDevOps & Observability

    Fabric CI/CD Deployment Checklist

    What to check before, during and after promoting Microsoft Fabric items from Git to Test and Production.

    • CI/CD
    • Git
    • Microsoft Fabric
    • Fabric Workspace
    • +1

  • ChecklistMicrosoft Fabric

    Data Platform Security Architecture Review Checklist

    The security architecture questions to verify before approving a production data platform, for single-organization and multi-tenant platforms alike.

    • Security
    • Data Architecture
    • Multi-Tenancy
    • Microsoft Fabric
    • +1

  • ChecklistDevOps & Observability

    Data Platform Monitoring Checklist

    A production checklist for application, API, report engine, SQL, Fabric and data-quality monitoring.

    • Observability
    • Monitoring
    • API
    • Reporting
    • +3

  • Data EngineeringData QualityIntermediate

    Data Quality Checks

    Build practical data quality checks for schema, completeness, uniqueness, validity, referential integrity, freshness, reconciliation, and pipeline drift without relying on arbitrary universal thresholds.

  • Data EngineeringData IntegrationIntermediate

    Incremental Load Patterns

    Design safe incremental loads using watermarks, CDC, high-water marks, MERGE, idempotent writes, late-data handling, and replayable checkpoints without skipping or duplicating data.