Microsoft Fabric

Recovery and Disaster Recovery Architecture for Microsoft Fabric: RPO, RTO, Replay and Rebuild

An architect's guide to recovery for production Microsoft Fabric platforms: RPO and RTO that the design can actually meet, failure domains, what Microsoft recovers and what you rebuild, the capacity disaster recovery setting, item-by-item recovery, replay and idempotency, multi-tenant recovery, security revalidation, RTO decomposition and recovery testing.

Par JaviPublié 35 min de lectureAvancé

Dans cet article

A Fabric capacity with the disaster recovery switch turned on is not a recoverable platform. The switch copies OneLake files to another region. It doesn’t recreate your notebooks, reconnect your pipelines, rebind your applications, restore your workspace identities, or tell you whether the data that arrived is complete. This guide answers one question:

If a production Fabric platform, workspace, item, dependency or region becomes unavailable, what can Microsoft recover automatically, what must I rebuild or replay, and how do I design for the RPO and RTO the business actually requires?

The architectural principle. Microsoft-managed availability is not the same thing as application recovery. Microsoft keeps the service running, spreads it across availability zones and, if you opt in, keeps a second copy of OneLake data. Your platform is more than that: item definitions, connections and credentials, run state and watermarks, downstream applications, security boundaries and the business’s trust in the numbers. An architect has to separate seven things that a single word, “recovery”, tends to blur:

Layer The question Typically owned by
Service availability Is Fabric itself up and reachable? Microsoft
Data redundancy Does a copy of the bytes exist somewhere? Microsoft, partly opt-in
Item and configuration recovery Do notebooks, pipelines, models and settings exist again? You
Pipeline and workflow state Do loads know where they stopped, and can they resume? You
Downstream application recovery Do APIs, reports and consumers point at the right place? You
Identity and security recovery Are identities, roles, secrets and tenant boundaries correct again? You
Business validation Are the numbers complete and correct enough to reopen? You and the business

And one more principle runs through the whole guide: recovery is a design problem before it is an incident-response problem. Every recovery capability that matters, from replayable sources to rebindable endpoints, has to exist before the incident. During the incident you can only use what you designed.

Where this guide sits: Designing a Production Data Platform introduces RPO and RTO among the platform’s other decisions, and SQL Server to Microsoft Fabric: What Maps to What? explains how backup and high availability change for a SQL Server team. This guide goes further: failure domains, recovery by item, replay, multi-tenant recovery, security revalidation and how to prove a recovery time.

The running example

The examples use a generic multi-customer analytics platform:

  • Sources: an operational database read through change data capture, nightly partner files, and an events feed.
  • Storage: a Lakehouse holding raw landing and curated Delta tables, and a Warehouse serving a curated star schema.
  • Processing: notebooks and pipelines orchestrated on schedules, with an execution log.
  • Serving: semantic models and reports for internal users, and a data API serving each customer its own data.
  • Tenancy: some customers share workspaces and tables (a tenant ID column), and a few large customers have a dedicated workspace.

The business has agreed on recovery targets per workload: internal finance reporting can tolerate the loss of the last daily load and a next-business-day recovery, while the customer API has a four-hour RTO and the largest customers have contractual targets of their own.

RPO and RTO, defined properly

RPO: Recovery Point Objective

The recovery point objective is the maximum acceptable amount of data loss, measured in time, when an incident happens. After recovery, data newer than the recovery point is either lost or must be reloaded.

RPO What it means in practice
Zero or near zero No acknowledged data may be lost. Requires synchronous or continuously replicated state, or a source that can always replay
15 minutes At most the last quarter hour of changes needs to be replayed or is lost
One hour Hourly batches or a CDC stream that can resume from a known position
Previous daily load Yesterday’s load is the recovery point; today’s must be rerun

RTO: Recovery Time Objective

The recovery time objective is the maximum acceptable time a workload can be unavailable after an incident. The clock starts at the incident, not when someone decides to recover.

RTO What it means in practice
30 minutes Standby environment and automated failover; no manual rebuilding
Four hours Pre-scripted rebuild, pre-approved decisions, rehearsed runbook
Next business day Manual rebuild is acceptable if it is documented and has been tried once

A platform cannot have an RPO or RTO merely because the underlying cloud service is resilient. Microsoft’s numbers describe the service: how fast it fails over, how its storage replicates. Your RPO depends on whether your sources can resend data and your loads can resume; your RTO depends on how long it takes to rebuild, rebind and validate your items. The architecture has to support the target, and only a test proves it.

Requirement Architectural consequence
RPO ≤ 15 minutes Source replay or replicated state must cover any 15-minute window. Asynchronous geo-replication of OneLake files alone can’t promise it, because Microsoft gives no RPO for it
RPO = previous daily load Raw files or source extracts for the current day must survive the incident, or the source must resend them
RTO ≤ 1 hour Manual recreation of dozens of workspaces, connections and permissions is not credible. Pre-provisioned or fully automated recovery is required
RTO ≤ 4 hours Infrastructure and items deployed by script from Git; data recovery parallelized; validation automated
RTO = next business day A documented manual rebuild is acceptable, if it has been rehearsed at least once
Tenant-specific SLA Recovery order, isolation pattern and standby resources may differ by customer tier
Regulatory retention Raw and audit data need their own protection, independent of the processing items

Javi Builds guidance: agree RPO and RTO per workload, not per platform. “The platform has a four-hour RTO” hides the fact that the finance model and the customer API recover through completely different paths.

Failure domains

Most recovery work is not about regions. It’s about a job that failed at 03:00, a deployment that broke a view, a table overwritten with bad data, or a credential that expired. A recovery architecture starts with a failure taxonomy, because each failure has a different blast radius, a different owner and a different recovery path.

  1. 1Local recoveryJob or activityRerun the step from its checkpoint
  2. 2Item recoveryItemRestore the item, its data or its definition
  3. 3Item recoveryWorkspaceRestore the workspace or redeploy its items
  4. 4Platform recoveryCapacityMove workspaces or recover on another capacity
  5. 5Microsoft-managedAvailability zoneService rebalances; restart interrupted work
  6. 6Regional recoveryRegionMicrosoft fails over; you rebuild in another region
Failure domains, from smallest to largest. Each step outward affects more of the platform and moves recovery from routine operations to platform rebuild. Most incidents sit at the top; the design must still have an answer for the bottom.
Failure Example Recovery class Primary recovery path
Activity or job failure A notebook fails on a timeout Local Retry or rerun from the last committed watermark
Pipeline failure A parent pipeline stops halfway through its children Local Rerun the failed children only; idempotent writes make it safe
Bad deployment A release drops a column a view depends on Item Redeploy the previous version from Git; restore data only if the release changed data
Bad data A source sends duplicated or wrong rows that get loaded Item Restore the affected tables to a point before the load (time travel, restore point), then replay the corrected source
Deleted item A Lakehouse or notebook is deleted by mistake Item Item recovery from the workspace recycle bin, within its retention period
Deleted or corrupted data Files removed, or a table overwritten Item OneLake soft delete, Delta time travel, Warehouse restore point, SQL database point-in-time restore
Workspace issue A workspace is deleted, or its permissions are broken Platform Workspace retention restore, or redeploy from Git into a new workspace
Capacity failure A capacity is paused, throttled or deleted Platform Resume, scale, or reassign workspaces to another capacity
Zone failure One availability zone in the region goes down Microsoft-managed, then local No customer action for the service; resubmit interrupted Spark jobs and queries
Regional outage The whole region is unavailable Regional Microsoft fails over; you create capacity, workspaces and items elsewhere and recover data
Upstream source failure The operational database or partner feed is down Local, sometimes platform Pause dependent loads; catch up from the source’s change retention once it returns
Credential or identity failure A secret expires, a service principal is deleted Platform Rotate or recreate the identity, regrant access, rebind connections
Network failure A gateway host or private endpoint fails Platform Gateway cluster failover, or an alternate connectivity path
External dependency failure A key vault, API or downstream system is unavailable Platform That dependency’s own recovery; the platform waits or degrades gracefully

Two points to take from the table. First, the most frequent failures are local and item-level, and they are recovered by design choices you make anyway for normal operations: restartable loads, idempotent writes, retained raw data and definitions in Git. Second, a regional outage is rare but needs every one of those capabilities at once, plus a few more. A platform that can’t recover from a bad deployment won’t recover from a region.

What Microsoft manages and what you manage

Microsoft describes reliability as a shared responsibility: Microsoft keeps the baseline infrastructure and platform services available, and you design the recovery of your workload. The Fabric reliability guidance is explicit that after a regional disaster you must follow manual steps to restore service.

  • Microsoft-managedPart of the service, or opt-in
    1. Zone distributionFabric resources spread across zones, no configuration
    2. OneLake storage redundancyZRS where available, LRS elsewhere
    3. Regional failover decisionInitiated by Microsoft, not by you
    4. OneLake geo-replicationOpt-in per capacity; asynchronous, file-level
    5. Power BI BCDROn by default where a paired region supports it
  • Customer-managedDesigned, automated and tested by you
    1. Recovery capacity and workspacesCreated in a healthy region
    2. Item definitionsRedeployed from Git or recreated
    3. Data recovery and replayCopied from replicas or reloaded from sources
    4. Connections, secrets, identitiesRecreated, regranted and rebound
    5. Consumers and endpointsApplications pointed at new workspaces and items
    6. Validation and reopeningSecurity, data and business checks before go-live
Who recovers what. Read each column top to bottom; rows don't correspond. Microsoft keeps the service available and, if you opt in, a second copy of OneLake files. Everything that makes it your platform again is yours to rebuild.
Recovery area Microsoft-managed Customer-managed
Infrastructure redundancy Datacenter hardware, storage replication inside the region Nothing, beyond choosing a region
Availability zones Automatic distribution and rebalancing of Fabric resources Resubmitting interrupted Spark jobs and queries; ZRS on your own external storage
Fabric service failover The decision to fail over a region, and the failover itself Monitoring the status page; executing your recovery plan
OneLake replication Asynchronous geo-replication, if you enabled it on the capacity Turning it on where justified; accepting that the copy may not be point-in-time consistent
Item definitions Some definitions survive in replicated OneLake files (for example Spark job definition code) Everything else: Git integration, or exported templates, then redeployment
Git source Nothing: the repository is in your Git provider Repository availability, branch discipline, committed state
Credentials Credential management for workspace identities Key Vault secrets, service principal credentials, connection credentials
Connections and gateways Nothing in the recovery workspace Recreating connections, gateways and their bindings
Environment-specific settings Nothing Variable library values, parameters, schedules, capacity assignment
Run state Nothing: run history and monitoring data stay with the original workspace Watermarks and checkpoints in durable storage; your own execution log
Custom logging Nothing Log tables protected like data; recovery events logged
Downstream endpoints Nothing Connection strings, item IDs and endpoints updated in every consumer
Tenant isolation validation Nothing Cross-tenant tests after every recovery
Business reconciliation Nothing Row counts, totals and freshness checked against sources

Read the right column carefully: it is most of the work. Microsoft’s responsibility ends at the service and, optionally, the bytes. Nothing in Fabric reconnects your API to a new Warehouse or decides that the recovered numbers are right.

Availability zones vs regional disaster recovery

These are different protections against different failures, and confusing them is the most common design error in Fabric recovery plans.

  1. Item or workloadYour notebooks, tables, models and APIs
  2. Zone resilienceAutomatic; survives a datacenter failure
  3. Regional DROpt-in OneLake geo-replication
  4. Customer recoveryRebuild, replay, rebind, validate
Layers of protection. Zone resilience is automatic and covers a datacenter. Regional DR is opt-in and covers a copy of OneLake data. Neither one rebuilds your platform; the last step is always yours.

Zone failure: within a region

Availability zones are physically separate groups of datacenters inside one Azure region. Fabric distributes its resources across zones automatically, with no customer configuration, and OneLake uses zone-redundant storage where the region supports it (locally redundant storage elsewhere).

During a zone outage, Microsoft states that no customer action is required: Fabric capabilities self-heal and rebalance to the healthy zones. The part that still affects you is work in flight: a running Spark job can fail if its driver was in the failed zone, and a Warehouse or SQL analytics endpoint query can fail if its front end was. Those must be resubmitted, which is safe only if the job is restartable and its writes are idempotent. Microsoft also notes that capacities in regions with high demand fluctuations might see higher latency during a zone failure. If your Spark workloads read external ADLS Gen2 storage rather than OneLake, that storage needs its own zone-redundant configuration.

Regional failure: a different event

A regional failure is a broader event that zones don’t cover. Fabric’s response, per the reliability guidance:

  • Microsoft initiates the failover. You can’t access the Fabric portal until it completes; a notice appears on the Fabric support page. Microsoft says it typically completes in less than one hour, but that is the time to a read-only service, not to your recovered platform.
  • After failover, Fabric is mostly read-only. You can browse workspaces and items, but write operations such as creating or modifying workspaces are paused. Power BI shows reports and dashboards but can’t refresh, publish or change metadata.
  • Most items can’t be opened. Lakehouses and Warehouses can’t be opened, but their files are reachable through OneLake APIs and tools, if geo-replication was on. Notebooks can’t be opened and their code content isn’t saved. Pipelines, Dataflow Gen2 and Eventstreams can’t be opened. KQL databases and querysets aren’t accessible.
  • Your recovery is a rebuild. Microsoft’s documented recovery path is to create a new capacity (it suggests a region outside your primary geo, because demand is high during such events), create workspaces, create items with the same names, then restore each item following its experience-specific guidance.

Two conditions sit underneath all of this. Cross-region failover applies only where your region has an Azure paired region that supports Fabric; elsewhere, you must build your own cross-region recovery. And the recovery guidance assumes the tenant’s home region is still operational: if the home region itself fails, your recovery waits for Microsoft to recover it first, because core service functionality is tied to the home region.

Availability zones are not regional DR. Zone resilience is the reason you rarely notice datacenter failures. It does nothing for a regional outage, a deleted Lakehouse or a bad load.

The Fabric disaster recovery capacity setting

Fabric has a disaster recovery switch in each capacity’s settings. Checked November 2026 against the Fabric reliability and OneLake disaster recovery documentation:

Aspect Current behavior
Scope Per capacity; available for Fabric and Premium capacities where Azure region pairing aligns with Fabric’s presence
Who can change it Capacity administrators
Data covered OneLake data, which includes Lakehouse and Warehouse data
Data not covered Anything stored outside OneLake: KQL databases and querysets need their own approach, and external sources, shortcut targets in other stores and your own Azure resources are unaffected
Secondary region Determined by the Azure region pair; you can’t choose it
Replication Asynchronous and at file level. A failover can lose data and leave files inconsistent with each other; the secondary might not hold a point-in-time-consistent copy
RPO Microsoft states none for OneLake geo-replication
Change frequency After you change the setting, you must wait 30 days before changing it again
Start-up time Replication can take time to start after enabling it or adding workspaces; the capacity settings show a OneLake Geo-replication status per workspace
Cost Extra storage and transactions, billed as BCDR Storage and BCDR Operations and visible in the Capacity Metrics app
Power BI Power BI BCDR applies whether the switch is on or off

The inconsistency warning deserves emphasis. A Delta table is a set of Parquet files plus a transaction log. If the log replicated a commit but some of the Parquet files it references didn’t, or the other way round, the copy isn’t a valid table at that version. That is why Microsoft’s Lakehouse recovery script trims transaction log entries newer than a failover timestamp that you obtain from Microsoft Support. Plan for reconciliation, not just copying.

Don’t enable it blindly. It costs storage and operations on every write, it locks the decision for 30 days, and it protects only part of the platform.

Scenario Enable DR setting? Additional recovery needed
Sources can resend everything within the RTO, and the RTO is days Often no: reloading from sources may be cheaper and simpler Git-based rebuild; source replay runbook
Raw landing data is the only copy (sources don’t retain history) Yes, for the capacity hosting raw landing Rebuild items; reconcile copied tables; replay after the last consistent point
Large curated history that would take longer than the RTO to recompute Yes Rebuild items; trim to a consistent point; replay recent changes
Near-zero RPO or sub-hour RTO Not sufficient on its own Active parallel environment in a second region with dual ingestion (Microsoft’s own guidance for Warehouse and KQL)
Dev, test and sandbox capacities Usually no Recreate from Git when needed
Shared multi-tenant capacity with tenants on different SLAs Decide by the strictest tenant, or separate capacities by tier Tenant-ordered recovery and per-tenant validation
KQL databases or Eventhouse data The switch doesn’t cover it Independent databases in two regions with parallel ingestion
Region without a Fabric-supported pair Not available Your own replication to a supported secondary region

Recovery by item type

Each Fabric item recovers differently after a regional failover. The table summarizes Microsoft’s experience-specific disaster recovery guidance and the Git integration supported-items list, checked November 2026. “Replicates” means available in the secondary region after a Microsoft failover with the capacity disaster recovery setting on.

Item What replicates What Git restores What you must recreate Data recovery needed?
Lakehouse Delta tables and files in OneLake Lakehouse metadata, shortcut definitions, OneLake security role definitions (Preview), never tables or files The Lakehouse (same name), non-Delta tables from your DDL scripts, materialized lake view schedules Yes: copy tables one at a time with Microsoft’s script (trimmed to the failover timestamp) or Azure Storage Explorer
Warehouse Its data, through OneLake Schema objects (tables, views, procedures, functions) The Warehouse, an interim Lakehouse for the copied data, application connection strings Yes: recover the Delta tables into an interim Lakehouse, then load the new Warehouse with INSERT INTO, SELECT INTO or CREATE TABLE AS SELECT
Notebook Nothing: code content isn’t saved after the disaster Notebook code (Microsoft’s notebook DR guidance labels the Git approach as Preview) Resource-folder files, default Lakehouse binding Only the data the notebook reads or writes
Pipeline Nothing: configuration isn’t replicated Pipeline definition Connections, schedules, references to recovered items Rerun for the missing window
Dataflow Gen2 Nothing: settings aren’t replicated Dataflow definition (Git supported); otherwise an exported .pqt template Destinations and connections Rerun for the missing window
Spark job definition Main and reference files in OneLake Definition The item with the same settings; Lakehouse references; command-line arguments; run history is lost Only what the job produces
SQL database in Fabric Nothing cross-region: backups are zone- or locally redundant, with no geo-replicated backups Schema (database project) The database, from a .bacpac you exported to another region beforehand Yes: the restored database reflects the time of the export; later changes must be replayed
Semantic model Covered by Power BI BCDR: read-only during failover; can’t export or back up during failover Model definition (listed as Preview in the Git supported-items list) Data source bindings, refresh schedules, credentials Refresh after redeployment
Report Covered by Power BI BCDR: viewable read-only during failover Report definition (listed as Preview, with exceptions) Binding to the recovered semantic model No
Eventhouse / KQL database Not covered: stored outside OneLake Definitions (tables, mappings, policies) Microsoft recommends independent databases in two regions with parallel ingestion Yes, unless you ingest in parallel
Eventstream Nothing; data safety depends on the destinations Definition Sources and destinations in the recovery region; optionally an active/active deployment Through the destinations’ recovery
User data function Nothing Function source and project assets (Microsoft’s recommended path) Dependencies, runtime settings, authentication; execution history is lost No, but its data stores must recover first
API for GraphQL Definitions and configuration aren’t replicated Definition, with environment-specific source workspace and item IDs to update Roles, authorization, RBAC, introspection and monitoring settings; consumer endpoints No, but its sources must recover first
Mirrored database Settings aren’t replicated Definition The mirrored database, in a workspace in another region Re-mirroring from the source
Copy job Nothing Definition (Microsoft’s DR guidance labels this approach as Preview) Lakehouse references Rerun for the missing window
Variable library Nothing Definition and value sets Select the active value set again No

What the table says, in one sentence: only data in OneLake can come back without you, and everything that makes it usable comes back through Git, scripts or people.

Recovery outside a regional disaster is usually item-level and in-region, with its own tools:

Item In-region recovery Retention (checked November 2026)
Warehouse Restore in place from a system restore point (every eight hours while the warehouse is active; Microsoft states an eight-hour RPO) or a user-defined restore point; time travel queries 30 days by default, configurable from 1 to 120 days; at least 20 restore points kept with the default. Only workspace Admins can restore; you can’t restore into a new warehouse
Lakehouse Delta time travel and RESTORE until VACUUM removes old files; OneLake soft delete for files Soft delete: seven days by default, configurable per workspace
SQL database in Fabric Point-in-time restore to a new database in the same workspace 7 days by default, up to 35. Microsoft documents that deleting the workspace deletes its databases unrecoverably
Deleted items Workspace recycle bin or REST API; properties, configuration and relationships are restored, but not shared item permissions On by default with three days for tenants that never configured it; 3 to 90 days. Supported item types only
Deleted workspaces Restore from the admin portal Seven days by default, 7 to 90 days

One interaction to know: Microsoft documents that a Git sync or deployment pipeline can re-create an item that was soft-deleted, as an empty definition without data. If you need the data, restore the soft-deleted item first, delete the copy Git created, then sync again.

Git is not backup

Git protects definitions and code, not your data.

Fabric Git integration is the backbone of item recovery: Microsoft’s guidance for notebooks, user data functions, GraphQL, Copy jobs, Activator, variable libraries and deployment plans all starts with “connect the workspace to Git”. It is also routinely misunderstood as a backup.

What source control helps recover:

  • Code: notebooks, Spark job definitions, user data functions, SQL objects.
  • Supported item definitions: pipelines, dataflows, Lakehouse metadata and shortcut definitions, Warehouse schema, semantic models and reports (where supported).
  • Configuration as code: variable libraries and their value sets, parameter files, deployment scripts.
  • Deployment structure: workspace folders and, with a deployment plan, the order in which items deploy.

What it generally does not recover:

  • Data: Lakehouse tables and files are explicitly not tracked; Git operations never touch them.
  • Run state: run history, schedules’ execution history, Spark job history, workspace monitoring data.
  • Secrets and credentials: connection credentials, Key Vault contents, gateway credentials.
  • Runtime state: watermarks and checkpoints, unless you store them as data you also protect.
  • Some permissions and settings: workspace role assignments, item shares and many item settings. Even where security definitions are tracked, such as OneLake security data access roles, Microsoft notes that Entra member IDs aren’t stored in Git; members carry over only between workspaces where role names match.
  • External dependencies: gateways, private endpoints, source system configuration, downstream applications.

Also note the Preview labels: the Git supported-items list still marks several items, including semantic models and reports, as Preview (checked November 2026). Recovery that depends on a Preview capability should be tested more often, and documented with a fallback.

For the deployment side, see Microsoft Fabric CI/CD with fabric-cicd, which deploys from Git with a service principal and parameterized environments. To see in practice what Git, deployment pipelines and Warehouse database projects do and don’t carry between environments, work through Deploy a Fabric Solution with Git, Deployment Pipelines and Database Projects. The same pipeline that promotes to production is the pipeline that rebuilds production elsewhere, if its target workspaces, capacities and parameters can be pointed at a recovery environment.

Rebuild architecture

A regional recovery is a rebuild in a healthy region. The steps below are the architecture of that rebuild, not a click path.

  1. 1Create recovery capacityIn a healthy region with available compute
  2. 2Create workspacesSame names where code depends on them
  3. 3Connect GitSame repository and branch
  4. 4Restore item definitionsSync or deploy from Git; recreate the rest
  5. 5Recover dataCopy from replicas, restore, or replay
  6. 6Restore connections and secretsCredentials, gateways, workspace identities
  7. 7Rebind dependenciesItem IDs, endpoints, schedules, parameters
  8. 8Validate securityRoles, data rules, tenant isolation
  9. 9Validate dataCounts, totals, freshness, duplicates
  10. 10Resume consumersAPIs, models, reports, downstream systems
Regional rebuild flow. The order is driven by dependencies, not by importance. Consumers resume last, after security and data have been validated, because a fast recovery that leaks or serves wrong numbers is a second incident.

Three design decisions make this flow fast or slow:

  • Names vs IDs. Microsoft recommends recreating workspaces and items with the same names, because code that refers to items by name keeps working. Anything that refers to workspace or item IDs breaks: new workspaces and items get new IDs. Variable libraries and deployment parameters turn that from a code change into a configuration change.
  • Scripted vs manual. Every manual step is minutes multiplied by the number of workspaces and tenants. Creating capacity, workspaces, Git connections, role assignments and connections should be scripts you have run before.
  • Pre-provisioned vs on demand. Creating capacity during a regional event competes with every other customer doing the same. Pre-provisioning a small recovery capacity and scaling it at recovery time is a cost decision that buys RTO.

Data recovery and replay

Copying replicated files is one way to recover data. A resilient platform should also know how to rebuild state from what it ingested, because replication can be inconsistent, late or absent, and because most data incidents (bad loads, corrupted tables) aren’t regional at all.

Capability What it gives recovery Design rule
Replay from source Reload any window the source still holds Agree retention with source owners; record what was loaded per window
CDC replay Resume from a log position instead of reloading everything Store the last applied position durably; know the source’s change retention and alert before it’s exceeded
Watermark restart Restart incremental loads from the last committed point Advance the watermark only after the target commit succeeds
Raw landing retention A copy of exactly what arrived, independent of transformations Keep raw at least as long as the longest realistic reprocessing need; protect it as the system of record when sources can’t resend
Immutable source history Append-only raw files never change after landing No in-place updates in raw; corrections arrive as new files
Idempotent loads The same window can be loaded twice safely Deterministic keys and MERGE, or replace-by-partition
Merge and upsert Replays update rather than duplicate Match on business keys plus tenant ID; never on surrogate keys generated at load time
Duplicate prevention Overlapping replay windows don’t double-count Deduplicate on key and source version before merging
Checkpoint and state recovery Streams and incremental loads resume where they stopped Checkpoints and watermarks in durable, replicated storage, not only in the processing item
Downstream consistency Gold tables and models reflect a coherent point in time Rebuild downstream layers after upstream recovery completes, in dependency order

The order of recovery for data is usually: restore or copy what is consistent, find the last point you can prove is complete, then replay everything after it from raw landing or the source. The execution log is what tells you that point: Designing an Execution Log for Data Pipelines records which windows each run committed, and Incremental Loads in Microsoft Fabric shows how to advance watermarks so that “committed” really means committed.

Idempotency during recovery

Every recovery step that writes data has to answer one question:

Can I run this workload twice without corrupting data?

During recovery, the answer is tested more than at any other time: windows overlap, operators rerun steps that may have partly succeeded, and replicated data may already contain some of the rows you are about to replay.

Technique What it protects against Watch out for
Deterministic keys The same source row always produces the same key Keys generated at load time (identity columns, random UUIDs) create a new row on every replay
MERGE / upsert Reloading a window updates rows instead of duplicating them MERGE with duplicate source keys fails or behaves unpredictably; deduplicate first
Replace by partition Rewriting a whole day or tenant partition is naturally repeatable The partition must contain only that window and tenant
Append-only with dedup Raw and event data stay complete Readers must deduplicate, or a downstream step must
Transaction boundaries A failed step leaves no partial result A notebook that writes several tables has several commits; design the step so each table can be rerun independently
Retry-safe side effects Notifications, exports and API calls aren’t repeated Record side effects in the log and check before repeating

Be careful with exactly-once language. Fabric pipelines retry activities, Spark retries tasks, and event sources commonly deliver at least once. Unless you have measured a specific end-to-end guarantee, design for at-least-once delivery and make the write idempotent; that is a property you can test.

A tenant-scoped, idempotent replay into a curated Delta table, as a sketch with generic names:

Requête SQL: Idempotent tenant replaySpark SQL (Delta Lake)
-- Replay one tenant's window from raw landing into the curated table.
-- Safe to run twice: duplicates are removed first, and MERGE matches on business key + tenant.
MERGE INTO curated.orders AS target
USING (
  SELECT tenant_id, order_id, order_status, amount, source_version, source_modified_at
  FROM (
    SELECT *,
           ROW_NUMBER() OVER (PARTITION BY tenant_id, order_id
                              ORDER BY source_version DESC) AS rn
    FROM raw.orders_landing
    WHERE tenant_id = :tenant_id
      AND source_modified_at >= :window_start
      AND source_modified_at <  :window_end
  ) AS ranked
  WHERE rn = 1
) AS source
ON  target.tenant_id = source.tenant_id
AND target.order_id  = source.order_id
WHEN MATCHED AND source.source_version > target.source_version THEN
  UPDATE SET order_status       = source.order_status,
             amount             = source.amount,
             source_version     = source.source_version,
             source_modified_at = source.source_modified_at
WHEN NOT MATCHED THEN
  INSERT (tenant_id, order_id, order_status, amount, source_version, source_modified_at)
  VALUES (source.tenant_id, source.order_id, source.order_status, source.amount,
          source.source_version, source.source_modified_at);
Spark SQL (Delta Lake)27 lignesAffichage en lecture seule. Aucune connexion à une base de données.

Notes

  • lignes 11–13:tenant_id, :window_start and :window_end are named parameter markers, supplied when the statement is run.
  • ligne 19The version guard: a replay never overwrites a newer row with an older one.

Three properties make it recovery-safe: the tenant predicate is in the source query and the join condition, the version check stops an older replay from overwriting newer data, and running it twice produces the same table. Incremental Loads in Microsoft Fabric covers reruns, deletes and late data in more depth.

Recovery order and dependencies

Recovery order matters because almost every item depends on something else: a notebook needs its Lakehouse and its connection, a semantic model needs its tables, an API needs its model or Warehouse and its identity. Recovering in the wrong order wastes time on failures that are only waiting for a dependency, and can make things worse: a pipeline resumed before its watermarks are restored will happily reload from the beginning, or skip a window.

  1. 1. Identity and secretsNothing else can authenticate without them
    • Service principals
    • Key Vault access
    • Workspace identities
  2. 2. Capacity and workspaces
    • Recovery capacity
    • Workspaces and roles
    • Git connections
  3. 3. Core storage and base data
    • Lakehouses
    • Warehouses
    • Raw landing
    • Watermarks and checkpoints
  4. 4. Transformations and pipelines
    • Notebooks
    • Pipelines
    • Connections and schedules
  5. 5. Serving
    • Semantic models
    • Data APIs
    • User data functions
  6. 6. Consumers
    • Reports
    • Applications
    • Downstream systems
Recovery dependency layers. Recover top to bottom. Each layer depends on the ones above it being available and validated. Identity and secrets come first because every other layer needs them to connect.
Step Recover Depends on Ready when
1 Identity and secrets Entra ID, Key Vault (their own resilience) Recovery identities can authenticate and read required secrets
2 Capacity and workspaces Step 1 Workspaces exist on a running capacity, with roles assigned and Git connected
3 Core storage Step 2 Lakehouses and Warehouses exist with schemas deployed
4 Base data Step 3, replicas or sources Raw and curated tables restored to a known consistent point; watermarks restored
5 Transformations Steps 3–4, connections Notebooks run against recovered storage and produce expected outputs
6 Pipelines Step 5, schedules held Pipelines run manually once; schedules enabled only after validation
7 Semantic models Step 4–5 data Models deployed, bound and refreshed
8 APIs and applications Steps 4–7, identities Endpoints and connection strings updated; authorization verified
9 Reports and consumers Steps 7–8 Reports bound to recovered models; consumers notified

Hold schedules until validation. A recovered pipeline that runs on its normal schedule before data is validated can overwrite restored tables, advance watermarks past unrecovered windows, or publish incomplete numbers.

Multi-tenant disaster recovery

On a multi-tenant platform, recovery has a second job besides bringing the platform back: recovery must not weaken tenant isolation. Recovery runs with elevated permissions, under time pressure, using scripts that touch every tenant’s data at once. That is exactly the situation in which tenant boundaries fail. The Microsoft Fabric Security Architecture guide describes the isolation patterns this section assumes: shared store, schema separation, workspace per tenant and dedicated capacity.

  • Shared recovery pathShared store and shared workspaces
    1. One recovery identityReaches every tenant's data
    2. Bulk data copyAll tenants in one operation
    3. Shared replayTenant predicate in every statement
    4. Platform-wide validationPlus cross-tenant leak tests
  • Isolated tenant recoveryWorkspace per tenant or dedicated
    1. Per-tenant identityScoped to one tenant's workspace
    2. Tenant-ordered rebuildHighest tier first
    3. Tenant secrets reboundEach tenant's own connections
    4. Per-tenant validationBefore that tenant reopens
Shared recovery path vs isolated tenant recovery. A shared path recovers everyone at once with one identity and one validation. An isolated path recovers tenants one by one, in priority order, with their own identities, secrets and checks. Most platforms need both, by tier.

What multi-tenant recovery has to cover:

Topic Design rule
Tenant-safe replay Every replay statement carries the tenant predicate in both the source and the target condition, as in the MERGE above
Tenant-safe restore Restoring a shared table restores every tenant in it. Know which tenants a restore point affects, and replay the others forward
Tenant-specific RPO and RTO Contracts may differ by customer; recovery plans must say which tenant gets which target
Tenant priority tiers Recovery order follows tiers, not alphabetical lists or whatever the script does first
Shared vs dedicated recovery Shared stores recover once for everyone; dedicated tenants recover independently, in parallel if capacity allows
Workspace-per-tenant recovery Only scripted provisioning scales: workspace, roles, identity, connections and items per tenant, from the same code used for onboarding
Shared-store recovery One recovery, but validation per tenant: counts and totals by tenant ID, not only overall
Tenant context preservation Staging tables, recovery copies and replay inputs keep the tenant ID; nothing derives it from paths or file names after the fact
Cross-tenant contamination risk The recovery identity is the most powerful identity on the platform during the incident; scope it, log it, and remove it afterwards
Tenant-specific secrets Each tenant’s source credentials are restored to that tenant’s connection, and only there
Tenant-specific validation Each tenant is checked for completeness and for isolation before its access is reopened

How recovery breaks isolation, in practice:

  • A replay omits the tenant filter, so one tenant’s window is reloaded with every tenant’s rows, or merged into the wrong partition.
  • Restored staging data loses its tenant ID, because the recovery copy came from a path-based folder layout and the tenant was inferred from the folder name.
  • A shared recovery identity can access all tenants, and stays in place after the incident because nobody removed it.
  • Tenant-specific secrets are rebound incorrectly, so a connection for one customer authenticates with another customer’s credential and loads its data.
  • Recovered shortcuts point to the wrong tenant store, because a shortcut definition restored from Git kept an external target that isn’t remapped per environment or per tenant.
  • Semantic model roles are recreated without members, so a recovered RLS role either filters everyone out or, worse, someone grants broad access “temporarily” to unblock reports.

Multi-tenant recovery checklist

  • Tenant priority recorded: every tenant has a recovery tier, owner and contact.
  • Tenant ID survives recovery: in raw landing, staging, recovery copies and replay inputs.
  • Replay statements reviewed: tenant predicate present in source and target conditions.
  • Recovery identity scoped: per tier or per tenant where isolation requires it; time-bound and logged.
  • Secrets mapped per tenant: a documented mapping from tenant to connection to secret, checked after rebinding.
  • Shortcut targets verified: every recovered shortcut points to the intended tenant’s store.
  • Per-tenant validation: counts, totals and freshness by tenant, not only platform-wide.
  • Cross-tenant tests rerun: automated tests prove one tenant can’t read another through each path, before reopening.
  • Workspace-per-tenant provisioning scripted: the same automation as onboarding, tested in recovery mode.

Tenant recovery priority

Different tenants can have different SLAs, RPOs, RTOs, regulatory requirements and business criticality. A recovery plan that treats them identically either overspends on small tenants or under-protects large ones.

Tenant tier Example RPO Example RTO Recovery pattern
Tier 1: contractual, high value 15 minutes 2 hours Dedicated workspace and capacity; pre-provisioned recovery capacity; continuous replay; recovered first
Tier 2: standard 4 hours 8 hours Workspace per tenant on shared capacity; scripted rebuild; recovered after tier 1
Tier 3: shared, low volume Previous daily load Next business day Shared store and workspace; recovered together after tier 2
Internal or trial Best effort Best effort Rebuilt from Git when capacity allows

These values are illustrative, not recommendations: real targets come from contracts and business impact analysis.

Priority shapes the architecture before it shapes the runbook:

  • Workspace per tenant lets you recover tier-1 tenants independently and first. A shared store forces everyone to wait for the same restore and the same validation.
  • Dedicated capacity for top tiers means their recovery doesn’t compete for compute with everyone else’s, and the disaster recovery setting can be enabled only where the targets require it, because the setting is per capacity.
  • Sequencing follows tiers: identities, workspaces and data for tier 1 first, then tier 2, with validation gates per tenant.
  • Cost follows tiers too: standby capacity, geo-replication and parallel ingestion for tier 1 only, if contracts pay for it.

Security during recovery

A recovered platform is not production-ready until its security boundaries are revalidated.

Recovery recreates resources. It doesn’t necessarily recreate every permission, role member or setting, and the documentation is explicit about several gaps:

Area What recovery does or doesn’t bring back What to do
Identity restoration Entra ID identities are outside the Fabric region; they’re available if Entra ID is Verify the recovery identities and break-glass accounts before the incident
Service principals Unchanged in Entra ID, but they hold no roles in the new workspaces Reassign workspace roles and item permissions by script
Workspace identity A new workspace has no workspace identity until you create one, and a restored workspace doesn’t get its identity back Create the identity, then regrant it on every source it authenticated to (storage, databases, trusted workspace access)
Key Vault Your vault’s own resilience applies; Fabric doesn’t copy secrets Use a geo-resilient vault design; document which secrets each connection uses
Credentials Connection credentials aren’t in Git and don’t follow items to a new workspace Recreate connections with credentials from the vault, not from memory
Workspace roles (RBAC) Not part of item definitions Assign from code; compare with the documented role matrix
SQL permissions Recreated only if your Warehouse project includes them Keep GRANT, RLS and masking definitions in source control and deploy them
RLS and OLS Model definitions can include roles, but members must be reassigned; OneLake security role definitions can be in Git (Preview) without Entra member IDs Reassign members from groups; test with real identities
Item shares Item recovery from the recycle bin doesn’t restore shared item permissions Re-share deliberately, from the access model, not from requests
Tenant boundaries Only as good as the replay, the secrets mapping and the shortcut targets Rerun cross-tenant tests before reopening any tenant
Emergency access Recovery often needs elevated rights Time-bound, logged, named, and removed when recovery closes
Customer-managed keys During failover, CMK supports read operations as long as Key Vault is healthy and permissions are intact; enabling, disabling or updating the key isn’t supported during failover Keep the key vault and its access policies resilient; don’t plan key rotation during a regional event

Also check what the original region’s controls were doing. Workspace outbound access protection rules and OneLake resource instance rules stay readable or enforced for reads during failover, but can’t be changed until the workspace is writable again; in a new recovery workspace they must be configured again. Private links, managed private endpoints and gateway configurations belong to the recovery environment’s network design, not to the item definitions.

The security checks that a recovery adds are the same ones the Data Platform Security Architecture Review Checklist lists for a new platform, especially access paths, service identities and tenant isolation. A recovered platform is, in practice, a new platform that has to pass the same review quickly.

Secrets and endpoint rebinding

A recovered platform has different identifiers than the original. Everything that pointed at the old ones must be found and changed, and that list should exist before the incident:

What changes Who depends on it Rebinding approach
Workspace IDs Notebooks with ABFS paths, GraphQL source references, scripts, APIs Resolve IDs from names at runtime, or from variable libraries and parameters
Item IDs Semantic models, shortcuts, pipelines invoking other items, external tools Deployment parameters; post-recovery rebinding script
SQL endpoints Applications, reporting tools, SSMS users Connection strings in configuration, not code; a DNS or configuration indirection you control
Connections Pipelines, dataflows, Copy jobs, semantic models Recreate connections by script; record the mapping from item to connection
Gateways On-premises and virtual network data sources Gateway cluster available to the recovery region; Microsoft notes gateways don’t function during a Power BI failover
Credentials Every connection From Key Vault; rotate afterwards if the incident involved credential exposure
APIs Customer applications, partner integrations Stable public endpoint in front of the data API; backend target changes, client URL doesn’t
GraphQL and user data functions Applications calling them Microsoft notes that downstream systems might require endpoint rebinding, and GraphQL source IDs must be updated
External consumers Downstream databases, exports, partner feeds A consumer register with owners and contact paths

Document rebinding before an incident. An undocumented dependency is found when a consumer complains, which is the slowest and least controlled discovery mechanism available. The fastest platforms keep a machine-readable inventory: item, what it references, how the reference is set, and who consumes it.

Observability during recovery

The recovery process itself should be observable. During an incident, people ask the same questions repeatedly: what has been recovered, which tenants are back, what failed, what is the RPO we actually achieved? A recovery execution record answers them from data instead of from a chat thread.

Field Purpose
recovery_event_id Identifies the recovery as a whole
incident_id Links to the incident or ticket
step The recovery step, from the runbook’s list
item_name, item_type What was recovered
tenant_id Which tenant, when the step is tenant-scoped
started_at, ended_at Duration, for RTO measurement
status Started, succeeded, failed, skipped
validation_result Outcome of the step’s validation
operator The person or identity that ran it
data_checkpoint The recovery point reached: watermark, restore point, failover timestamp
error_message What failed

A minimal table, aligned with the execution log’s identifiers so that replay runs and recovery steps can be joined:

Requête SQL: ops.recovery_executionSpark SQL (Delta Lake)
CREATE TABLE IF NOT EXISTS ops.recovery_execution (
  recovery_event_id   STRING    NOT NULL,  -- one per recovery
  incident_id         STRING,
  step_order          INT       NOT NULL,
  step                STRING    NOT NULL,  -- e.g. 'restore_definitions', 'replay', 'validate_security'
  item_name           STRING,
  item_type           STRING,
  tenant_id           STRING,              -- null for platform-wide steps
  execution_id        STRING,              -- the replay run in the execution log, if any
  started_at          TIMESTAMP NOT NULL,
  ended_at            TIMESTAMP,
  status              STRING    NOT NULL,  -- started | succeeded | failed | skipped
  validation_result   STRING,
  operator            STRING    NOT NULL,
  data_checkpoint     STRING,              -- watermark, restore point or failover timestamp
  error_message       STRING
) USING DELTA;
Spark SQL (Delta Lake)17 lignesAffichage en lecture seule. Aucune connexion à une base de données.

Two details matter. The record must live somewhere that survives the incident, which during a regional recovery means the recovery region, not the failed one. And replay runs should write their normal execution log records with the same correlation ID as the recovery step that started them, as described in Designing an Execution Log for Data Pipelines. Remember also that workspace monitoring data doesn’t carry over to a recovered workspace: enable it again, and expect a new monitoring baseline.

Validation after recovery

Recovery ends when the platform is proven correct, not when the last step turns green.

Area Check Evidence Pass criterion
Platform Capacity active Capacity state, metrics app Running, sized for the recovery load
Workspaces available Workspace list, role assignments Every expected workspace exists with its role matrix
Connections restored Test connection per data source Every connection authenticates with the intended identity
Data Row counts Counts by table and tenant vs source or pre-incident baseline Within agreed tolerance
Business totals Revenue, quantities or balances vs source system Match, or differences explained by the RPO window
Freshness Latest business timestamp per table Within the RPO
Completeness Execution log: every window committed No gaps between the recovery point and now
Duplicates Key uniqueness checks Zero duplicates on business keys
Tenant boundaries Rows per tenant per table No rows in another tenant’s partition or workspace
Security Roles Workspace and item roles vs the documented matrix Exact match; no temporary broad grants left
Permissions SQL grants, RLS and OLS members, OneLake security roles Rules present and members assigned
Identities Service principals and workspace identities Expected identities only, with expected scope
Tenant isolation Automated cross-tenant tests Every test fails to read another tenant’s data
Operations Pipelines Manual run of each critical pipeline Succeeds against recovered storage
Schedules Schedule list Enabled only after validation, at intended times
Alerts Test alert Reaches the on-call person
Monitoring Workspace monitoring and execution log Enabled and recording in the recovery region
Consumers Semantic models Refresh and a known query Refreshes; returns validated totals
Reports Key reports opened by business owners Correct numbers, correct filtering
APIs Contract tests per tenant Correct data, correct tenant scope
Downstream systems Consumer register Each consumer confirmed or notified

Recovery testing

A DR plan that has never been tested is an assumption.

Testing is where RTO stops being a number in a document. Use several kinds of test, because they answer different questions:

Test What it proves Frequency (illustrative)
Tabletop exercise People know their roles, decisions are pre-approved, the runbook is understandable Twice a year, and after major changes
Partial recovery drill One workspace or item type can be rebuilt from Git and data in a separate environment Quarterly
Full regional simulation The whole platform can be rebuilt in another region within the RTO Yearly, where practical and justified by the targets
Tenant-isolation test Recovery scripts and replays don’t leak across tenants With every drill
Replay test A window can be replayed twice with identical results With every change to load logic
Credential recovery test Secrets can be rotated and identities recreated without breaking loads Twice a year
Recovery time measurement How long each stage actually takes Every drill

A simulation doesn’t need to fail over Fabric. Rebuilding the production platform into an empty workspace on another capacity, from Git and from copies of the data, exercises almost every step that a real regional recovery needs.

Compare measured recovery to the target per stage, not only in total. A drill that meets the RTO with 10 minutes to spare because one engineer remembered an undocumented step hasn’t met it; record the gap and fix the runbook. The smaller, more frequent recovery tests (rerun, replay and idempotency checks that belong in every release) are covered in Testing a Production Data Platform, and the recovery-replay test itself (rebuild from retained batches, compare with a full-load result) in Testing Incremental Loads and Idempotency.

RTO decomposition

An RTO is the sum of stages, and each stage has its own owner and its own levers. Breaking the target into stages turns “we need four hours” into a planning framework.

  1. ProvisionCapacity, workspaces, roles
  2. DeployItem definitions from Git
  3. RecoverData copied, restored or replayed
  4. RebindConnections, secrets, endpoints
  5. ValidateSecurity, data, consumers
  6. ResumeSchedules and access reopened
RTO decomposition. The RTO is the sum of these stages, plus the time to decide to recover. Measure each stage in drills; the slowest one is where automation pays off first.
Stage What drives its duration Target Measured
Decide Detection, escalation, authority to declare recovery
Provision Capacity availability in the recovery region; scripted vs manual workspace creation
Deploy Number of items; Git sync or deployment automation; manual recreation for unsupported items
Recover data Data volume, copy throughput, replay windows, source retention
Rebind Number of connections, secrets and consumers; whether references use names, IDs or configuration
Validate Automated vs manual checks; number of tenants
Resume Schedule enablement, consumer notification, staged reopening by tenant
Total Sum of the above The agreed RTO From the last drill

Fill in targets by working backwards from the RTO, and measured values from drills. Don’t borrow timings from other platforms or from vendor statements: Microsoft’s “typically less than one hour” describes its failover to a read-only service, which is not one of your stages at all.

Recovery cost architecture

Recovery is a trade between money spent before an incident and time lost during it.

Lower cost

  • Reconstruct when needed: no standby resources; rebuild from Git and sources at recovery time.
  • Source replay: rely on sources and raw landing retention instead of replicated curated data.
  • Fewer standby resources: no pre-provisioned capacity; the disaster recovery setting only where raw data is the only copy.

Faster recovery

  • Pre-created capacity: a small capacity in the recovery region, scaled at recovery time.
  • Duplicated data: the disaster recovery setting on production capacities, or dual ingestion into a second region for the most critical data.
  • Dedicated recovery environment: workspaces, connections and identities kept ready in the recovery region.
  • Automation: every step scripted, tested and timed.

Lower RTO usually costs more.

The cost of the disaster recovery setting appears as BCDR Storage and BCDR Operations in the Capacity Metrics app, and standby capacity costs whatever its SKU costs while it exists. Price them with current Microsoft pricing for your region rather than from rules of thumb, and put them next to the business cost of the RTO they buy. The decision belongs to the business owner; the architect’s job is to make both numbers visible.

Runbooks

Architecture and runbooks answer different questions, and both are needed:

Architecture

What recovery capability exists: replayable sources, Git coverage, scripted provisioning, tenant tiers, standby resources, validation tests. It is designed once and changed through review.

Runbook

What operators do during the incident: the steps in order, the decisions and who makes them, the evidence to collect, the checks before reopening. It is followed under pressure, so it stays short and concrete. The site’s Fabric capacity saturation runbook shows the format.

Recommended runbooks to write from this architecture:

  • Fabric regional recovery: declare, provision, deploy, recover, rebind, validate, resume, with the recovery execution record.
  • Corrupted data and replay: identify the bad window, restore to the last good point, replay, reconcile.
  • Failed deployment rollback: redeploy the previous version from Git, and decide whether data changed by the release needs a restore or a forward fix.

Recovery architecture decision framework

Use the same twelve steps for each workload or customer tier:

Step Question Output Validation criterion
1. Business process What does this workload support, and who owns it? Workload, owner, impact of outage Owner has signed off the impact description
2. Failure scenarios Which failure domains apply? Scenarios from job to region Each scenario has a recovery path
3. RPO How much data loss is acceptable? RPO per workload or tier Agreed by the business owner
4. RTO How long can it be unavailable? RTO per workload or tier Agreed by the business owner
5. Dependencies What must recover first? Dependency layers No circular or unknown dependencies
6. Recovery source Where does the data come back from? Replica, restore point, raw landing or source Source retention covers the RPO plus the RTO
7. Rebuild method How do items come back? Git, scripts, templates, manual Every production item has a method
8. Security How are identities, secrets and isolation restored? Identity, secret and tenant mapping Security validation is scripted
9. Validation How do we know recovery is correct? Validation matrix Checks automated where possible
10. Test frequency How often is it proven? Test calendar Last test date within the agreed interval
11. Cost What does the chosen design cost? Standby and replication cost vs outage cost Approved by the business owner
12. Owner Who maintains and executes it? Named owner and deputy Owner has run the last drill

Record the decisions as architecture decision records, like any other platform decision, so the reasoning survives the people who made it.

Recovery architecture review checklist

Use this before approving a production platform, a new workload or a new customer tier. Every unchecked item is either a decision to make or a risk to accept in writing.

Requirements

  • RPO and RTO agreed per workload: signed by the business owner, including tenant-specific targets.
  • Recovery scope defined: which failure domains the plan covers, and which it explicitly doesn’t.

Failure domains

  • Recovery path per failure: job, item, workspace, capacity, zone, region, source, identity, network and external dependency.
  • Home region and pairing understood: the paired region supports Fabric, or a custom cross-region plan exists.

Data

  • Recovery source per dataset: replica, restore point, raw landing or source replay.
  • Source retention verified: long enough to cover the RPO plus the time to recover.
  • Loads idempotent: every load can run twice with identical results.
  • Watermarks and checkpoints protected: stored as data and recovered with it.
  • Disaster recovery setting decided per capacity: with the reason, cost and the 30-day change constraint.
  • Data outside OneLake covered: KQL databases, SQL databases and external stores have their own approach.

Code and configuration

  • Definitions in Git: every supported production item, with unsupported items exported or scripted.
  • Environment values externalized: variable libraries or parameters, no hard-coded IDs.
  • Preview dependencies documented: with a fallback if the Preview capability fails.

Identity and security

  • Recovery identities ready: scoped, tested, time-bound when elevated.
  • Role matrix as code: workspace roles, item permissions, SQL permissions and data-role members reassignable by script.
  • Workspace identities and grants documented: to recreate and regrant on every source.
  • Secrets recoverable: vault resilience and the mapping from connection to secret documented.

Multi-tenant

  • Tenant tiers and order: every tenant has a recovery tier.
  • Tenant-safe replay and restore: tenant predicates reviewed; tenant ID preserved through recovery copies.
  • Cross-tenant tests part of recovery: run before any tenant reopens.

Dependencies

  • Dependency layers documented: recovery order from identity to consumers.
  • Rebinding inventory: every reference to workspace IDs, item IDs, endpoints and connections, with its consumers.

Recovery procedure

  • Runbooks written: regional recovery, data replay, deployment rollback.
  • Provisioning scripted: capacity, workspaces, Git connections, roles and connections.
  • Schedules held until validation: documented in the runbook.

Validation

  • Validation matrix: platform, data, security, operations and consumers, with pass criteria.
  • Recovery execution record: stored in the recovery region, linked to the execution log.

Testing

  • Drills scheduled: tabletop, partial and full where justified, with dates.
  • RTO measured per stage: the last measured total is within the target.

Cost

  • Standby and replication costs known: compared with the outage cost and approved.

Ownership

  • Owners named: for the recovery architecture, each runbook and each tenant tier.
  • Review date set: the plan is reviewed after major platform changes and at least yearly.

Recommendation

Start from the business: RPO and RTO per workload and per tenant tier, agreed by the people who own the impact. Then design the platform so that most recoveries are routine: restartable, idempotent loads; protected raw landing and watermarks; every definition in Git; environment values in configuration. Enable the capacity disaster recovery setting where data would otherwise be lost or take too long to recompute, and treat what it copies as material to reconcile, not as a recovered platform. Script the rebuild, keep a rebinding inventory, recover in dependency order, and hold schedules until validation. On a multi-tenant platform, recover by tier and never let recovery weaken isolation. Then measure: a recovery time is real only after a drill has produced it.

Sources

Fabric behavior checked against Microsoft Learn in November 2026. Disaster recovery support, Git integration coverage and item recovery are tracked per item and change often, and several capabilities referenced here are in Preview. Recheck the linked pages before relying on a specific behavior, retention period or limit.

Étiquettes

  • Microsoft Fabric
  • Architecture de données
  • Multi-locataire
  • CI/CD
  • Fabric Lakehouse
  • Fabric Warehouse