Microsoft Fabric
Recovery and Disaster Recovery Architecture for Microsoft Fabric: RPO, RTO, Replay and Rebuild
An architect's guide to recovery for production Microsoft Fabric platforms: RPO and RTO that the design can actually meet, failure domains, what Microsoft recovers and what you rebuild, the capacity disaster recovery setting, item-by-item recovery, replay and idempotency, multi-tenant recovery, security revalidation, RTO decomposition and recovery testing.
En este artículo
A Fabric capacity with the disaster recovery switch turned on is not a recoverable platform. The switch copies OneLake files to another region. It doesn’t recreate your notebooks, reconnect your pipelines, rebind your applications, restore your workspace identities, or tell you whether the data that arrived is complete. This guide answers one question:
If a production Fabric platform, workspace, item, dependency or region becomes unavailable, what can Microsoft recover automatically, what must I rebuild or replay, and how do I design for the RPO and RTO the business actually requires?
The architectural principle. Microsoft-managed availability is not the same thing as application recovery. Microsoft keeps the service running, spreads it across availability zones and, if you opt in, keeps a second copy of OneLake data. Your platform is more than that: item definitions, connections and credentials, run state and watermarks, downstream applications, security boundaries and the business’s trust in the numbers. An architect has to separate seven things that a single word, “recovery”, tends to blur:
| Layer | The question | Typically owned by |
|---|---|---|
| Service availability | Is Fabric itself up and reachable? | Microsoft |
| Data redundancy | Does a copy of the bytes exist somewhere? | Microsoft, partly opt-in |
| Item and configuration recovery | Do notebooks, pipelines, models and settings exist again? | You |
| Pipeline and workflow state | Do loads know where they stopped, and can they resume? | You |
| Downstream application recovery | Do APIs, reports and consumers point at the right place? | You |
| Identity and security recovery | Are identities, roles, secrets and tenant boundaries correct again? | You |
| Business validation | Are the numbers complete and correct enough to reopen? | You and the business |
And one more principle runs through the whole guide: recovery is a design problem before it is an incident-response problem. Every recovery capability that matters, from replayable sources to rebindable endpoints, has to exist before the incident. During the incident you can only use what you designed.
Where this guide sits: Designing a Production Data Platform introduces RPO and RTO among the platform’s other decisions, and SQL Server to Microsoft Fabric: What Maps to What? explains how backup and high availability change for a SQL Server team. This guide goes further: failure domains, recovery by item, replay, multi-tenant recovery, security revalidation and how to prove a recovery time.
The running example
The examples use a generic multi-customer analytics platform:
- Sources: an operational database read through change data capture, nightly partner files, and an events feed.
- Storage: a Lakehouse holding raw landing and curated Delta tables, and a Warehouse serving a curated star schema.
- Processing: notebooks and pipelines orchestrated on schedules, with an execution log.
- Serving: semantic models and reports for internal users, and a data API serving each customer its own data.
- Tenancy: some customers share workspaces and tables (a tenant ID column), and a few large customers have a dedicated workspace.
The business has agreed on recovery targets per workload: internal finance reporting can tolerate the loss of the last daily load and a next-business-day recovery, while the customer API has a four-hour RTO and the largest customers have contractual targets of their own.
RPO and RTO, defined properly
RPO: Recovery Point Objective
The recovery point objective is the maximum acceptable amount of data loss, measured in time, when an incident happens. After recovery, data newer than the recovery point is either lost or must be reloaded.
| RPO | What it means in practice |
|---|---|
| Zero or near zero | No acknowledged data may be lost. Requires synchronous or continuously replicated state, or a source that can always replay |
| 15 minutes | At most the last quarter hour of changes needs to be replayed or is lost |
| One hour | Hourly batches or a CDC stream that can resume from a known position |
| Previous daily load | Yesterday’s load is the recovery point; today’s must be rerun |
RTO: Recovery Time Objective
The recovery time objective is the maximum acceptable time a workload can be unavailable after an incident. The clock starts at the incident, not when someone decides to recover.
| RTO | What it means in practice |
|---|---|
| 30 minutes | Standby environment and automated failover; no manual rebuilding |
| Four hours | Pre-scripted rebuild, pre-approved decisions, rehearsed runbook |
| Next business day | Manual rebuild is acceptable if it is documented and has been tried once |
A platform cannot have an RPO or RTO merely because the underlying cloud service is resilient. Microsoft’s numbers describe the service: how fast it fails over, how its storage replicates. Your RPO depends on whether your sources can resend data and your loads can resume; your RTO depends on how long it takes to rebuild, rebind and validate your items. The architecture has to support the target, and only a test proves it.
| Requirement | Architectural consequence |
|---|---|
| RPO ≤ 15 minutes | Source replay or replicated state must cover any 15-minute window. Asynchronous geo-replication of OneLake files alone can’t promise it, because Microsoft gives no RPO for it |
| RPO = previous daily load | Raw files or source extracts for the current day must survive the incident, or the source must resend them |
| RTO ≤ 1 hour | Manual recreation of dozens of workspaces, connections and permissions is not credible. Pre-provisioned or fully automated recovery is required |
| RTO ≤ 4 hours | Infrastructure and items deployed by script from Git; data recovery parallelized; validation automated |
| RTO = next business day | A documented manual rebuild is acceptable, if it has been rehearsed at least once |
| Tenant-specific SLA | Recovery order, isolation pattern and standby resources may differ by customer tier |
| Regulatory retention | Raw and audit data need their own protection, independent of the processing items |
Javi Builds guidance: agree RPO and RTO per workload, not per platform. “The platform has a four-hour RTO” hides the fact that the finance model and the customer API recover through completely different paths.
Failure domains
Most recovery work is not about regions. It’s about a job that failed at 03:00, a deployment that broke a view, a table overwritten with bad data, or a credential that expired. A recovery architecture starts with a failure taxonomy, because each failure has a different blast radius, a different owner and a different recovery path.
- 1Local recoveryJob or activityRerun the step from its checkpoint
- 2Item recoveryItemRestore the item, its data or its definition
- 3Item recoveryWorkspaceRestore the workspace or redeploy its items
- 4Platform recoveryCapacityMove workspaces or recover on another capacity
- 5Microsoft-managedAvailability zoneService rebalances; restart interrupted work
- 6Regional recoveryRegionMicrosoft fails over; you rebuild in another region
| Failure | Example | Recovery class | Primary recovery path |
|---|---|---|---|
| Activity or job failure | A notebook fails on a timeout | Local | Retry or rerun from the last committed watermark |
| Pipeline failure | A parent pipeline stops halfway through its children | Local | Rerun the failed children only; idempotent writes make it safe |
| Bad deployment | A release drops a column a view depends on | Item | Redeploy the previous version from Git; restore data only if the release changed data |
| Bad data | A source sends duplicated or wrong rows that get loaded | Item | Restore the affected tables to a point before the load (time travel, restore point), then replay the corrected source |
| Deleted item | A Lakehouse or notebook is deleted by mistake | Item | Item recovery from the workspace recycle bin, within its retention period |
| Deleted or corrupted data | Files removed, or a table overwritten | Item | OneLake soft delete, Delta time travel, Warehouse restore point, SQL database point-in-time restore |
| Workspace issue | A workspace is deleted, or its permissions are broken | Platform | Workspace retention restore, or redeploy from Git into a new workspace |
| Capacity failure | A capacity is paused, throttled or deleted | Platform | Resume, scale, or reassign workspaces to another capacity |
| Zone failure | One availability zone in the region goes down | Microsoft-managed, then local | No customer action for the service; resubmit interrupted Spark jobs and queries |
| Regional outage | The whole region is unavailable | Regional | Microsoft fails over; you create capacity, workspaces and items elsewhere and recover data |
| Upstream source failure | The operational database or partner feed is down | Local, sometimes platform | Pause dependent loads; catch up from the source’s change retention once it returns |
| Credential or identity failure | A secret expires, a service principal is deleted | Platform | Rotate or recreate the identity, regrant access, rebind connections |
| Network failure | A gateway host or private endpoint fails | Platform | Gateway cluster failover, or an alternate connectivity path |
| External dependency failure | A key vault, API or downstream system is unavailable | Platform | That dependency’s own recovery; the platform waits or degrades gracefully |
Two points to take from the table. First, the most frequent failures are local and item-level, and they are recovered by design choices you make anyway for normal operations: restartable loads, idempotent writes, retained raw data and definitions in Git. Second, a regional outage is rare but needs every one of those capabilities at once, plus a few more. A platform that can’t recover from a bad deployment won’t recover from a region.
What Microsoft manages and what you manage
Microsoft describes reliability as a shared responsibility: Microsoft keeps the baseline infrastructure and platform services available, and you design the recovery of your workload. The Fabric reliability guidance is explicit that after a regional disaster you must follow manual steps to restore service.
- Microsoft-managedPart of the service, or opt-in
- Zone distributionFabric resources spread across zones, no configuration
- OneLake storage redundancyZRS where available, LRS elsewhere
- Regional failover decisionInitiated by Microsoft, not by you
- OneLake geo-replicationOpt-in per capacity; asynchronous, file-level
- Power BI BCDROn by default where a paired region supports it
- Customer-managedDesigned, automated and tested by you
- Recovery capacity and workspacesCreated in a healthy region
- Item definitionsRedeployed from Git or recreated
- Data recovery and replayCopied from replicas or reloaded from sources
- Connections, secrets, identitiesRecreated, regranted and rebound
- Consumers and endpointsApplications pointed at new workspaces and items
- Validation and reopeningSecurity, data and business checks before go-live
| Recovery area | Microsoft-managed | Customer-managed |
|---|---|---|
| Infrastructure redundancy | Datacenter hardware, storage replication inside the region | Nothing, beyond choosing a region |
| Availability zones | Automatic distribution and rebalancing of Fabric resources | Resubmitting interrupted Spark jobs and queries; ZRS on your own external storage |
| Fabric service failover | The decision to fail over a region, and the failover itself | Monitoring the status page; executing your recovery plan |
| OneLake replication | Asynchronous geo-replication, if you enabled it on the capacity | Turning it on where justified; accepting that the copy may not be point-in-time consistent |
| Item definitions | Some definitions survive in replicated OneLake files (for example Spark job definition code) | Everything else: Git integration, or exported templates, then redeployment |
| Git source | Nothing: the repository is in your Git provider | Repository availability, branch discipline, committed state |
| Credentials | Credential management for workspace identities | Key Vault secrets, service principal credentials, connection credentials |
| Connections and gateways | Nothing in the recovery workspace | Recreating connections, gateways and their bindings |
| Environment-specific settings | Nothing | Variable library values, parameters, schedules, capacity assignment |
| Run state | Nothing: run history and monitoring data stay with the original workspace | Watermarks and checkpoints in durable storage; your own execution log |
| Custom logging | Nothing | Log tables protected like data; recovery events logged |
| Downstream endpoints | Nothing | Connection strings, item IDs and endpoints updated in every consumer |
| Tenant isolation validation | Nothing | Cross-tenant tests after every recovery |
| Business reconciliation | Nothing | Row counts, totals and freshness checked against sources |
Read the right column carefully: it is most of the work. Microsoft’s responsibility ends at the service and, optionally, the bytes. Nothing in Fabric reconnects your API to a new Warehouse or decides that the recovered numbers are right.
Availability zones vs regional disaster recovery
These are different protections against different failures, and confusing them is the most common design error in Fabric recovery plans.
- Item or workloadYour notebooks, tables, models and APIs
- Zone resilienceAutomatic; survives a datacenter failure
- Regional DROpt-in OneLake geo-replication
- Customer recoveryRebuild, replay, rebind, validate
Zone failure: within a region
Availability zones are physically separate groups of datacenters inside one Azure region. Fabric distributes its resources across zones automatically, with no customer configuration, and OneLake uses zone-redundant storage where the region supports it (locally redundant storage elsewhere).
During a zone outage, Microsoft states that no customer action is required: Fabric capabilities self-heal and rebalance to the healthy zones. The part that still affects you is work in flight: a running Spark job can fail if its driver was in the failed zone, and a Warehouse or SQL analytics endpoint query can fail if its front end was. Those must be resubmitted, which is safe only if the job is restartable and its writes are idempotent. Microsoft also notes that capacities in regions with high demand fluctuations might see higher latency during a zone failure. If your Spark workloads read external ADLS Gen2 storage rather than OneLake, that storage needs its own zone-redundant configuration.
Regional failure: a different event
A regional failure is a broader event that zones don’t cover. Fabric’s response, per the reliability guidance:
- Microsoft initiates the failover. You can’t access the Fabric portal until it completes; a notice appears on the Fabric support page. Microsoft says it typically completes in less than one hour, but that is the time to a read-only service, not to your recovered platform.
- After failover, Fabric is mostly read-only. You can browse workspaces and items, but write operations such as creating or modifying workspaces are paused. Power BI shows reports and dashboards but can’t refresh, publish or change metadata.
- Most items can’t be opened. Lakehouses and Warehouses can’t be opened, but their files are reachable through OneLake APIs and tools, if geo-replication was on. Notebooks can’t be opened and their code content isn’t saved. Pipelines, Dataflow Gen2 and Eventstreams can’t be opened. KQL databases and querysets aren’t accessible.
- Your recovery is a rebuild. Microsoft’s documented recovery path is to create a new capacity (it suggests a region outside your primary geo, because demand is high during such events), create workspaces, create items with the same names, then restore each item following its experience-specific guidance.
Two conditions sit underneath all of this. Cross-region failover applies only where your region has an Azure paired region that supports Fabric; elsewhere, you must build your own cross-region recovery. And the recovery guidance assumes the tenant’s home region is still operational: if the home region itself fails, your recovery waits for Microsoft to recover it first, because core service functionality is tied to the home region.
Availability zones are not regional DR. Zone resilience is the reason you rarely notice datacenter failures. It does nothing for a regional outage, a deleted Lakehouse or a bad load.
The Fabric disaster recovery capacity setting
Fabric has a disaster recovery switch in each capacity’s settings. Checked November 2026 against the Fabric reliability and OneLake disaster recovery documentation:
| Aspect | Current behavior |
|---|---|
| Scope | Per capacity; available for Fabric and Premium capacities where Azure region pairing aligns with Fabric’s presence |
| Who can change it | Capacity administrators |
| Data covered | OneLake data, which includes Lakehouse and Warehouse data |
| Data not covered | Anything stored outside OneLake: KQL databases and querysets need their own approach, and external sources, shortcut targets in other stores and your own Azure resources are unaffected |
| Secondary region | Determined by the Azure region pair; you can’t choose it |
| Replication | Asynchronous and at file level. A failover can lose data and leave files inconsistent with each other; the secondary might not hold a point-in-time-consistent copy |
| RPO | Microsoft states none for OneLake geo-replication |
| Change frequency | After you change the setting, you must wait 30 days before changing it again |
| Start-up time | Replication can take time to start after enabling it or adding workspaces; the capacity settings show a OneLake Geo-replication status per workspace |
| Cost | Extra storage and transactions, billed as BCDR Storage and BCDR Operations and visible in the Capacity Metrics app |
| Power BI | Power BI BCDR applies whether the switch is on or off |
The inconsistency warning deserves emphasis. A Delta table is a set of Parquet files plus a transaction log. If the log replicated a commit but some of the Parquet files it references didn’t, or the other way round, the copy isn’t a valid table at that version. That is why Microsoft’s Lakehouse recovery script trims transaction log entries newer than a failover timestamp that you obtain from Microsoft Support. Plan for reconciliation, not just copying.
Don’t enable it blindly. It costs storage and operations on every write, it locks the decision for 30 days, and it protects only part of the platform.
| Scenario | Enable DR setting? | Additional recovery needed |
|---|---|---|
| Sources can resend everything within the RTO, and the RTO is days | Often no: reloading from sources may be cheaper and simpler | Git-based rebuild; source replay runbook |
| Raw landing data is the only copy (sources don’t retain history) | Yes, for the capacity hosting raw landing | Rebuild items; reconcile copied tables; replay after the last consistent point |
| Large curated history that would take longer than the RTO to recompute | Yes | Rebuild items; trim to a consistent point; replay recent changes |
| Near-zero RPO or sub-hour RTO | Not sufficient on its own | Active parallel environment in a second region with dual ingestion (Microsoft’s own guidance for Warehouse and KQL) |
| Dev, test and sandbox capacities | Usually no | Recreate from Git when needed |
| Shared multi-tenant capacity with tenants on different SLAs | Decide by the strictest tenant, or separate capacities by tier | Tenant-ordered recovery and per-tenant validation |
| KQL databases or Eventhouse data | The switch doesn’t cover it | Independent databases in two regions with parallel ingestion |
| Region without a Fabric-supported pair | Not available | Your own replication to a supported secondary region |
Recovery by item type
Each Fabric item recovers differently after a regional failover. The table summarizes Microsoft’s experience-specific disaster recovery guidance and the Git integration supported-items list, checked November 2026. “Replicates” means available in the secondary region after a Microsoft failover with the capacity disaster recovery setting on.
| Item | What replicates | What Git restores | What you must recreate | Data recovery needed? |
|---|---|---|---|---|
| Lakehouse | Delta tables and files in OneLake | Lakehouse metadata, shortcut definitions, OneLake security role definitions (Preview), never tables or files | The Lakehouse (same name), non-Delta tables from your DDL scripts, materialized lake view schedules | Yes: copy tables one at a time with Microsoft’s script (trimmed to the failover timestamp) or Azure Storage Explorer |
| Warehouse | Its data, through OneLake | Schema objects (tables, views, procedures, functions) | The Warehouse, an interim Lakehouse for the copied data, application connection strings | Yes: recover the Delta tables into an interim Lakehouse, then load the new Warehouse with INSERT INTO, SELECT INTO or CREATE TABLE AS SELECT |
| Notebook | Nothing: code content isn’t saved after the disaster | Notebook code (Microsoft’s notebook DR guidance labels the Git approach as Preview) | Resource-folder files, default Lakehouse binding | Only the data the notebook reads or writes |
| Pipeline | Nothing: configuration isn’t replicated | Pipeline definition | Connections, schedules, references to recovered items | Rerun for the missing window |
| Dataflow Gen2 | Nothing: settings aren’t replicated | Dataflow definition (Git supported); otherwise an exported .pqt template |
Destinations and connections | Rerun for the missing window |
| Spark job definition | Main and reference files in OneLake | Definition | The item with the same settings; Lakehouse references; command-line arguments; run history is lost | Only what the job produces |
| SQL database in Fabric | Nothing cross-region: backups are zone- or locally redundant, with no geo-replicated backups | Schema (database project) | The database, from a .bacpac you exported to another region beforehand |
Yes: the restored database reflects the time of the export; later changes must be replayed |
| Semantic model | Covered by Power BI BCDR: read-only during failover; can’t export or back up during failover | Model definition (listed as Preview in the Git supported-items list) | Data source bindings, refresh schedules, credentials | Refresh after redeployment |
| Report | Covered by Power BI BCDR: viewable read-only during failover | Report definition (listed as Preview, with exceptions) | Binding to the recovered semantic model | No |
| Eventhouse / KQL database | Not covered: stored outside OneLake | Definitions (tables, mappings, policies) | Microsoft recommends independent databases in two regions with parallel ingestion | Yes, unless you ingest in parallel |
| Eventstream | Nothing; data safety depends on the destinations | Definition | Sources and destinations in the recovery region; optionally an active/active deployment | Through the destinations’ recovery |
| User data function | Nothing | Function source and project assets (Microsoft’s recommended path) | Dependencies, runtime settings, authentication; execution history is lost | No, but its data stores must recover first |
| API for GraphQL | Definitions and configuration aren’t replicated | Definition, with environment-specific source workspace and item IDs to update | Roles, authorization, RBAC, introspection and monitoring settings; consumer endpoints | No, but its sources must recover first |
| Mirrored database | Settings aren’t replicated | Definition | The mirrored database, in a workspace in another region | Re-mirroring from the source |
| Copy job | Nothing | Definition (Microsoft’s DR guidance labels this approach as Preview) | Lakehouse references | Rerun for the missing window |
| Variable library | Nothing | Definition and value sets | Select the active value set again | No |
What the table says, in one sentence: only data in OneLake can come back without you, and everything that makes it usable comes back through Git, scripts or people.
Recovery outside a regional disaster is usually item-level and in-region, with its own tools:
| Item | In-region recovery | Retention (checked November 2026) |
|---|---|---|
| Warehouse | Restore in place from a system restore point (every eight hours while the warehouse is active; Microsoft states an eight-hour RPO) or a user-defined restore point; time travel queries | 30 days by default, configurable from 1 to 120 days; at least 20 restore points kept with the default. Only workspace Admins can restore; you can’t restore into a new warehouse |
| Lakehouse | Delta time travel and RESTORE until VACUUM removes old files; OneLake soft delete for files |
Soft delete: seven days by default, configurable per workspace |
| SQL database in Fabric | Point-in-time restore to a new database in the same workspace | 7 days by default, up to 35. Microsoft documents that deleting the workspace deletes its databases unrecoverably |
| Deleted items | Workspace recycle bin or REST API; properties, configuration and relationships are restored, but not shared item permissions | On by default with three days for tenants that never configured it; 3 to 90 days. Supported item types only |
| Deleted workspaces | Restore from the admin portal | Seven days by default, 7 to 90 days |
One interaction to know: Microsoft documents that a Git sync or deployment pipeline can re-create an item that was soft-deleted, as an empty definition without data. If you need the data, restore the soft-deleted item first, delete the copy Git created, then sync again.
Git is not backup
Git protects definitions and code, not your data.
Fabric Git integration is the backbone of item recovery: Microsoft’s guidance for notebooks, user data functions, GraphQL, Copy jobs, Activator, variable libraries and deployment plans all starts with “connect the workspace to Git”. It is also routinely misunderstood as a backup.
What source control helps recover:
- Code: notebooks, Spark job definitions, user data functions, SQL objects.
- Supported item definitions: pipelines, dataflows, Lakehouse metadata and shortcut definitions, Warehouse schema, semantic models and reports (where supported).
- Configuration as code: variable libraries and their value sets, parameter files, deployment scripts.
- Deployment structure: workspace folders and, with a deployment plan, the order in which items deploy.
What it generally does not recover:
- Data: Lakehouse tables and files are explicitly not tracked; Git operations never touch them.
- Run state: run history, schedules’ execution history, Spark job history, workspace monitoring data.
- Secrets and credentials: connection credentials, Key Vault contents, gateway credentials.
- Runtime state: watermarks and checkpoints, unless you store them as data you also protect.
- Some permissions and settings: workspace role assignments, item shares and many item settings. Even where security definitions are tracked, such as OneLake security data access roles, Microsoft notes that Entra member IDs aren’t stored in Git; members carry over only between workspaces where role names match.
- External dependencies: gateways, private endpoints, source system configuration, downstream applications.
Also note the Preview labels: the Git supported-items list still marks several items, including semantic models and reports, as Preview (checked November 2026). Recovery that depends on a Preview capability should be tested more often, and documented with a fallback.
For the deployment side, see Microsoft Fabric CI/CD with fabric-cicd, which deploys from Git with a service principal and parameterized environments. To see in practice what Git, deployment pipelines and Warehouse database projects do and don’t carry between environments, work through Deploy a Fabric Solution with Git, Deployment Pipelines and Database Projects. The same pipeline that promotes to production is the pipeline that rebuilds production elsewhere, if its target workspaces, capacities and parameters can be pointed at a recovery environment.
Rebuild architecture
A regional recovery is a rebuild in a healthy region. The steps below are the architecture of that rebuild, not a click path.
- 1Create recovery capacityIn a healthy region with available compute
- 2Create workspacesSame names where code depends on them
- 3Connect GitSame repository and branch
- 4Restore item definitionsSync or deploy from Git; recreate the rest
- 5Recover dataCopy from replicas, restore, or replay
- 6Restore connections and secretsCredentials, gateways, workspace identities
- 7Rebind dependenciesItem IDs, endpoints, schedules, parameters
- 8Validate securityRoles, data rules, tenant isolation
- 9Validate dataCounts, totals, freshness, duplicates
- 10Resume consumersAPIs, models, reports, downstream systems
Three design decisions make this flow fast or slow:
- Names vs IDs. Microsoft recommends recreating workspaces and items with the same names, because code that refers to items by name keeps working. Anything that refers to workspace or item IDs breaks: new workspaces and items get new IDs. Variable libraries and deployment parameters turn that from a code change into a configuration change.
- Scripted vs manual. Every manual step is minutes multiplied by the number of workspaces and tenants. Creating capacity, workspaces, Git connections, role assignments and connections should be scripts you have run before.
- Pre-provisioned vs on demand. Creating capacity during a regional event competes with every other customer doing the same. Pre-provisioning a small recovery capacity and scaling it at recovery time is a cost decision that buys RTO.
Data recovery and replay
Copying replicated files is one way to recover data. A resilient platform should also know how to rebuild state from what it ingested, because replication can be inconsistent, late or absent, and because most data incidents (bad loads, corrupted tables) aren’t regional at all.
| Capability | What it gives recovery | Design rule |
|---|---|---|
| Replay from source | Reload any window the source still holds | Agree retention with source owners; record what was loaded per window |
| CDC replay | Resume from a log position instead of reloading everything | Store the last applied position durably; know the source’s change retention and alert before it’s exceeded |
| Watermark restart | Restart incremental loads from the last committed point | Advance the watermark only after the target commit succeeds |
| Raw landing retention | A copy of exactly what arrived, independent of transformations | Keep raw at least as long as the longest realistic reprocessing need; protect it as the system of record when sources can’t resend |
| Immutable source history | Append-only raw files never change after landing | No in-place updates in raw; corrections arrive as new files |
| Idempotent loads | The same window can be loaded twice safely | Deterministic keys and MERGE, or replace-by-partition |
| Merge and upsert | Replays update rather than duplicate | Match on business keys plus tenant ID; never on surrogate keys generated at load time |
| Duplicate prevention | Overlapping replay windows don’t double-count | Deduplicate on key and source version before merging |
| Checkpoint and state recovery | Streams and incremental loads resume where they stopped | Checkpoints and watermarks in durable, replicated storage, not only in the processing item |
| Downstream consistency | Gold tables and models reflect a coherent point in time | Rebuild downstream layers after upstream recovery completes, in dependency order |
The order of recovery for data is usually: restore or copy what is consistent, find the last point you can prove is complete, then replay everything after it from raw landing or the source. The execution log is what tells you that point: Designing an Execution Log for Data Pipelines records which windows each run committed, and Incremental Loads in Microsoft Fabric shows how to advance watermarks so that “committed” really means committed.
Idempotency during recovery
Every recovery step that writes data has to answer one question:
Can I run this workload twice without corrupting data?
During recovery, the answer is tested more than at any other time: windows overlap, operators rerun steps that may have partly succeeded, and replicated data may already contain some of the rows you are about to replay.
| Technique | What it protects against | Watch out for |
|---|---|---|
| Deterministic keys | The same source row always produces the same key | Keys generated at load time (identity columns, random UUIDs) create a new row on every replay |
| MERGE / upsert | Reloading a window updates rows instead of duplicating them | MERGE with duplicate source keys fails or behaves unpredictably; deduplicate first |
| Replace by partition | Rewriting a whole day or tenant partition is naturally repeatable | The partition must contain only that window and tenant |
| Append-only with dedup | Raw and event data stay complete | Readers must deduplicate, or a downstream step must |
| Transaction boundaries | A failed step leaves no partial result | A notebook that writes several tables has several commits; design the step so each table can be rerun independently |
| Retry-safe side effects | Notifications, exports and API calls aren’t repeated | Record side effects in the log and check before repeating |
Be careful with exactly-once language. Fabric pipelines retry activities, Spark retries tasks, and event sources commonly deliver at least once. Unless you have measured a specific end-to-end guarantee, design for at-least-once delivery and make the write idempotent; that is a property you can test.
A tenant-scoped, idempotent replay into a curated Delta table, as a sketch with generic names:
-- Replay one tenant's window from raw landing into the curated table.
-- Safe to run twice: duplicates are removed first, and MERGE matches on business key + tenant.
MERGE INTO curated.orders AS target
USING (
SELECT tenant_id, order_id, order_status, amount, source_version, source_modified_at
FROM (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY tenant_id, order_id
ORDER BY source_version DESC) AS rn
FROM raw.orders_landing
WHERE tenant_id = :tenant_id
AND source_modified_at >= :window_start
AND source_modified_at < :window_end
) AS ranked
WHERE rn = 1
) AS source
ON target.tenant_id = source.tenant_id
AND target.order_id = source.order_id
WHEN MATCHED AND source.source_version > target.source_version THEN
UPDATE SET order_status = source.order_status,
amount = source.amount,
source_version = source.source_version,
source_modified_at = source.source_modified_at
WHEN NOT MATCHED THEN
INSERT (tenant_id, order_id, order_status, amount, source_version, source_modified_at)
VALUES (source.tenant_id, source.order_id, source.order_status, source.amount,
source.source_version, source.source_modified_at);Notas
- líneas 11–13:tenant_id, :window_start and :window_end are named parameter markers, supplied when the statement is run.
- línea 19The version guard: a replay never overwrites a newer row with an older one.
Three properties make it recovery-safe: the tenant predicate is in the source query and the join condition, the version check stops an older replay from overwriting newer data, and running it twice produces the same table. Incremental Loads in Microsoft Fabric covers reruns, deletes and late data in more depth.
Recovery order and dependencies
Recovery order matters because almost every item depends on something else: a notebook needs its Lakehouse and its connection, a semantic model needs its tables, an API needs its model or Warehouse and its identity. Recovering in the wrong order wastes time on failures that are only waiting for a dependency, and can make things worse: a pipeline resumed before its watermarks are restored will happily reload from the beginning, or skip a window.
- 1. Identity and secretsNothing else can authenticate without them
- Service principals
- Key Vault access
- Workspace identities
- 2. Capacity and workspaces
- Recovery capacity
- Workspaces and roles
- Git connections
- 3. Core storage and base data
- Lakehouses
- Warehouses
- Raw landing
- Watermarks and checkpoints
- 4. Transformations and pipelines
- Notebooks
- Pipelines
- Connections and schedules
- 5. Serving
- Semantic models
- Data APIs
- User data functions
- 6. Consumers
- Reports
- Applications
- Downstream systems
| Step | Recover | Depends on | Ready when |
|---|---|---|---|
| 1 | Identity and secrets | Entra ID, Key Vault (their own resilience) | Recovery identities can authenticate and read required secrets |
| 2 | Capacity and workspaces | Step 1 | Workspaces exist on a running capacity, with roles assigned and Git connected |
| 3 | Core storage | Step 2 | Lakehouses and Warehouses exist with schemas deployed |
| 4 | Base data | Step 3, replicas or sources | Raw and curated tables restored to a known consistent point; watermarks restored |
| 5 | Transformations | Steps 3–4, connections | Notebooks run against recovered storage and produce expected outputs |
| 6 | Pipelines | Step 5, schedules held | Pipelines run manually once; schedules enabled only after validation |
| 7 | Semantic models | Step 4–5 data | Models deployed, bound and refreshed |
| 8 | APIs and applications | Steps 4–7, identities | Endpoints and connection strings updated; authorization verified |
| 9 | Reports and consumers | Steps 7–8 | Reports bound to recovered models; consumers notified |
Hold schedules until validation. A recovered pipeline that runs on its normal schedule before data is validated can overwrite restored tables, advance watermarks past unrecovered windows, or publish incomplete numbers.
Multi-tenant disaster recovery
On a multi-tenant platform, recovery has a second job besides bringing the platform back: recovery must not weaken tenant isolation. Recovery runs with elevated permissions, under time pressure, using scripts that touch every tenant’s data at once. That is exactly the situation in which tenant boundaries fail. The Microsoft Fabric Security Architecture guide describes the isolation patterns this section assumes: shared store, schema separation, workspace per tenant and dedicated capacity.
- Shared recovery pathShared store and shared workspaces
- One recovery identityReaches every tenant's data
- Bulk data copyAll tenants in one operation
- Shared replayTenant predicate in every statement
- Platform-wide validationPlus cross-tenant leak tests
- Isolated tenant recoveryWorkspace per tenant or dedicated
- Per-tenant identityScoped to one tenant's workspace
- Tenant-ordered rebuildHighest tier first
- Tenant secrets reboundEach tenant's own connections
- Per-tenant validationBefore that tenant reopens
What multi-tenant recovery has to cover:
| Topic | Design rule |
|---|---|
| Tenant-safe replay | Every replay statement carries the tenant predicate in both the source and the target condition, as in the MERGE above |
| Tenant-safe restore | Restoring a shared table restores every tenant in it. Know which tenants a restore point affects, and replay the others forward |
| Tenant-specific RPO and RTO | Contracts may differ by customer; recovery plans must say which tenant gets which target |
| Tenant priority tiers | Recovery order follows tiers, not alphabetical lists or whatever the script does first |
| Shared vs dedicated recovery | Shared stores recover once for everyone; dedicated tenants recover independently, in parallel if capacity allows |
| Workspace-per-tenant recovery | Only scripted provisioning scales: workspace, roles, identity, connections and items per tenant, from the same code used for onboarding |
| Shared-store recovery | One recovery, but validation per tenant: counts and totals by tenant ID, not only overall |
| Tenant context preservation | Staging tables, recovery copies and replay inputs keep the tenant ID; nothing derives it from paths or file names after the fact |
| Cross-tenant contamination risk | The recovery identity is the most powerful identity on the platform during the incident; scope it, log it, and remove it afterwards |
| Tenant-specific secrets | Each tenant’s source credentials are restored to that tenant’s connection, and only there |
| Tenant-specific validation | Each tenant is checked for completeness and for isolation before its access is reopened |
How recovery breaks isolation, in practice:
- A replay omits the tenant filter, so one tenant’s window is reloaded with every tenant’s rows, or merged into the wrong partition.
- Restored staging data loses its tenant ID, because the recovery copy came from a path-based folder layout and the tenant was inferred from the folder name.
- A shared recovery identity can access all tenants, and stays in place after the incident because nobody removed it.
- Tenant-specific secrets are rebound incorrectly, so a connection for one customer authenticates with another customer’s credential and loads its data.
- Recovered shortcuts point to the wrong tenant store, because a shortcut definition restored from Git kept an external target that isn’t remapped per environment or per tenant.
- Semantic model roles are recreated without members, so a recovered RLS role either filters everyone out or, worse, someone grants broad access “temporarily” to unblock reports.
Multi-tenant recovery checklist
- Tenant priority recorded: every tenant has a recovery tier, owner and contact.
- Tenant ID survives recovery: in raw landing, staging, recovery copies and replay inputs.
- Replay statements reviewed: tenant predicate present in source and target conditions.
- Recovery identity scoped: per tier or per tenant where isolation requires it; time-bound and logged.
- Secrets mapped per tenant: a documented mapping from tenant to connection to secret, checked after rebinding.
- Shortcut targets verified: every recovered shortcut points to the intended tenant’s store.
- Per-tenant validation: counts, totals and freshness by tenant, not only platform-wide.
- Cross-tenant tests rerun: automated tests prove one tenant can’t read another through each path, before reopening.
- Workspace-per-tenant provisioning scripted: the same automation as onboarding, tested in recovery mode.
Tenant recovery priority
Different tenants can have different SLAs, RPOs, RTOs, regulatory requirements and business criticality. A recovery plan that treats them identically either overspends on small tenants or under-protects large ones.
| Tenant tier | Example RPO | Example RTO | Recovery pattern |
|---|---|---|---|
| Tier 1: contractual, high value | 15 minutes | 2 hours | Dedicated workspace and capacity; pre-provisioned recovery capacity; continuous replay; recovered first |
| Tier 2: standard | 4 hours | 8 hours | Workspace per tenant on shared capacity; scripted rebuild; recovered after tier 1 |
| Tier 3: shared, low volume | Previous daily load | Next business day | Shared store and workspace; recovered together after tier 2 |
| Internal or trial | Best effort | Best effort | Rebuilt from Git when capacity allows |
These values are illustrative, not recommendations: real targets come from contracts and business impact analysis.
Priority shapes the architecture before it shapes the runbook:
- Workspace per tenant lets you recover tier-1 tenants independently and first. A shared store forces everyone to wait for the same restore and the same validation.
- Dedicated capacity for top tiers means their recovery doesn’t compete for compute with everyone else’s, and the disaster recovery setting can be enabled only where the targets require it, because the setting is per capacity.
- Sequencing follows tiers: identities, workspaces and data for tier 1 first, then tier 2, with validation gates per tenant.
- Cost follows tiers too: standby capacity, geo-replication and parallel ingestion for tier 1 only, if contracts pay for it.
Security during recovery
A recovered platform is not production-ready until its security boundaries are revalidated.
Recovery recreates resources. It doesn’t necessarily recreate every permission, role member or setting, and the documentation is explicit about several gaps:
| Area | What recovery does or doesn’t bring back | What to do |
|---|---|---|
| Identity restoration | Entra ID identities are outside the Fabric region; they’re available if Entra ID is | Verify the recovery identities and break-glass accounts before the incident |
| Service principals | Unchanged in Entra ID, but they hold no roles in the new workspaces | Reassign workspace roles and item permissions by script |
| Workspace identity | A new workspace has no workspace identity until you create one, and a restored workspace doesn’t get its identity back | Create the identity, then regrant it on every source it authenticated to (storage, databases, trusted workspace access) |
| Key Vault | Your vault’s own resilience applies; Fabric doesn’t copy secrets | Use a geo-resilient vault design; document which secrets each connection uses |
| Credentials | Connection credentials aren’t in Git and don’t follow items to a new workspace | Recreate connections with credentials from the vault, not from memory |
| Workspace roles (RBAC) | Not part of item definitions | Assign from code; compare with the documented role matrix |
| SQL permissions | Recreated only if your Warehouse project includes them | Keep GRANT, RLS and masking definitions in source control and deploy them |
| RLS and OLS | Model definitions can include roles, but members must be reassigned; OneLake security role definitions can be in Git (Preview) without Entra member IDs | Reassign members from groups; test with real identities |
| Item shares | Item recovery from the recycle bin doesn’t restore shared item permissions | Re-share deliberately, from the access model, not from requests |
| Tenant boundaries | Only as good as the replay, the secrets mapping and the shortcut targets | Rerun cross-tenant tests before reopening any tenant |
| Emergency access | Recovery often needs elevated rights | Time-bound, logged, named, and removed when recovery closes |
| Customer-managed keys | During failover, CMK supports read operations as long as Key Vault is healthy and permissions are intact; enabling, disabling or updating the key isn’t supported during failover | Keep the key vault and its access policies resilient; don’t plan key rotation during a regional event |
Also check what the original region’s controls were doing. Workspace outbound access protection rules and OneLake resource instance rules stay readable or enforced for reads during failover, but can’t be changed until the workspace is writable again; in a new recovery workspace they must be configured again. Private links, managed private endpoints and gateway configurations belong to the recovery environment’s network design, not to the item definitions.
The security checks that a recovery adds are the same ones the Data Platform Security Architecture Review Checklist lists for a new platform, especially access paths, service identities and tenant isolation. A recovered platform is, in practice, a new platform that has to pass the same review quickly.
Secrets and endpoint rebinding
A recovered platform has different identifiers than the original. Everything that pointed at the old ones must be found and changed, and that list should exist before the incident:
| What changes | Who depends on it | Rebinding approach |
|---|---|---|
| Workspace IDs | Notebooks with ABFS paths, GraphQL source references, scripts, APIs | Resolve IDs from names at runtime, or from variable libraries and parameters |
| Item IDs | Semantic models, shortcuts, pipelines invoking other items, external tools | Deployment parameters; post-recovery rebinding script |
| SQL endpoints | Applications, reporting tools, SSMS users | Connection strings in configuration, not code; a DNS or configuration indirection you control |
| Connections | Pipelines, dataflows, Copy jobs, semantic models | Recreate connections by script; record the mapping from item to connection |
| Gateways | On-premises and virtual network data sources | Gateway cluster available to the recovery region; Microsoft notes gateways don’t function during a Power BI failover |
| Credentials | Every connection | From Key Vault; rotate afterwards if the incident involved credential exposure |
| APIs | Customer applications, partner integrations | Stable public endpoint in front of the data API; backend target changes, client URL doesn’t |
| GraphQL and user data functions | Applications calling them | Microsoft notes that downstream systems might require endpoint rebinding, and GraphQL source IDs must be updated |
| External consumers | Downstream databases, exports, partner feeds | A consumer register with owners and contact paths |
Document rebinding before an incident. An undocumented dependency is found when a consumer complains, which is the slowest and least controlled discovery mechanism available. The fastest platforms keep a machine-readable inventory: item, what it references, how the reference is set, and who consumes it.
Observability during recovery
The recovery process itself should be observable. During an incident, people ask the same questions repeatedly: what has been recovered, which tenants are back, what failed, what is the RPO we actually achieved? A recovery execution record answers them from data instead of from a chat thread.
| Field | Purpose |
|---|---|
recovery_event_id |
Identifies the recovery as a whole |
incident_id |
Links to the incident or ticket |
step |
The recovery step, from the runbook’s list |
item_name, item_type |
What was recovered |
tenant_id |
Which tenant, when the step is tenant-scoped |
started_at, ended_at |
Duration, for RTO measurement |
status |
Started, succeeded, failed, skipped |
validation_result |
Outcome of the step’s validation |
operator |
The person or identity that ran it |
data_checkpoint |
The recovery point reached: watermark, restore point, failover timestamp |
error_message |
What failed |
A minimal table, aligned with the execution log’s identifiers so that replay runs and recovery steps can be joined:
CREATE TABLE IF NOT EXISTS ops.recovery_execution (
recovery_event_id STRING NOT NULL, -- one per recovery
incident_id STRING,
step_order INT NOT NULL,
step STRING NOT NULL, -- e.g. 'restore_definitions', 'replay', 'validate_security'
item_name STRING,
item_type STRING,
tenant_id STRING, -- null for platform-wide steps
execution_id STRING, -- the replay run in the execution log, if any
started_at TIMESTAMP NOT NULL,
ended_at TIMESTAMP,
status STRING NOT NULL, -- started | succeeded | failed | skipped
validation_result STRING,
operator STRING NOT NULL,
data_checkpoint STRING, -- watermark, restore point or failover timestamp
error_message STRING
) USING DELTA;Two details matter. The record must live somewhere that survives the incident, which during a regional recovery means the recovery region, not the failed one. And replay runs should write their normal execution log records with the same correlation ID as the recovery step that started them, as described in Designing an Execution Log for Data Pipelines. Remember also that workspace monitoring data doesn’t carry over to a recovered workspace: enable it again, and expect a new monitoring baseline.
Validation after recovery
Recovery ends when the platform is proven correct, not when the last step turns green.
| Area | Check | Evidence | Pass criterion |
|---|---|---|---|
| Platform | Capacity active | Capacity state, metrics app | Running, sized for the recovery load |
| Workspaces available | Workspace list, role assignments | Every expected workspace exists with its role matrix | |
| Connections restored | Test connection per data source | Every connection authenticates with the intended identity | |
| Data | Row counts | Counts by table and tenant vs source or pre-incident baseline | Within agreed tolerance |
| Business totals | Revenue, quantities or balances vs source system | Match, or differences explained by the RPO window | |
| Freshness | Latest business timestamp per table | Within the RPO | |
| Completeness | Execution log: every window committed | No gaps between the recovery point and now | |
| Duplicates | Key uniqueness checks | Zero duplicates on business keys | |
| Tenant boundaries | Rows per tenant per table | No rows in another tenant’s partition or workspace | |
| Security | Roles | Workspace and item roles vs the documented matrix | Exact match; no temporary broad grants left |
| Permissions | SQL grants, RLS and OLS members, OneLake security roles | Rules present and members assigned | |
| Identities | Service principals and workspace identities | Expected identities only, with expected scope | |
| Tenant isolation | Automated cross-tenant tests | Every test fails to read another tenant’s data | |
| Operations | Pipelines | Manual run of each critical pipeline | Succeeds against recovered storage |
| Schedules | Schedule list | Enabled only after validation, at intended times | |
| Alerts | Test alert | Reaches the on-call person | |
| Monitoring | Workspace monitoring and execution log | Enabled and recording in the recovery region | |
| Consumers | Semantic models | Refresh and a known query | Refreshes; returns validated totals |
| Reports | Key reports opened by business owners | Correct numbers, correct filtering | |
| APIs | Contract tests per tenant | Correct data, correct tenant scope | |
| Downstream systems | Consumer register | Each consumer confirmed or notified |
Recovery testing
A DR plan that has never been tested is an assumption.
Testing is where RTO stops being a number in a document. Use several kinds of test, because they answer different questions:
| Test | What it proves | Frequency (illustrative) |
|---|---|---|
| Tabletop exercise | People know their roles, decisions are pre-approved, the runbook is understandable | Twice a year, and after major changes |
| Partial recovery drill | One workspace or item type can be rebuilt from Git and data in a separate environment | Quarterly |
| Full regional simulation | The whole platform can be rebuilt in another region within the RTO | Yearly, where practical and justified by the targets |
| Tenant-isolation test | Recovery scripts and replays don’t leak across tenants | With every drill |
| Replay test | A window can be replayed twice with identical results | With every change to load logic |
| Credential recovery test | Secrets can be rotated and identities recreated without breaking loads | Twice a year |
| Recovery time measurement | How long each stage actually takes | Every drill |
A simulation doesn’t need to fail over Fabric. Rebuilding the production platform into an empty workspace on another capacity, from Git and from copies of the data, exercises almost every step that a real regional recovery needs.
Compare measured recovery to the target per stage, not only in total. A drill that meets the RTO with 10 minutes to spare because one engineer remembered an undocumented step hasn’t met it; record the gap and fix the runbook. The smaller, more frequent recovery tests (rerun, replay and idempotency checks that belong in every release) are covered in Testing a Production Data Platform, and the recovery-replay test itself (rebuild from retained batches, compare with a full-load result) in Testing Incremental Loads and Idempotency.
RTO decomposition
An RTO is the sum of stages, and each stage has its own owner and its own levers. Breaking the target into stages turns “we need four hours” into a planning framework.
- ProvisionCapacity, workspaces, roles
- DeployItem definitions from Git
- RecoverData copied, restored or replayed
- RebindConnections, secrets, endpoints
- ValidateSecurity, data, consumers
- ResumeSchedules and access reopened
| Stage | What drives its duration | Target | Measured |
|---|---|---|---|
| Decide | Detection, escalation, authority to declare recovery | ||
| Provision | Capacity availability in the recovery region; scripted vs manual workspace creation | ||
| Deploy | Number of items; Git sync or deployment automation; manual recreation for unsupported items | ||
| Recover data | Data volume, copy throughput, replay windows, source retention | ||
| Rebind | Number of connections, secrets and consumers; whether references use names, IDs or configuration | ||
| Validate | Automated vs manual checks; number of tenants | ||
| Resume | Schedule enablement, consumer notification, staged reopening by tenant | ||
| Total | Sum of the above | The agreed RTO | From the last drill |
Fill in targets by working backwards from the RTO, and measured values from drills. Don’t borrow timings from other platforms or from vendor statements: Microsoft’s “typically less than one hour” describes its failover to a read-only service, which is not one of your stages at all.
Recovery cost architecture
Recovery is a trade between money spent before an incident and time lost during it.
Lower cost
- Reconstruct when needed: no standby resources; rebuild from Git and sources at recovery time.
- Source replay: rely on sources and raw landing retention instead of replicated curated data.
- Fewer standby resources: no pre-provisioned capacity; the disaster recovery setting only where raw data is the only copy.
Faster recovery
- Pre-created capacity: a small capacity in the recovery region, scaled at recovery time.
- Duplicated data: the disaster recovery setting on production capacities, or dual ingestion into a second region for the most critical data.
- Dedicated recovery environment: workspaces, connections and identities kept ready in the recovery region.
- Automation: every step scripted, tested and timed.
Lower RTO usually costs more.
The cost of the disaster recovery setting appears as BCDR Storage and BCDR Operations in the Capacity Metrics app, and standby capacity costs whatever its SKU costs while it exists. Price them with current Microsoft pricing for your region rather than from rules of thumb, and put them next to the business cost of the RTO they buy. The decision belongs to the business owner; the architect’s job is to make both numbers visible.
Runbooks
Architecture and runbooks answer different questions, and both are needed:
Architecture
What recovery capability exists: replayable sources, Git coverage, scripted provisioning, tenant tiers, standby resources, validation tests. It is designed once and changed through review.
Runbook
What operators do during the incident: the steps in order, the decisions and who makes them, the evidence to collect, the checks before reopening. It is followed under pressure, so it stays short and concrete. The site’s Fabric capacity saturation runbook shows the format.
Recommended runbooks to write from this architecture:
- Fabric regional recovery: declare, provision, deploy, recover, rebind, validate, resume, with the recovery execution record.
- Corrupted data and replay: identify the bad window, restore to the last good point, replay, reconcile.
- Failed deployment rollback: redeploy the previous version from Git, and decide whether data changed by the release needs a restore or a forward fix.
Recovery architecture decision framework
Use the same twelve steps for each workload or customer tier:
| Step | Question | Output | Validation criterion |
|---|---|---|---|
| 1. Business process | What does this workload support, and who owns it? | Workload, owner, impact of outage | Owner has signed off the impact description |
| 2. Failure scenarios | Which failure domains apply? | Scenarios from job to region | Each scenario has a recovery path |
| 3. RPO | How much data loss is acceptable? | RPO per workload or tier | Agreed by the business owner |
| 4. RTO | How long can it be unavailable? | RTO per workload or tier | Agreed by the business owner |
| 5. Dependencies | What must recover first? | Dependency layers | No circular or unknown dependencies |
| 6. Recovery source | Where does the data come back from? | Replica, restore point, raw landing or source | Source retention covers the RPO plus the RTO |
| 7. Rebuild method | How do items come back? | Git, scripts, templates, manual | Every production item has a method |
| 8. Security | How are identities, secrets and isolation restored? | Identity, secret and tenant mapping | Security validation is scripted |
| 9. Validation | How do we know recovery is correct? | Validation matrix | Checks automated where possible |
| 10. Test frequency | How often is it proven? | Test calendar | Last test date within the agreed interval |
| 11. Cost | What does the chosen design cost? | Standby and replication cost vs outage cost | Approved by the business owner |
| 12. Owner | Who maintains and executes it? | Named owner and deputy | Owner has run the last drill |
Record the decisions as architecture decision records, like any other platform decision, so the reasoning survives the people who made it.
Recovery architecture review checklist
Use this before approving a production platform, a new workload or a new customer tier. Every unchecked item is either a decision to make or a risk to accept in writing.
Requirements
- RPO and RTO agreed per workload: signed by the business owner, including tenant-specific targets.
- Recovery scope defined: which failure domains the plan covers, and which it explicitly doesn’t.
Failure domains
- Recovery path per failure: job, item, workspace, capacity, zone, region, source, identity, network and external dependency.
- Home region and pairing understood: the paired region supports Fabric, or a custom cross-region plan exists.
Data
- Recovery source per dataset: replica, restore point, raw landing or source replay.
- Source retention verified: long enough to cover the RPO plus the time to recover.
- Loads idempotent: every load can run twice with identical results.
- Watermarks and checkpoints protected: stored as data and recovered with it.
- Disaster recovery setting decided per capacity: with the reason, cost and the 30-day change constraint.
- Data outside OneLake covered: KQL databases, SQL databases and external stores have their own approach.
Code and configuration
- Definitions in Git: every supported production item, with unsupported items exported or scripted.
- Environment values externalized: variable libraries or parameters, no hard-coded IDs.
- Preview dependencies documented: with a fallback if the Preview capability fails.
Identity and security
- Recovery identities ready: scoped, tested, time-bound when elevated.
- Role matrix as code: workspace roles, item permissions, SQL permissions and data-role members reassignable by script.
- Workspace identities and grants documented: to recreate and regrant on every source.
- Secrets recoverable: vault resilience and the mapping from connection to secret documented.
Multi-tenant
- Tenant tiers and order: every tenant has a recovery tier.
- Tenant-safe replay and restore: tenant predicates reviewed; tenant ID preserved through recovery copies.
- Cross-tenant tests part of recovery: run before any tenant reopens.
Dependencies
- Dependency layers documented: recovery order from identity to consumers.
- Rebinding inventory: every reference to workspace IDs, item IDs, endpoints and connections, with its consumers.
Recovery procedure
- Runbooks written: regional recovery, data replay, deployment rollback.
- Provisioning scripted: capacity, workspaces, Git connections, roles and connections.
- Schedules held until validation: documented in the runbook.
Validation
- Validation matrix: platform, data, security, operations and consumers, with pass criteria.
- Recovery execution record: stored in the recovery region, linked to the execution log.
Testing
- Drills scheduled: tabletop, partial and full where justified, with dates.
- RTO measured per stage: the last measured total is within the target.
Cost
- Standby and replication costs known: compared with the outage cost and approved.
Ownership
- Owners named: for the recovery architecture, each runbook and each tenant tier.
- Review date set: the plan is reviewed after major platform changes and at least yearly.
Recommendation
Start from the business: RPO and RTO per workload and per tenant tier, agreed by the people who own the impact. Then design the platform so that most recoveries are routine: restartable, idempotent loads; protected raw landing and watermarks; every definition in Git; environment values in configuration. Enable the capacity disaster recovery setting where data would otherwise be lost or take too long to recompute, and treat what it copies as material to reconcile, not as a recovered platform. Script the rebuild, keep a rebinding inventory, recover in dependency order, and hold schedules until validation. On a multi-tenant platform, recover by tier and never let recovery weaken isolation. Then measure: a recovery time is real only after a drill has produced it.
What to read next
- Designing a Production Data Platform: where recovery sits among the platform’s other architecture decisions.
- Microsoft Fabric Security Architecture: the identities, access paths and isolation patterns that recovery must restore.
- Data Platform Security Architecture Review Checklist: the review a recovered platform should pass before reopening.
- Incremental Loads in Microsoft Fabric: watermarks, MERGE and safe reruns, the foundation of replay.
- Monitoring Fabric Pipelines and Notebooks: the alerts and signals that start a recovery.
- Migrate SSIS Packages to Microsoft Fabric: migration is the moment to make loads restartable and recoverable.
Sources
- Reliability and disaster recovery: Reliability in Microsoft Fabric (availability zones, disaster recovery capacity setting, failover behavior, recovery steps), Experience-specific disaster recovery guidance (recovery per item type), OneLake disaster recovery and data protection, OneLake compute and storage consumption, Fabric region availability
- Power BI continuity: Power BI high availability, failover, and disaster recovery FAQ
- Item-level recovery: Restore in place of a warehouse, Data retention in Fabric Data Warehouse, Automatic backups in SQL database in Fabric, Recover deleted files in OneLake, Retention and recovery in Fabric, Recover or permanently delete items
- Git and deployment: Overview of Fabric Git integration (supported items and Preview status), Lakehouse Git integration and deployment pipelines
- Security: Workspace identity
- Recovery design: Shared responsibility for reliability, Recommendations for designing a disaster recovery strategy
Fabric behavior checked against Microsoft Learn in November 2026. Disaster recovery support, Git integration coverage and item recovery are tracked per item and change often, and several capabilities referenced here are in Preview. Recheck the linked pages before relying on a specific behavior, retention period or limit.
¿Te resultó útil este contenido?
¿Qué podríamos mejorar?
Gracias — esto me ayuda a mejorar Javi Builds.
Sigue aprendiendo
- Guía rápidaContenido en inglésMicrosoft Fabric Workspace Settings Cheat SheetThe Microsoft Fabric workspace settings that change how a data engineering workspace behaves: Spark pools, environments and jobs, domain assignment, OneLake shortcut caching and diagnostics, and data workflow (Apache Airflow job) runtime pools, with what to verify for each.Abrir guía rápida
- RecursoContenido en inglésMicrosoft Fabric Certification Study PlannerAn adaptable 6–8 week example plan that connects official Fabric certification objectives with hands-on practice, review, and exam readiness.Ver recurso
¿Qué debería crear después?
Estoy construyendo Javi Builds alrededor de problemas reales que enfrentan los ingenieros de datos. Dime qué tema te gustaría ver después.
Mantente al día
Recibe solo las novedades de Javi Builds que realmente te interesan.
Revisa tu bandeja de entrada
Te he enviado un enlace de confirmación. Tu suscripción se activará cuando la confirmes.