Microsoft Fabric
Designing a Production Data Platform: From Requirements to Operations
The decisions a data solution architect makes before building a production data platform: requirements and NFRs, source assessment, batch vs streaming, storage and serving, idempotency, security, observability, capacity, cost, RPO/RTO, governance and operational ownership, with a reusable review checklist.
On this page
Most data platform designs start with a technology question: Lakehouse or Warehouse? Pipelines or notebooks? Do we need streaming? Those questions matter, but they are the wrong place to start. This article answers a different one:
If I am designing a production data platform, what decisions do I need to make before I start building?
The architectural principle. A production data platform should not start with “Lakehouse or Warehouse?” It should start with business requirements and non-functional requirements. Every technology choice that follows (the ingestion pattern, the storage engine, the number of layers, the capacity size, the recovery design) is a consequence of those requirements, and it should be possible to trace each choice back to the requirement that justified it.
The rest of the article walks that path in order, from requirements to operations:
- 1Business requirementsOutcomes, consumers and the decisions the data supports
- 2Non-functional requirementsVolume, latency, concurrency, RPO, RTO, security, cost
- 3Source assessmentOwners, extraction method, change capture, failure behavior
- 4Architecture decisionsOptions, tradeoffs and a written decision record
- 5Platform architectureIngestion, storage, transformation and serving
- 6OperationsSecurity, monitoring, cost, recovery, governance, ownership
This is the work of a data solution architect: turning business intent into decisions that engineers can build, operators can run and auditors can follow. It is not a Microsoft Fabric tutorial, a tour of Fabric services or a medallion-architecture explainer. The principles apply to any analytical platform. The examples use Microsoft Fabric because that is where Javi Builds goes deepest, and because concrete examples expose tradeoffs that generic diagrams hide.
How to read the claims in this article. Two kinds of statements appear below, and they are kept apart:
| Kind of statement | How you can tell | Example |
|---|---|---|
| Platform fact | Describes documented product behavior; the Microsoft Learn page is listed in Sources | OneLake geo-replication is asynchronous and file-level |
| Javi Builds guidance | Written as a recommendation, a framework or a checklist | Write a decision record for every choice that is expensive to reverse |
If you already have a Fabric platform and want to know how to draw workspace, capacity and storage boundaries inside it, read Designing a Scalable Microsoft Fabric Architecture. This article sits one step earlier: it is about deciding what to build, and proving the decision before the first pipeline exists.
The running example
To keep the decisions concrete, one generic scenario runs through the article. A mid-sized retailer wants one analytical platform for four sources and four groups of consumers:
| Source | Shape | Consumer need |
|---|---|---|
| ERP order database | SQL Server, on-premises, change data capture available | Finance reporting, ready by 07:00 every day |
| Point-of-sale and inventory events | Event stream, several million events per day | Store operations dashboard, at most 15 minutes old |
| Supplier price files | CSV over SFTP, daily, schema changes a few times a year | Margin analysis, next day is fine |
| CRM | SaaS REST API, rate-limited, customer personal data | Customer analytics and a weekly data science extract |
About 400 report users concentrate between 08:30 and 10:00. Customer names, emails and addresses are personal data. The scenario is invented and deliberately ordinary; it contains no customer or employer details.
Start with requirements, not technology
Requirements come in two kinds. Both are needed, and the second kind drives most architecture decisions.
Functional requirements
Functional requirements describe what the platform does:
- ingest data from the source systems that matter;
- transform it into clean, conformed and trusted entities;
- model it for the questions the business asks;
- expose it to reports, semantic models, applications and data science;
- support downstream systems that consume curated data, such as a partner API or an operational application.
Functional requirements tell you which pipelines and tables must exist. They rarely tell you how to build them. “Ingest the ERP orders table” is satisfied by a nightly full copy, an hourly watermark load, CDC replication or Mirroring. The choice between those comes from the second kind of requirement.
Non-functional requirements
Non-functional requirements (NFRs) describe how well, how fast, how safely and at what cost the platform must do it. They are the ones teams forget to ask about, and the ones that cause redesigns:
| Area | Questions to ask |
|---|---|
| Volume and growth | How many rows and gigabytes today? How fast does it grow? What is the largest table in three years? |
| Latency | How old may the data be when someone reads it? Is that a business need or a preference? |
| Concurrency | How many users and jobs run at the same time? When is the peak? |
| Availability | When must the platform be usable? What happens to the business if it is not? |
| RPO and RTO | How much data can be lost in an incident? How long can the platform be unavailable? |
| Security and compliance | Which data is sensitive? Which regulations apply? Who may see what? |
| Retention | How long must raw and curated data be kept? When must it be deleted? |
| Observability | How will the team know that a load is late, wrong or missing, before a user does? |
| Recoverability | Can every load be rerun safely? Can a bad day be reprocessed? |
| Cost | What is the budget? Which costs grow with data, with users, with environments? |
| Maintainability | Who will operate it, with which skills? How many people are on call? |
| Deployment model | How do changes reach production? Who approves them? How are they rolled back? |
Ask for numbers, not adjectives. “Real time” can mean 2 seconds or 2 hours. “Secure” can mean workspace roles or column-level masking with audited access. A requirement you cannot test is not yet a requirement; turn “fast” into “the finance model refreshes by 07:00 on 95% of business days”.
From requirement to architectural consequence
The value of an NFR is the decision it forces. This table is the core of the requirements phase:
| Requirement | Architectural consequence |
|---|---|
| Near-real-time latency (minutes) | Scheduled batch is not enough. CDC, Mirroring, micro-batch or event ingestion is required, with its own monitoring and replay design |
| Data needed by 07:00 for finance | The load window, its dependencies and its retry budget become part of the design; late detection must alert before 07:00, not after |
| RTO under 1 hour | The recovery strategy must be designed, automated and tested before production. Redeploying from Git and rebuilding by hand will not meet it |
| RPO close to zero for a source | The platform cannot be the only copy. The source must be replayable, or ingestion must be continuous and durable |
| 500 concurrent BI users | Serving and concurrency architecture matter: semantic model design, query patterns, capacity headroom at peak, and isolation from heavy background jobs |
| Sensitive personal data (PII) | Security, isolation, masking, classification and access review become architectural requirements, not later tasks |
| 7-year retention for audit | Storage cost, tiering, immutability of raw data and deletion procedures have to be designed |
| Source schema changes without notice | Raw landing that tolerates drift, schema-change detection and a quarantine path are required |
| A team of two engineers | Fewer layers, fewer engines, managed services and strong defaults beat a flexible but complex design |
| Strict monthly budget | Cost drivers must be estimated per workload, non-production environments sized down, and consumption monitored as a first-class signal |
The same discipline applies to every row: the requirement is written by the business, the consequence is written by the architect, and both sign the result.
For the running example, three NFRs dominate. The 15-minute inventory freshness rules out a purely nightly platform. The 07:00 finance deadline makes the overnight load window and its retries a design object. And the CRM personal data makes access control and classification part of the first release, not a later hardening project.
An architecture decision framework
Once requirements are written down, decisions follow. Making them consistently matters more than making each one brilliantly. Use the same six steps for every significant decision:
| Step | Question | Example: serving layer for finance |
|---|---|---|
| 1. Requirement | Which requirement forces this decision? | Finance analysts query curated data in T-SQL and Power BI, by 07:00 |
| 2. Constraints | What limits the options? | Team strong in T-SQL, light in Spark; existing stored-procedure logic; PII in the customer dimension |
| 3. Options | What are the realistic alternatives? | Lakehouse tables via the SQL analytics endpoint; a Warehouse; both |
| 4. Tradeoffs | What does each option cost and risk? | Endpoint is read-only from SQL; Warehouse offers multi-table T-SQL writes and transactions; two items mean two security surfaces |
| 5. Decision | What did we choose, and why? | Spark engineering in a Lakehouse; finance serving in a Warehouse |
| 6. Validation criteria | How will we know it was right, and when do we revisit it? | Refresh by 07:00 on 95% of days; zero ad hoc data copies by analysts; review after two quarters |
Step 6 is the one most often skipped, and the one that makes the framework useful. A decision without validation criteria cannot be proven wrong, so it is never revisited.
Record it: architecture decision records
An architecture decision record (ADR) is a short document that captures one decision: its context, the options considered, the choice and its consequences. Microsoft’s Well-Architected guidance calls the ADR one of the most important deliverables of a solution architect, started at the onset of a workload and maintained for its lifespan. It is append-only: when a decision changes, you write a new record that supersedes the old one and link the two, so the history of why survives staff changes. Recording the confidence of a decision helps the next person judge whether to revisit it.
Write an ADR when a decision is expensive to reverse, crosses team boundaries, or will surprise someone in a year. Do not write one for every table name.
Example: Warehouse or Lakehouse for a workload
The serving-layer decision above, documented as an ADR:
# ADR-004: Serve finance reporting from a Warehouse; keep Spark engineering in a Lakehouse
Status: Accepted · Date: 2026-11-08 · Supersedes: none · Confidence: high · Review: after two quarters in production
## Context
Finance needs curated order, margin and customer data by 07:00 for Power BI and ad hoc T-SQL.
The engineering team uses PySpark for semi-structured event and supplier data.
## Requirement
- Curated finance tables queryable in T-SQL and through a semantic model by 07:00 (NFR-02).
- Customer PII visible only to approved roles (NFR-09).
## Constraints
- Finance analysts know T-SQL; existing logic is in stored procedures.
- Two platform engineers; no appetite for a second transformation engine in finance.
## Options considered
1. Lakehouse only, served through the SQL analytics endpoint.
2. Warehouse only, including raw and semi-structured ingestion.
3. Lakehouse for ingestion and Spark transformation; Warehouse for finance serving.
## Decision
Option 3. Raw and conformed data stay in the Lakehouse (Delta tables in OneLake).
Finance facts and dimensions are built in the Warehouse with T-SQL and served from it.
## Tradeoffs accepted
- Two items, two security surfaces (OneLake security and SQL permissions) to keep aligned.
- A defined hand-off contract between the conformed Lakehouse tables and the Warehouse loads.
## Validation criteria
- Finance model refreshed by 07:00 on at least 95% of business days for a quarter.
- No analyst-maintained copies of finance data outside the platform.
- PII columns masked or excluded for non-approved roles, verified through every engine used.
## Consequences
- Warehouse loads must be idempotent (delete-insert or MERGE by business date).
- Access reviews cover both items.
The decision itself is not the point. A different team, with Spark skills and no stored-procedure estate, could reasonably choose option 1 with the same framework. What matters is that the reasoning is written, testable and revisitable. Microsoft Fabric Lakehouse vs Warehouse covers the technical differences in depth, and the Lakehouse vs Warehouse Decision Guide is the one-page version.
Source system assessment
Sources constrain the platform more than any technology choice. A source that cannot provide changes forces full loads. A source with a fragile API forces throttling and retries. A source that cannot replay old data turns your raw layer into the only copy. Collect these facts before choosing ingestion technology:
| Attribute | What to record | Why it matters |
|---|---|---|
| Source | System name and the specific objects (tables, endpoints, folders) | Scope is per object, not per system |
| Owner | A named business owner and a technical contact | Someone must approve schema changes and answer incidents |
| Technology | Engine, version, API style, file format | Decides connectors and extraction options |
| Location | On-premises, cloud region, SaaS, partner network | Decides gateways, network paths and data residency |
| Connectivity | Network route, firewall rules, authentication method | Often the longest lead time in a project |
| Data volume | Rows and size of each object today | Sizes the initial load and the storage plan |
| Daily growth | New and changed rows per day, and the peak day | Sizes incremental loads and capacity |
| Extraction method | Full, watermark, CDC, file drop, API pagination, event push | The single biggest driver of ingestion design |
| CDC available? | Whether change tracking or CDC is enabled, or could be | Decides whether deletes and updates can be captured reliably |
| Latency requirement | How fresh the consumers need this source to be | Decides batch, micro-batch or streaming for this source |
| Schema volatility | How often columns change, and whether you are told | Decides raw-landing tolerance and schema checks |
| Sensitive data | Personal, financial, health or confidential columns | Decides classification, masking and access design |
| Retention requirement | How long raw and history must be kept | Decides storage tiers and deletion procedures |
| Expected SLA | When the source is available, and maintenance windows | Defines the realistic load window |
| Dependency | Upstream jobs or other sources this one waits for | Defines orchestration order and failure propagation |
| Failure behavior | What happens on timeout, partial extract, late or duplicate files | Decides retry, replay and idempotency design |
Two rows from the running example show how quickly the assessment turns into design:
| Attribute | ERP orders | Supplier price files |
|---|---|---|
| Extraction method | CDC on the orders and order-lines tables | One CSV file per supplier per day on SFTP |
| Daily growth | About 200,000 changed rows | 30 files, each under 50 MB |
| Schema volatility | Low, changes go through a release process | A few unannounced changes per year |
| Failure behavior | Source can replay changes from its retention window | A missed file is never resent automatically |
| Design consequence | Incremental CDC load with MERGE; deletes captured |
Land every file unchanged; detect schema drift; alert on a missing file by 06:00 |
The last row of that table is why this phase exists. A source that never resends a missed file means the raw landing zone is the system of record for that data, which changes its retention, its backup and its recovery priority.
If the sources are SSIS packages and SQL Agent jobs today, the assessment overlaps with a migration inventory. Migrate SSIS Packages to Microsoft Fabric shows how to inventory and score an existing estate, and Incremental Loads in Microsoft Fabric covers choosing between watermarks, Change Tracking, CDC and Mirroring for a source.
The shape of the platform
With requirements and sources understood, the platform takes a shape. The figure below is a reference shape, not a blueprint. Each tier is a decision the following sections make; the item names in each tier are one possible Microsoft Fabric implementation.
Sources
- Operational databases
- SaaS APIs
- Files
- Event streams
Ingestion
- Batch and CDC copyCopy job, Copy activity, Mirroring
- Event ingestionEventstream
- OrchestrationPipelines, schedules, retries
Storage
- Raw landingReplayable, source-aligned
- Curated tablesDelta tables in OneLake
Transformation
- SQLWarehouse T-SQL
- SparkNotebooks, Spark jobs
- Low-codeDataflow Gen2
Serving
- SQL servingWarehouse, SQL analytics endpoint
- Semantic modelsMeasures, security, Direct Lake
- Operational SQLSQL database, APIs
Consumers
- Reports
- Applications
- Data science
Cross-cutting concerns
- Security
- Observability
- Governance
- CI/CD
- Cost
- Recovery
Read it as a set of responsibilities. Ingestion moves data and records what it moved. Storage keeps it replayable and governed. Transformation turns it into trusted entities. Serving publishes stable contracts to consumers. The rail on the side is not decoration: security, observability, governance, CI/CD, cost and recovery have to be decided for every tier, and the second half of this article is about those decisions.
Batch, micro-batch, CDC, event-driven or streaming
Latency is the requirement teams most often inflate. Streaming is not automatically better: it costs more to build, to run and to debug, and it changes how you think about correctness. Choose the slowest pattern that meets the latency requirement, then confirm the source supports it.
| Pattern | Typical latency | Complexity | Cost profile | Ordering | Replayability | Operational burden | Source support needed |
|---|---|---|---|---|---|---|---|
| Scheduled batch | Hours to a day | Low | Bursty; easy to schedule off-peak | Not an issue within a batch | Rerun the batch | Low | Any readable source |
| Micro-batch | Minutes | Medium | Frequent small runs; startup overhead adds up | Per batch | Rerun a window | Medium: many runs to monitor | A reliable watermark or change marker |
| CDC | Minutes to near real time | Medium | Proportional to change volume | Source commit order, if preserved | From the source’s change retention | Medium: change retention and gaps to watch | CDC or change tracking enabled |
| Event-driven | Seconds to minutes after the event | Medium | Proportional to event count | Per trigger; no global order | Only if the event or file is kept | Medium: missed or duplicate triggers | The source emits events (file arrival, message) |
| Streaming | Seconds | High | Continuous compute | Per partition or key; late events are normal | From the stream’s retention window | High: always on, needs replay and backfill | A stream or a CDC feed |
A few rules keep this decision honest:
- Latency is per consumer and per source, not per platform. In the running example, inventory events need minutes; supplier prices are fine the next day. One platform can, and usually should, use several patterns.
- “Near real time” for a dashboard that people read twice a day is batch. Ask who acts on the data and how fast they act.
- Every streaming design needs a batch path too. Backfills, reprocessing and recovery are batch problems, even on a streaming platform.
- Ordering and duplicates are your problem. Microsoft documents Fabric Eventstream delivery as at least once, so the consumer must tolerate duplicates. Events can also arrive late or out of order; the design must say what happens then.
For the running example, the ERP uses CDC into incremental MERGE loads on a 30-minute cycle, inventory events flow through an event stream into Delta tables, supplier files are event-driven on arrival with a scheduled check for missing files, and the CRM is a nightly batch within its API rate limit.
ETL vs ELT vs CDC vs Streaming explains each integration pattern and its Fabric implementation, and the batch vs streaming cheat sheet is the quick reference.
Storage and serving decisions
Storage and serving are not one choice between mutually exclusive products. A production platform usually combines several, each chosen for a workload. Ask workload questions first:
| Workload question | If the answer is… | It points toward |
|---|---|---|
| Analytical or operational? | Operational: single-row reads and writes, application transactions | An operational SQL database (SQL database in Fabric, Azure SQL, SQL Server), not an analytical store |
| SQL-first or Spark-heavy? | SQL-first team, stored-procedure logic | A Warehouse for transformation and serving |
| Spark-heavy team, PySpark transformations, ML features | A Lakehouse with Delta tables | |
| Structured or semi-structured? | JSON, nested files, schema drift, raw files to keep | A Lakehouse (files and tables) for landing and conformance |
| Transaction requirements? | Multi-table writes that must commit together | A Warehouse or an operational database; check the engine’s documented transaction support |
| BI consumption? | Many report users, governed measures, row-level security for reports | A semantic model over curated tables |
| Data science? | Notebooks, feature engineering, model training | Lakehouse tables and files |
| Interoperability? | Other engines or teams must read the same data | Open Delta tables in OneLake, shared through shortcuts rather than copies |
In Microsoft Fabric, these map to items that share one storage layer:
- OneLake is the storage foundation. Lakehouse and Warehouse data are both stored there as Delta tables, which is why the choice between them is about engines and write paths, not about where bytes live.
- Lakehouse is the natural home for files, raw landing and Spark transformation. Its SQL analytics endpoint gives a read-only T-SQL surface over its tables.
- Warehouse is a T-SQL engine with full write support, suited to SQL-first transformation and relational serving.
- SQL database in Fabric is the operational, transactional database, for application workloads; it is not a substitute for a Warehouse.
- Semantic models are the business-facing serving layer: measures, relationships and report security. They are part of the architecture, not a reporting afterthought.
- External operational SQL stays where it belongs. An order-entry system remains on its SQL Server or Azure SQL database; the analytical platform reads from it, through CDC, Mirroring or extraction, and never becomes its transactional store.
For SQL Server teams, the biggest change is that “the database” splits into several items with one storage layer underneath. SQL Server to Microsoft Fabric: What Maps to What? is the mental model for that shift, including where backups, Query Store and HA/DR went. Semantic Models in Microsoft Fabric covers the serving layer for BI.
Data layers and modeling
Layers exist to create contracts: a point where data has a known quality, a known schema and a known owner. They do not exist because a diagram has three boxes.
| Layer | Contract it provides | Keep it when | You can skip or merge it when |
|---|---|---|---|
| Raw / landing | Exactly what the source sent, with load metadata | The source cannot replay, audit requires the original, or schema drifts | The source is replayable on demand and audit does not require the original |
| Bronze | Source-aligned tables, typed, with batch identity | Many consumers read source-shaped data, or you need table history | Raw landing is already tabular and queried directly |
| Silver | Cleaned, deduplicated, conformed entities | Several sources describe the same entity, or several products share it | One trusted source feeds one model |
| Gold | Business-shaped tables for a process or product | Consumers need stable, governed definitions | The semantic model already provides the business shape over Silver |
| Warehouse serving | Relational, T-SQL-native serving tables | SQL-first consumers, stored-procedure logic, multi-table writes | Consumers use the semantic model or Spark only |
| Semantic layer | Measures, relationships, report security | Business users consume through BI tools | No BI consumption |
Fewer layers are often enough. A small, trusted source that feeds one report may need only landing and a curated serving table. The running example keeps raw landing for supplier files (they are never resent) and for inventory events (they are needed for replay), conforms customers and products in Silver because three sources describe them, and builds Gold only for finance and store operations.
Modeling still matters inside the layers: dimensional models for BI serving, wide denormalized tables for some data science use, and explicit history (snapshots or slowly changing dimensions) where the business asks “what did we know at the time?”. Medallion Architecture Explained covers each layer, and when fewer are enough, in depth.
Ingestion architecture
Choose ingestion tools by responsibility, not by habit. The most common anti-pattern is one tool doing everything: notebooks that copy, transform, orchestrate, retry and alert. The opposite anti-pattern is three tools doing one job. Assign each responsibility once:
| Responsibility | Fits best | Why | Watch out for |
|---|---|---|---|
| Orchestration: order, dependencies, schedules, retries, alerts | Pipelines | Built for control flow and run history; no data processing of their own | Business logic hidden in long chains of activities |
| Bulk and incremental copy from a source | Copy job, or the pipeline Copy activity | Managed movement with connectors; Copy job tracks incremental state itself | Write mode: Copy job appends by default, so reruns need merge or overwrite |
| Continuous replication of a supported database | Mirroring | Near-real-time replica in OneLake without building a pipeline | Read-only replica; transformation still happens downstream |
| Complex transformation, semi-structured parsing, large scale | Notebooks (Spark) | Code, libraries, scale and testability | Using a notebook to copy a table a connector could copy |
| Set-based relational transformation | T-SQL in the Warehouse | SQL skills, transactions, stored-procedure reuse | Row-by-row procedural logic ported from old systems |
| Low-code shaping by analysts | Dataflow Gen2 | Power Query skills, quick delivery | Large volumes and complex joins that belong in Spark or SQL |
| Events and streams | Eventstream into Eventhouse or Lakehouse | Continuous ingestion, routing and light processing | At-least-once delivery; replay depends on retention |
A few decisions belong to the architecture, not to each pipeline:
- One orchestration pattern. A parent pipeline per domain or schedule, child pipelines for reusable steps, notebooks and SQL for logic. Everyone should find a failure in the same place.
- One configuration pattern. Source lists, watermarks and connection details in metadata tables or variable libraries, not hard-coded in each item.
- One landing convention. Folder and table naming, load metadata columns, and file retention, defined once.
- Network paths decided early. On-premises sources need a gateway; private Azure sources may need a VNet data gateway or managed private endpoints. These have lead times measured in weeks.
Pipelines vs Notebooks vs Dataflow Gen2 vs Copy Job compares the four tools in depth. If you are replacing SSIS, the SSIS to Fabric migration guide shows how each package responsibility lands in a different Fabric item, which is the same responsibility split applied to an existing estate.
Transformation architecture
The transformation engine is a long-lived decision: it shapes hiring, testing, code review and on-call. Compare the options on the criteria that matter in production, not only on raw speed:
| Criterion | SQL (Warehouse T-SQL) | Spark / PySpark (notebooks, Spark jobs) | Dataflow Gen2 |
|---|---|---|---|
| Scale | Strong for set-based relational work | Strongest for very large, semi-structured or compute-heavy work | Best for modest volumes |
| Maintainability | High when logic is set-based and modular | High when code is structured into functions and libraries | High for simple shaping; harder to review as it grows |
| Skillset | SQL developers, analysts, DBAs | Data engineers comfortable with Python and Spark | BI developers, Power Query users |
| Observability | Query history and execution details from the engine | Spark monitoring plus your own logging | Refresh history; less detail for deep diagnosis |
| Testing | SQL unit tests, data checks | Unit tests on functions, data checks | Mostly data checks after refresh |
| Reuse | Views, procedures, functions | Libraries, parameterized notebooks | Limited reuse across dataflows |
| Operational complexity | Low to medium | Medium: sessions, pools, startup, concurrency | Low |
Javi Builds guidance: pick one primary transformation engine per layer and per team, and treat a second engine as a decision that needs a reason. Two engines in one layer double the testing patterns, the monitoring views and the skills on call. Mixing engines across layers is normal and healthy: Spark to conform raw events, T-SQL to build finance facts.
Notebooks are not the default for everything. A notebook that only copies a table a connector could copy pays for a Spark session, adds startup time and moves logic out of the pipeline’s run history. Debugging Fabric Notebooks and Pipelines shows what each engine gives you when something fails, which is part of choosing it.
Idempotency and restartability
This is one of the most important architecture decisions in the whole platform, because every production pipeline will fail, be retried and be rerun. A load is idempotent when running it twice for the same input leaves the same result as running it once. Without that property, a retry after a network blip becomes a data-quality incident: duplicated orders, double-counted revenue, a dashboard nobody trusts.
Microsoft’s own services assume you handle this. Eventstream documents at-least-once delivery, so duplicates are expected. Copy job keeps its incremental state and resumes from the end of the last successful run, but its default write mode is append, so a design that resets to a full copy without merge or overwrite creates duplicates. Spark jobs can fail mid-run during a zone outage and have to be resubmitted. Restartability is your job.
Design for it with these elements:
| Element | What it does | Design rule |
|---|---|---|
| Watermarks | Record how far a source has been read | Advance the watermark only after the target write is committed, never before |
| Deterministic loads | Same input window, same output | Bound every incremental window with a fixed upper limit captured at the start of the run, not “now” during the run |
| Merge / upsert | Apply changes by business key | MERGE or delete-then-insert by key or by business date; never blind append for data that can be resent |
| Checkpoints and state | Remember progress in long or streaming jobs | Keep state in durable storage, owned by the job, and back it up with the definitions |
| Duplicate prevention | Stop the same record or file from loading twice | Record processed file names, batch IDs or event IDs; deduplicate on a stable key |
| Retry design | Recover from transient failures | Bounded retries with backoff for transient errors only; a bad file is not transient |
| Partial failure | Handle a run that wrote some targets and not others | Each step idempotent on its own, so the whole run can be restarted from the start or from the failed step |
| Downstream consistency | Keep Gold and serving consistent with corrected inputs | Recompute affected slices (dates, keys) downstream, instead of adding deltas on top of deltas |
Retry vs rerun. A retry is automatic and immediate, for transient faults. A rerun is a deliberate decision to process a window again, for example after a source correction. Both must be safe, and both must be visible in the logs.
The failure scenario to design for is concrete: the ERP load reads orders up to 10:00, writes them to Silver, fails before updating Gold, and is retried at 10:05. With a watermark advanced after the Silver write and a Gold step that recomputes the affected business dates, the retry produces the same result as a clean run. With a watermark advanced at the start and an append into Gold, it double-counts a morning of revenue.
Incremental Loads in Microsoft Fabric walks through safe watermark advancement, MERGE, deletes, late data and reruns step by step. Designing an Execution Log shows how the log records watermarks and batch identity without blocking legitimate reruns, and Design Patterns for Data Engineering covers idempotent processing, quarantine and retry with backoff as reusable patterns.
Security architecture
This section stays at platform level: which boundaries exist, which decisions each one forces, and what to design before production. Microsoft Fabric Security Architecture for Production Data Platforms goes deep on each boundary, including access-path bypasses, multi-tenant isolation and service identities. Neither replaces a security review.
- IdentityMicrosoft Entra ID authenticates every request
- People in groups
- Workspace identity
- Service principals
- Key Vault referencesFor sources without Entra authentication
- signed inNetworkWho can reach the platform, and what it can reach
- Conditional Access
- Private Link
- Data gateways
- Managed private endpoints
- reachableWorkspaceEnvironment, domain and team boundary; roles apply to every item
- Dev
- Test
- Prod
- workspace role or shareItemItem permissions give access to one item without the workspace
- item accessDataEnforced where the data is read, per engine
- OneLake security
- SQL GRANT, RLS, CLS
- Semantic model RLS
- Sensitivity labels
The architecture decisions for each boundary:
| Area | Decision to make | Platform facts that shape it |
|---|---|---|
| Identity | Which identities exist: people, groups, deployment identities, run-time identities | Fabric authenticates every request with Microsoft Entra ID. A workspace identity is an automatically managed service principal Fabric creates for a workspace; it can authenticate connections and enable trusted access to firewall-enabled ADLS Gen2 |
| Least privilege | Who gets which workspace role; who only gets item access | Workspace Admin, Member and Contributor can use all items and read OneLake data; data-level rules do not restrict them. Consumers belong in Viewer or item-level access |
| Workspace boundaries | How many workspaces, split by environment, domain or team | Workspaces are the main permission and lifecycle boundary; items can also be shared without granting the workspace |
| Service and run-time identities | Which identity runs scheduled work and deployments | Production jobs and deployments should not run as a person who can leave. Use a service principal or workspace identity with the minimum roles needed |
| Secrets | Where credentials for non-Entra sources live | With Azure Key Vault references, a Fabric connection stores only the vault and secret name and fetches the latest secret version at run time. Support is per connector, so check the current list |
| Data access | Where row, column and object rules are enforced | OneLake security, SQL permissions with row- and column-level security, and semantic model security each protect a different read path. A rule defined in one path does not protect the others |
| Network and connectivity | Inbound restrictions; outbound routes to private sources | Conditional Access and Private Link restrict inbound access. On-premises and VNet data gateways, managed private endpoints and trusted workspace access cover outbound routes to protected sources |
| Environment separation | How Dev, Test and Prod differ in access and data | Separate workspaces per environment, different role assignments, and no production personal data in development without masking |
| Sensitive-data classification | How sensitive data is labelled and tracked | Microsoft Purview sensitivity labels classify Fabric items and follow the data to supported exports; workspace customer-managed keys add a layer of encryption at rest |
Three rules from Javi Builds practice tie it together:
- Design access by group and by workspace first, and grant item-level exceptions rarely. Joining or leaving a team should be an identity change, not a Fabric change.
- Test permissions through every engine your users will use. A user who reads the same table through Spark, the SQL endpoint and a report crosses three different enforcement points.
- Keep builders and consumers apart. People who build in a workspace can read everything in it, so consumers read from production through Viewer or item permissions, never as Contributors.
The governance architecture guide details which access control applies at which layer, and How to Secure SQL Endpoints in Azure covers private network access to SQL endpoints outside Fabric.
Observability architecture
Logging is not an afterthought. It is an architecture decision because it needs a schema, a store, a retention policy, identifiers that flow between components and an owner. Retrofitting it after go-live means the first serious incident is investigated without evidence.
Three kinds of evidence answer three different questions, and a data platform needs all three:
| Evidence | Tells you | In a data platform |
|---|---|---|
| Metrics | Something is wrong | Run duration against baseline, rows loaded against expected range, freshness lag, capacity utilization, failed-run counts |
| Logs | What happened | One record per run and step: what ran, which window, how many rows, which error, how many retries |
| Tracing / correlation | Where it happened | A correlation ID passed from the parent pipeline into child pipelines, notebooks and SQL procedures, so one failed business load is one query |
Metrics tell you something is wrong. Logs tell you what happened. Correlation tells you where it happened. In a data platform the “request” being traced is usually a business load, such as “finance orders for 7 November”, that crosses a pipeline, a notebook, a Warehouse procedure and a semantic model refresh.
The architecture should define, before the first pipeline, the fields every run records:
| Field | Purpose |
|---|---|
execution_id |
Identifies one run of one unit of work |
correlation_id |
Ties every run of one business load together across items |
source |
What was read, at object level |
target |
What was written |
start_time, end_time |
The execution window |
status |
Started, succeeded, failed, skipped |
rows_read, rows_written |
Volume signals for anomaly detection and reconciliation |
retry_count |
Separates transient noise from real failures |
error_category |
Source, connectivity, data quality, capacity, code: decides who is paged |
duration |
Compared against a baseline, not a fixed threshold |
sla_state |
On time, at risk, breached: the signal the business cares about |
Also decide who is alerted for what. A failed load at 02:00 that will be retried automatically is a log entry. A finance load that will miss 07:00 is a page. Alert on SLA risk and on data correctness, not on every failed activity.
Designing an Execution Log for Fabric Pipelines and Notebooks gives the full schema, SQL and Delta implementations, and status transitions. Monitoring Microsoft Fabric Pipelines and Notebooks in Production covers what Fabric’s built-in monitoring shows and where the log fills the gaps, and Monitoring a Data Platform: What Should You Actually Measure? and the monitoring checklist give the minimum set of signals for a first workload.
Capacity and performance architecture
Capacity selection is an architecture decision, not a post-production optimization. The platform’s workload profile, meaning what runs, when and how concurrently, decides whether a capacity is comfortable, saturated at peak or wasted overnight. Profile it during design:
| Profile element | Question | Running example |
|---|---|---|
| Expected workload | Which jobs, refreshes and query patterns exist? | CDC loads every 30 minutes, continuous events, nightly CRM, daily supplier files, finance refresh |
| Concurrency | How many jobs and users run at once? | Up to 400 report users; 4–6 concurrent loads |
| Peaks | When does demand concentrate? | Report peak 08:30–10:00; overnight load window 01:00–06:30 |
| Background operations | What runs without a user waiting? | Pipelines, notebooks, refreshes, table maintenance |
| Interactive workloads | What does a user wait for? | Report queries, ad hoc SQL |
| Workload overlap | Do heavy background jobs run during interactive peaks? | Risk: a late overnight load still running at 08:30 |
| Throttling risk | What happens when demand exceeds capacity? | Report queries delayed, then rejected, while the overnight backlog is paid off |
The platform facts that make this an architecture topic: in Microsoft Fabric, workloads share a capacity measured in capacity units. Background operations are smoothed over 24 hours and interactive operations over shorter windows, and throttling is staged. Interactive requests are delayed first, then rejected, and background requests are rejected when sustained overuse is extreme. A heavy job can therefore make reports slow hours after it ran, which is a design problem that no amount of query tuning on the report fixes.
The architecture responses, in the order Javi Builds recommends considering them:
- Shape the workload: incremental loads instead of full reloads, bounded retries, staggered schedules, no heavy background work in the interactive peak.
- Isolate when workloads conflict: production on its own capacity, development and test elsewhere, and a separate capacity for a workload that repeatedly disrupts a critical one.
- Size with evidence: a starting hypothesis from the workload profile, then measurement on a representative slice, then a decision on the peak window, not the daily average.
- Scale last, and on purpose: after the waste is removed, when sustained pressure comes from workloads that are already efficient.
Microsoft Fabric Capacity Explained covers capacity units, smoothing, throttling and sizing in depth. The Fabric Capacity Planner turns measured utilization into a sizing read before you commit to a SKU. In production, the Capacity Saturation Runbook is the incident procedure, and How I Diagnosed a Microsoft Fabric Capacity Saturation Problem shows an investigation end to end, including why a larger SKU was not the first answer. For engine-level tuning, see Troubleshooting Microsoft Fabric Performance in Production.
Cost architecture
Cost is a design input, not a monthly surprise. This section names cost drivers and the decisions that control them; it deliberately quotes no prices, because prices change and vary by region and agreement. Current rates are on the Microsoft Fabric pricing page.
| Cost driver | What drives it | Architecture decision that controls it |
|---|---|---|
| Compute (capacity) | The capacity size and how long it runs; in Fabric, capacity units are shared by every workload on the capacity | Workload shaping, isolation, right-sizing per environment, pausing capacities that are not needed |
| Storage | Data volume, copies, history and retention. OneLake storage is billed per GB, separately from capacity units, and continues to be billed while a capacity is paused. Soft-deleted data is billed at the same rate | Fewer unnecessary layers and copies, retention and vacuum policies, deliberate history |
| Data movement and transactions | Reads and writes to storage. OneLake transactions consume capacity units; reads through a shortcut count against the consumer’s capacity, and reads from an external shortcut target are charged by that external service | Shortcuts instead of copies where they fit; avoid rereading full tables; incremental processing |
| Concurrency | Many jobs or users at once raise peak demand, which is what sizing must cover | Scheduling, session reuse, semantic model design, isolation of heavy workloads |
| Idle resources | Capacities and sessions running with nothing to do | Pause non-production capacities out of hours; close idle Spark sessions; avoid always-on clusters for nightly work |
| Environment strategy | Every environment is a cost | Smaller capacities for Dev and Test, shared where isolation is not needed; sampled or synthetic data instead of full production copies |
| Scale-up vs scale-out | One larger capacity vs several smaller ones | One capacity pools headroom; several isolate workloads and support chargeback. Choose by isolation need, not by price alone |
| Resilience features | Geo-replication and retention | Enabling OneLake disaster recovery adds geo-replicated storage and higher write consumption, billed separately; enable it where the RPO requires it |
| Monitoring cost | Logs, diagnostics and metrics also consume storage and compute | Log one record per run, not per row; set retention for logs; sample verbose diagnostics |
Some Fabric facts change the arithmetic in useful ways. Mirroring’s background replication compute does not consume capacity, and mirrored storage is free up to a limit tied to the capacity size, which can make Mirroring cheaper than a hand-built copy pipeline for supported sources. Pausing a capacity stops compute billing, but any remaining smoothed usage is added to the bill at the moment you pause, so pausing does not erase a backlog.
Javi Builds guidance: estimate cost per workload, not per platform. “The finance load costs X capacity-seconds a day” is a number you can optimize and defend; “Fabric costs Y a month” is not. The Capacity Metrics app shows consumption by item and operation, and that is where the estimate is validated after go-live.
Resilience: RPO and RTO
Resilience starts with two numbers, defined by Microsoft’s Well-Architected Framework:
RPO: how much data can be lost
The recovery point objective is the maximum acceptable duration of data loss during an incident. An RPO of four hours means that after recovery, at most the last four hours of data may be missing and need to be reloaded or accepted as lost.
RTO: how long the platform can be unavailable
The recovery time objective is the maximum acceptable time the workload can be unavailable after an incident. An RTO of eight hours means reports and loads must work again within eight hours of the incident.
Both are business decisions with a cost. The architect’s job is to make that cost visible, then design to the agreed numbers, per workload. In the running example, finance reporting can accept an RPO of 24 hours (it can be rebuilt from the ERP) and an RTO of 8 hours, while store operations needs inventory within 2 hours and cannot lose events that the source will not resend.
Microsoft-managed availability vs your recovery architecture
The most important distinction in this section is who is responsible for what. Microsoft describes reliability as a shared responsibility: Microsoft keeps the platform available, and you design the recovery of your workload.
- Microsoft-managed availabilityPart of the service
- Zone-redundant storageWhere the region supports it; locally redundant elsewhere
- Zone failure handlingServices rebalance; in-flight Spark jobs or queries may need a restart
- OneLake geo-replicationOpt-in per capacity; asynchronous and file-level
- Regional failoverInitiated by Microsoft; portal and Power BI become read-only
- Your recovery architectureDesigned, tested and owned by you
- Definitions in GitNotebooks, pipelines and SQL objects can be redeployed
- Replayable sourcesRaw landing kept for sources that cannot resend
- Restartable loadsWatermarks, checkpoints and idempotent writes
- Rebuild capacity and itemsNew capacity, workspaces and items, then data restored
- Reconcile before reopeningRow counts and totals checked against sources
- Tested recovery runbookRehearsed, timed against the RTO
The Fabric platform facts that matter for this design:
- Within a region, OneLake uses zone-redundant storage where available and locally redundant storage elsewhere. During a zone outage no customer action is required, but running Spark jobs and Warehouse queries may fail and need to be resubmitted, which is why restartable loads matter even without a disaster.
- Across regions, disaster recovery is a capacity setting that replicates OneLake data, which includes Lakehouse and Warehouse data, to the Azure paired region. Replication is asynchronous and file-level, so after a failover the secondary may not hold a point-in-time-consistent copy. Data stored outside OneLake, such as KQL databases, needs its own approach. The setting cannot be changed again for 30 days, and it is billed as separate storage and operations.
- After a regional failover, Microsoft documents that the portal and Power BI are read-only, Lakehouse and Warehouse data is reachable through OneLake APIs, and notebook code content is not saved. Recovery means creating a new capacity, recreating workspaces and items, and restoring each item. That only works if definitions are in Git and the steps are rehearsed.
- No platform RPO replaces yours. The Fabric reliability documentation describes replication as asynchronous and does not give an RPO for it. Your RPO is met by your design: replayable sources, raw landing and restartable loads.
How RPO and RTO shape the design:
| Design element | Driven mostly by | Decision |
|---|---|---|
| Ingestion replay | RPO | Can every source resend a window? If not, raw landing becomes the system of record for that data and needs its own protection |
| Checkpoint and state | RPO and RTO | Watermarks and stream checkpoints in durable storage, so recovery resumes instead of restarting from zero |
| Retention | RPO | Raw retention at least as long as the longest realistic reprocessing need; source change retention long enough to cover an outage |
| Backups and restore behavior | RPO | Know each item’s restore model: Warehouse restore points, Delta time travel until vacuum, SQL database point-in-time restore. Different items, different procedures |
| Source replayability | RPO | Agree change retention with source owners (CDC retention, file archives, event retention) |
| Cross-region strategy | RTO and RPO | Enable OneLake disaster recovery only where the targets require it, and plan capacity in another region for recovery |
| Operational procedures | RTO | A written, rehearsed recovery runbook with owners, timed against the RTO |
SQL Server to Microsoft Fabric: What Maps to What? details the backup and restore model per Fabric item and how HA/DR changes for a SQL Server team.
CI/CD and environments
A production platform changes every week. The architecture decides how changes move, and how they are undone:
| Decision | What to define |
|---|---|
| Environments | Development, test and production as separate workspaces, with separate data, connections and access; production edited only through deployment |
| Source control | Which item definitions live in Git, the branch model, and pull-request review before anything reaches test |
| Deployment | One mechanism per artifact type (Fabric deployment pipelines, Git-based deployment, or a scripted tool such as fabric-cicd), run by a non-personal identity |
| Configuration | Environment-specific values (connections, workspace and item IDs, paths) parameterized outside the definitions, for example in variable libraries |
| Release validation | Automated checks after deployment: items exist, connections resolve, a smoke-test load runs, row counts are plausible |
| Rollback | Redeploy the previous version from Git; for data changes, a forward fix or restore procedure, because code rollback does not undo data that was written |
Two failure modes to design against: workspace drift, where production is edited by hand and Git no longer describes reality, and schema changes without data migration, where a deployment changes a table definition that existing data or consumers depend on.
Git Integration in Microsoft Fabric covers the development workflow, Microsoft Fabric CI/CD with fabric-cicd covers deployment from Git to DEV, QA and PROD with parameterized values, and the Fabric CI/CD Deployment Checklist is the release-day reference. Testing a Production Data Platform covers what release validation should prove, from unit tests to the quality gates that decide whether a change ships.
Governance
Governance is designed into the platform or it does not happen. If ownership, naming and classification are not decided before the first release, the platform grows faster than anyone can document it. Architecture-level governance covers:
| Area | Architecture decision |
|---|---|
| Ownership | Every source, table, pipeline and semantic model has a named owning team; “the platform team” is not the owner of business data |
| Domain boundaries | Which business domains exist and which workspaces belong to each; domains decide who sets rules for their data |
| Naming | One convention for workspaces, items, tables and columns, enforced in review |
| Metadata | Descriptions, owners and freshness expectations published where consumers search for data |
| Lineage | Source-to-report lineage that can answer “what breaks if this column changes?” |
| Classification | Sensitive data classified at ingestion, with labels that follow it downstream |
| Retention | Retention and deletion rules per dataset, including personal-data deletion requests |
| Access review | A recurring review of who holds workspace roles and item permissions, owned by the domain owner |
| Data quality ownership | Rules defined by the business owner, implemented by engineering, monitored by operations, with a named person who decides whether bad data is published or held |
In Fabric, domains, workspaces, the OneLake catalog, lineage, endorsement and Microsoft Purview sensitivity labels are the building blocks. The Microsoft Fabric governance guide shows how they fit together as layers, with a reference governance architecture and an operating model.
Operational ownership
Many platforms fail not on technology but on the question “whose problem is this?” at 06:40, twenty minutes before a deadline. Ownership has to be designed, written down and agreed. The matrix below is an example, not a universal organizational model. A small company may have one team holding every column; a large one may split the platform team further. What matters is that every row has exactly one owner.
| Area | Platform team | Data engineering | DBA / data platform | Security | Business / domain owner |
|---|---|---|---|---|---|
| Source ownership | Informed | Supports | Supports (database sources) | Consulted | Owns |
| Pipeline ownership | Supports | Owns | Consulted | — | Informed |
| Schema changes | Informed | Owns (platform side) | Consulted | Consulted (sensitive data) | Approves (source side) |
| Data quality | Informed | Implements | — | — | Owns the rules |
| Incidents | Owns platform incidents | Owns pipeline incidents | Supports | Owns security incidents | Informed, decides on publishing |
| Capacity | Owns | Supports | Supports | — | Informed |
| Security | Implements | Implements | Implements (database) | Owns policy and review | Approves access |
| Cost | Owns monitoring | Optimizes workloads | Optimizes queries | — | Owns budget |
| SLAs | Supports | Supports | Supports | — | Owns the target |
| Recovery | Owns the runbook | Supports | Supports (databases) | Consulted | Approves RPO and RTO |
A design review should reject any row with no owner, or with two owners and no tie-break. Write the matrix into the operations documentation and the on-call rotation, not only into the architecture deck.
The architecture lifecycle
An architecture is not finished at go-live. It is validated by operation and improved by measurement:
- 1DesignRequirements, NFRs, decisions recorded as ADRs
- 2BuildSmallest slice that exercises every tier
- 3ValidateLoad, failure, rerun and recovery tests against the NFRs
- 4OperateOwners, alerts, runbooks, on-call
- 5MeasureSLAs met, cost per workload, capacity at peak, incidents
- 6ImproveNew ADRs that supersede old ones; back to design
The step most often skipped is Validate. Before production, run the tests that prove the NFRs: a full-volume load inside the window, a peak-concurrency report test, a forced failure and rerun, a permission test through every engine, and a timed recovery rehearsal. A platform that has never been rerun or recovered on purpose will be rerun and recovered for the first time during an incident.
Architecture review checklist
Use this before approving a production data platform, or a major new workload on an existing one. Every unchecked item is either a decision still to make or a risk to accept explicitly, in writing.
Requirements
- Volume known: current rows and size per major source and table.
- Growth known: daily change volume, peak days and a three-year projection.
- Latency known: freshness target per consumer and per source, in minutes or hours.
- SLA defined: what must be ready, by when, measured how.
- RPO and RTO agreed: per workload, signed by the business owner.
Data
- Source ownership: a named owner and technical contact for each source.
- Schema evolution: how changes are announced, detected and handled.
- Retention: raw, curated and log retention, plus deletion procedures for personal data.
- Data quality: rules, owners, and what happens to data that fails them.
Security
- Identity: groups for people; non-personal identities for scheduled work and deployments.
- Least privilege: consumers are not workspace Contributors; builders do not edit production by hand.
- Sensitive data: classified, labelled, masked or restricted, and tested through every engine.
- Environment boundaries: separate workspaces, connections and data per environment.
Operations
- Monitoring: execution log with correlation IDs, freshness and volume checks.
- Alerts: on SLA risk and data correctness, routed to an owner.
- Retries: bounded, with backoff, for transient errors only.
- Reruns: every load idempotent; rerun procedure documented and tested.
- Support ownership: the responsibility matrix agreed, and on-call staffed.
Performance
- Concurrency: expected users and parallel jobs at peak.
- Peaks: interactive and background peaks identified and separated.
- Capacity: sized from a workload profile and measured on a representative slice.
- Workload isolation: production isolated from development; conflicting workloads separated.
Cost
- Major cost drivers: compute, storage, data movement and resilience estimated per workload.
- Expected growth: cost projected with data and user growth.
- Non-production strategy: smaller or paused capacities, reduced or synthetic data.
Recovery
- Replayability: each source can resend a window, or its raw data is protected as the only copy.
- Recovery runbook: written, owned and rehearsed against the RTO.
- Definitions in Git: every item needed to rebuild the platform can be redeployed.
Delivery
- Source control: item definitions in Git, with review before deployment.
- CI/CD: one deployment mechanism per artifact type, run by a non-personal identity.
- Rollback: code rollback tested; data correction procedure defined.
- Release validation: automated post-deployment checks and a smoke-test load.
Governance
- Ownership and domains: every dataset has an owning domain and team.
- Lineage and metadata: consumers can find data, its owner and its freshness.
- Access review: a recurring review with a named owner.
Recommendation
Start with requirements and write them as testable numbers. Assess every source before choosing a tool. Make each significant decision with the same six steps and record it as an ADR with validation criteria. Choose the slowest ingestion pattern that meets the latency need, the fewest layers that create real contracts, and one primary transformation engine per layer. Make every load idempotent before you make it fast. Design security, observability, capacity, cost, recovery, delivery and ownership before go-live, because each of them is cheap to design and expensive to retrofit. Then validate the design on purpose, operate it, measure it, and let the measurements write the next ADR.
What to read next
- Designing a Scalable Microsoft Fabric Architecture: workspace, storage, capacity and deployment boundaries inside a Fabric platform.
- SQL Server to Microsoft Fabric: What Maps to What?: the mental model for SQL Server teams, including backup, HA and DR.
- Migrate SSIS Packages to Microsoft Fabric: applying the responsibility split to an existing SSIS estate.
- Incremental Loads in Microsoft Fabric: watermarks, CDC,
MERGEand safe reruns. - Microsoft Fabric Capacity Explained: capacity units, smoothing, throttling and sizing.
- Microsoft Fabric Governance with Microsoft Purview: governance layers, access control and an operating model.
- Fabric Data Platform Template: the project that turns these decisions into a reusable platform layout.
Sources
Microsoft-documented behavior in this article comes from these Microsoft Learn pages:
- Reliability and recovery: Reliability in Microsoft Fabric (availability zones, disaster recovery setting, failover experience, recovery steps), OneLake disaster recovery and data protection (ZRS/LRS, asynchronous file-level replication, soft delete), Architecture strategies for defining reliability targets (RPO and RTO definitions)
- Architecture decisions: Maintain an architecture decision record
- Security: Security in Microsoft Fabric (Entra ID, Conditional Access, Private Link, gateways, managed private endpoints, trusted workspace access, workspace roles, customer-managed keys), Workspace identity, Configure Azure Key Vault references, OneLake security overview
- Capacity and cost: Understand capacity throttling and smoothing, Pause and resume your capacity, OneLake consumption (storage billing, transactions, shortcuts, paused capacity, disaster recovery billing), Buy Fabric capacity, Microsoft Fabric pricing
- Ingestion: What is Copy job (incremental state, resume after failure, write modes), Mirroring overview (replication, cost of mirroring), Eventstreams overview (at-least-once delivery, retention), Data movement decision guide
- Storage and serving: Lakehouse and Warehouse decision guide, What is a lakehouse?, What is Fabric Data Warehouse?, SQL database in Fabric
- Delivery and governance: Git integration, Deployment pipelines, Variable library integration with pipelines, Governance and compliance in Microsoft Fabric
The requirement-to-consequence table, the six-step decision framework, the source assessment template, the pattern and engine comparisons, the cost-driver table, the ownership matrix and the review checklist are Javi Builds architectural guidance, not Microsoft documentation. The running example is invented and contains no customer or employer details.
Fabric behavior checked against Microsoft Learn on 8 November 2026. Product capabilities, connector support and preview status change often; recheck the linked pages before relying on a specific limit or feature.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetFabric Medallion ArchitectureDesign a practical Bronze, Silver, and Gold architecture in Microsoft Fabric with clear data contracts, replayability, incremental processing, quality gates, and serving responsibilities.Open cheat sheet
- ResourceMicrosoft Fabric Certification Study PlannerAn adaptable 6–8 week example plan that connects official Fabric certification objectives with hands-on practice, review, and exam readiness.View resource
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.