DevOps & Observability
Microsoft Fabric CI/CD with fabric-cicd: From Git to DEV, QA and PROD
A production guide to deploying Microsoft Fabric items from Git with Microsoft’s fabric-cicd library, covering repo layout, authentication, parameterization, CI and CD.
On this page
Connecting a Fabric workspace to Git gives you version history. It does not give you a release process. Sooner or later someone has to answer harder questions: how does a reviewed notebook reach QA, how does the production copy end up pointing at the production Lakehouse, and how do you know that what is running in PROD is what is in main?
Microsoft’s open-source fabric-cicd library answers those questions with a small Python API. This guide shows how I use it in production: where it fits, how to structure the repository, how to authenticate, how environment-specific IDs are replaced, and what CI and CD should each be responsible for. The examples use Azure DevOps, but the same ideas apply to GitHub Actions.
Why fabric-cicd exists
Three activities are often blurred together:
- Development in Fabric. Engineers build notebooks, pipelines and models in a workspace, usually through the Fabric UI.
- Source control. Fabric Git integration serializes each item into files (
.platformmetadata plus definition files such asnotebook-content.pyorpipeline-content.json) and syncs them with a branch. - Deployment. The same definitions are applied to another workspace, with environment-specific values replaced, in a repeatable and auditable way.
Git integration covers the first two. It is a sync mechanism between one workspace and one branch, not a promotion mechanism between environments. fabric-cicd covers the third: it reads item definitions from a repository directory and calls the Fabric REST APIs to create or update those items in a target workspace.
That distinction has an operational consequence. Workspaces that receive deployments are not development workspaces. If people edit items directly in QA or PROD, the next deployment overwrites the change, or worse, the change survives in some items and not others. A deployed workspace should only change when the pipeline runs.
The deployment model
- 1Feature branchCreated from main
- 2Development workspaceConnected through Git Sync
- 3Pull requestReview and CI validation
- 4mainProtected branch
- 5fabric-cicd deploymentRelease pipeline
- 6QADeployed workspace
- 7PRODSame commit, approved
The flow follows the Git flow and deployment philosophy that Microsoft documents for fabric-cicd:
- Feature branches are connected to development workspaces through Git Sync. That is where people work in the Fabric UI.
- Deployed workspaces are not connected to Git. They are updated only by script-based deployments.
- Every run is a full deployment. fabric-cicd does not inspect commit history or compute diffs. Every item whose type is in
item_type_in_scopeis created or updated on each run, so the target converges to the repository state regardless of what happened before. - Orphans are removed explicitly.
unpublish_all_orphan_items()deletes items of in-scope types that exist in the workspace but no longer exist in the repository.
Selective deployment and Git-based change detection exist as experimental features, but Microsoft does not recommend selective deployment as the default because unresolved dependencies can break a release. Deploying everything in scope is slower than deploying one changed notebook, but it is predictable, and predictability is the point.
Microsoft’s reference scenario uses one deployed branch per environment, with features merged into the default branch and cherry-picked upward. A common variant, and the one used in this guide, keeps a single protected main and promotes the same commit through QA, STAGING and PROD with approvals between stages. Both keep the essential rule: deployed workspaces are never edited by hand and never Git-synced.
Installation
fabric-cicd is a Python package:
pip install fabric-cicd
In a pipeline, pin the version you validated, for example in a requirements-deploy.txt file, so a new release never changes deployment behavior silently. Read the changelog before upgrading; authentication requirements, for example, have changed between versions.
If you prefer a command line to a Python script, the Fabric CLI runs fabric-cicd under the hood through fab deploy, using a config.yml file and the same parameter.yml:
pip install ms-fabric-cli
fab deploy --config config.yml --target_env prod
Repository structure
fabric-cicd deploys from a directory that contains the items committed through Fabric source control, plus an optional parameter.yml at its root. Keep anything else, such as deployment scripts and pipeline definitions, outside that directory.
.
├── .deploy/
│ └── deploy_workspace.py
├── pipelines/
│ ├── fabric-release.yml
│ └── templates/
│ └── deploy-fabric.yml
├── requirements-deploy.txt
└── workspace/
├── Ingest Orders.Notebook/
│ ├── .platform
│ └── notebook-content.py
├── Load Orders.DataPipeline/
├── Sales.Lakehouse/
├── Sales Warehouse.Warehouse/
├── Platform Settings.VariableLibrary/
├── Sales.SemanticModel/
├── Sales Overview.Report/
└── parameter.yml
Each item lives in a folder named <item name>.<item type>. Workspace subfolders are supported, so you can group items the way you group them in Fabric. The development workspace’s Git connection should point at this workspace/ folder, so what engineers commit from the UI is exactly what the pipeline deploys.
Decide early which item types the pipeline owns. The projects I work on deploy Notebook, DataPipeline, Lakehouse, Warehouse, VariableLibrary, SemanticModel and Report. Every type you add to scope becomes something the pipeline creates, updates and can unpublish, so add types deliberately.
Authentication
Current versions of fabric-cicd require an explicit credential. The token_credential parameter must be an object that implements TokenCredential from azure-identity. Two older behaviors are no longer supported: the DefaultAzureCredential fallback, and implicit authentication inside a Fabric notebook without a token_credential. Scripts that relied on either now fail at authentication, by design.
| Where the deployment runs | Credential | Notes |
|---|---|---|
| Local development | AzureCliCredential or AzurePowerShellCredential |
Uses the identity from az login or Connect-AzAccount |
| Azure DevOps with a service connection | AzureCliCredential or AzurePowerShellCredential |
Run the script inside an AzureCLI@2 or AzurePowerShell@5 task; the platform handles the login |
| Pipeline with a service principal secret | ClientSecretCredential |
Read the secret from a secure variable or Key Vault, never from the repository |
| Self-hosted agent on Azure | ManagedIdentityCredential |
No secret to rotate; the agent’s managed identity needs workspace access |
| Fabric notebook | An explicit credential | The authentication examples show a notebook session and a service principal pattern |
In Azure DevOps, I prefer AzureCliCredential inside an AzureCLI@2 task with a service connection. The pipeline platform owns the secret or federated credential, and the Python script contains no authentication logic beyond one line.
Whatever identity runs the deployment needs permission to create and update items in each target workspace. It also needs access to any connections that data pipelines use; if it cannot see a connection, the pipeline item fails to deploy.
The deployment script
The script is intentionally small. Everything that varies by environment comes from the pipeline: the target workspace and the environment name.
"""Deploy the Fabric items in ./workspace to one target workspace."""
import os
import sys
from pathlib import Path
from azure.identity import AzureCliCredential
from fabric_cicd import (
FabricWorkspace,
change_log_level,
publish_all_items,
unpublish_all_orphan_items,
)
# Stream output immediately in pipeline logs
sys.stdout.reconfigure(line_buffering=True, write_through=True)
sys.stderr.reconfigure(line_buffering=True, write_through=True)
if os.getenv("SYSTEM_DEBUG", "false").lower() == "true":
change_log_level("DEBUG")
repo_root = Path(__file__).resolve().parent.parent
target_workspace = FabricWorkspace(
workspace_id=os.environ["TARGET_WORKSPACE_ID"],
environment=os.environ["TARGET_ENVIRONMENT"], # DEV, QA, STAGING or PROD
repository_directory=str(repo_root / "workspace"),
item_type_in_scope=[
"Lakehouse",
"Warehouse",
"VariableLibrary",
"Notebook",
"DataPipeline",
"SemanticModel",
"Report",
],
token_credential=AzureCliCredential(),
)
publish_all_items(target_workspace)
unpublish_all_orphan_items(target_workspace)
The environment value selects the replacements in parameter.yml and the active value set of each Variable Library. It must match the keys used there exactly. fabric-cicd publishes item types in its own dependency order (Variable Libraries and Warehouses before the items that reference them), so the order of item_type_in_scope does not matter.
DEV, QA, STAGING and PROD
| Workspace | How it changes | Who edits it |
|---|---|---|
| Feature or development workspaces | Git Sync with a feature branch | Engineers, through the Fabric UI |
| DEV (integration) | Pipeline deployment from main |
Nobody directly |
| QA | Pipeline deployment, after DEV | Nobody directly |
| STAGING | Pipeline deployment, with approval | Nobody directly |
| PROD | Pipeline deployment of the same commit, with approval | Nobody directly |
Not every team needs four deployed environments. What matters is the boundary: the only workspaces that people edit are Git-synced development workspaces, and every other environment is a deployment target.
Give each environment its own workspace, its own Lakehouse and Warehouse data, and its own connections. Deploying item definitions does not move data; a PROD notebook that still points at the DEV Lakehouse is the most common, and most dangerous, deployment mistake. Decide capacity placement too: a QA load test running on the production capacity can throttle production reports (how capacity is shared).
Parameterizing environment-specific values
Item definitions committed from the development workspace contain development identifiers: the default Lakehouse of each notebook, the workspace that owns it, SQL analytics endpoint IDs in pipelines and semantic models, connection IDs. parameter.yml tells fabric-cicd how to replace them for each environment. It sits at the root of the repository directory and is validated at the start of every deployment; an invalid file stops the run.
find_replace:
# Default Lakehouse attached to notebooks
- find_value: "your-dev-lakehouse-id"
replace_value:
_ALL_: "$items.Lakehouse.Sales.$id"
item_type: "Notebook"
# Workspace that owns the default Lakehouse
- find_value: "your-dev-workspace-id"
replace_value:
_ALL_: "$workspace.$id"
item_type: "Notebook"
# Lakehouse SQL analytics endpoint used by pipelines and the semantic model
- find_value: "your-dev-lakehouse-sql-endpoint-id"
replace_value:
_ALL_: "$items.Lakehouse.Sales.$sqlendpointid"
item_type: ["DataPipeline", "SemanticModel"]
key_value_replace:
# Run schedules only in production
- find_key: $.schedules[?(@.jobType=="Execute")].enabled
replace_value:
DEV: false
QA: false
STAGING: false
PROD: true
file_path: "**/.schedules"
semantic_model_binding:
default:
connection_id:
DEV: "your-dev-connection-id"
QA: "your-qa-connection-id"
STAGING: "your-staging-connection-id"
PROD: "your-prod-connection-id"
A few details make this file reliable:
- Prefer dynamic values to hard-coded GUIDs.
$workspace.$idresolves to the target workspace and$items.Lakehouse.Sales.$idto the Lakehouse namedSalesin that workspace.$sqlendpointidreturns a Lakehouse’s SQL analytics endpoint ID. You no longer maintain a table of production GUIDs, and the same entry works for every environment. _ALL_applies one value to any environment. It must be the only key in thatreplace_value.- Filter every replacement.
item_type,item_nameandfile_pathlimit where a value is replaced. An unfiltered GUID replacement touches every file in the repository. - Use regex only with context.
find_replacesupportsis_regex, for example to capture the GUID after# META "default_lakehouse":. Never match the bare GUID format, or you will replace unrelated IDs. - Name environments consistently.
QAandqaare different keys. If the environment passed toFabricWorkspaceis missing from an entry, that replacement is skipped.
Some limits are worth knowing before you rely on parameterization. find_replace is not applied to Lakehouse or Warehouse items. Connections are not source controlled, so they must exist in each environment before deployment. When any dynamic variable is used, fabric-cicd resolves SQL endpoints for every Lakehouse and Warehouse in the target workspace; if one is still provisioning, the deployment fails before anything is published.
Variable Libraries complement parameter.yml. Name each value set after an environment (DEV, QA, PROD), and fabric-cicd activates the set that matches the environment value. Keep runtime configuration that notebooks and pipelines read in Variable Libraries, and keep item-to-item wiring, such as default Lakehouses and endpoint IDs, in parameter.yml.
What happens in CI
CI runs on every pull request into main. It never deploys to QA or PROD. Its job is to stop a broken definition from reaching the branch that releases are built from:
- Protected
main: changes arrive only through pull requests, with at least one reviewer and a passing build. - Structure checks: every folder under
workspace/follows<name>.<type>and contains a.platformfile. - Parameter validation:
parameter.ymlparses, every environment key is spelled consistently, and every dynamic variable references an item that exists in the repository. fabric-cicd ships adebug_parameterization.pyscript in its repository that validates the file without deploying. - Leaked identifiers: fail the build if a known development workspace or Lakehouse ID appears in a file that
parameter.ymldoes not cover. - Secrets: no client secrets, keys or connection strings in item definitions.
Some teams also deploy each merged commit to the shared DEV workspace as the first CD stage. That is a useful integration test: it proves the commit deploys before anyone asks QA to look at it.
What happens in CD
The release pipeline deploys one commit to each environment in turn. In Azure DevOps, a reusable stage template keeps every environment identical except for its inputs:
# pipelines/templates/deploy-fabric.yml
parameters:
- name: environment
type: string
- name: serviceConnection
type: string
- name: workspaceId
type: string
- name: dependsOn
type: object
default: []
stages:
- stage: Deploy_${{ parameters.environment }}
dependsOn: ${{ parameters.dependsOn }}
jobs:
- deployment: Deploy
environment: fabric-${{ parameters.environment }}
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: UsePythonVersion@0
inputs:
versionSpec: '3.12'
- script: pip install -r requirements-deploy.txt
displayName: Install fabric-cicd
- task: AzureCLI@2
displayName: Deploy to ${{ parameters.environment }}
inputs:
azureSubscription: ${{ parameters.serviceConnection }}
scriptType: bash
scriptLocation: inlineScript
inlineScript: python -u .deploy/deploy_workspace.py
env:
TARGET_ENVIRONMENT: ${{ parameters.environment }}
TARGET_WORKSPACE_ID: ${{ parameters.workspaceId }}
# pipelines/fabric-release.yml
trigger:
branches:
include:
- main
pr: none
stages:
- template: templates/deploy-fabric.yml
parameters:
environment: QA
serviceConnection: your-qa-service-connection
workspaceId: your-qa-workspace-id
- template: templates/deploy-fabric.yml
parameters:
environment: STAGING
serviceConnection: your-staging-service-connection
workspaceId: your-staging-workspace-id
dependsOn: [Deploy_QA]
- template: templates/deploy-fabric.yml
parameters:
environment: PROD
serviceConnection: your-prod-service-connection
workspaceId: your-prod-workspace-id
dependsOn: [Deploy_STAGING]
Approvals and checks live on the Azure DevOps environments (fabric-STAGING, fabric-PROD), not in the YAML, so nobody can remove them in a pull request. Workspace IDs are not secrets, but keeping them in a variable group per environment avoids editing the pipeline when a workspace is recreated. A separate service connection per environment means the QA identity cannot touch PROD.
Because the template is shared, a fix to the deployment process reaches every environment and every Fabric repository that uses it. That is the main reason to centralize pipeline templates rather than copy them between projects.
Avoiding workspace drift
Drift is the difference between what the repository says and what a workspace contains. A full deployment removes most of it on every run, but only for what is in scope:
- Restrict permissions. In deployed workspaces, people get Viewer or limited roles; the deployment identity is the only principal that changes items.
- Send hotfixes through Git. A fix made directly in PROD is overwritten by the next release. Make it in a branch, merge it and deploy it.
- Decide what unpublishing means.
unpublish_all_orphan_items()deletes in-scope items that are not in the repository. Lakehouse and Warehouse unpublishing is disabled by default to protect data. Useitem_name_exclude_regexfor items that are intentionally managed outside Git. - Know what is not deployed. Connections, notebook resources, Lakehouse data and schemas (unless a shortcut defines them) and Warehouse DDL are outside fabric-cicd. Only the Warehouse item shell is deployed; deploy its schema with a DACPAC, dbt or another database tool.
- Never Git-sync a deployed workspace. It creates a second, competing way to change the same items.
What to validate after deployment
A green pipeline means the APIs accepted the definitions. It does not prove the platform works. After each deployment, check:
- Every item reported success. A failure should stop the stage, not leave a partially updated workspace that looks finished.
- Notebooks point at the target Lakehouse. Open one or query its metadata; the default Lakehouse and its workspace must belong to the target environment, never DEV.
- Pipelines call target items and use the environment’s connections.
- Semantic models are bound and refresh. Run a refresh and confirm it reads the target SQL endpoint.
- Reports open against the deployed semantic model.
- Variable Libraries have the expected active value set.
- Schedules are enabled only where they should run.
- A smoke run succeeds: a small pipeline or notebook run that exercises the main path and appears in your pipeline and notebook monitoring.
- The release is recorded: commit ID, environment, approver and result.
The printable Fabric CI/CD deployment checklist covers the same steps before, during and after a release. These checks verify the deployment; Testing a Production Data Platform covers the gates before it that verify the change itself. Testing Microsoft Fabric Pipelines and Notebooks shows how to run those checks from CI as a gate: trigger a test run, poll it, assert on the data and block promotion.
Common mistakes and failure modes
- Relying on default authentication. Without an explicit
token_credential, current fabric-cicd versions fail. Old scripts that never passed one stop working after an upgrade. - Mismatched environment names.
Prodin the pipeline andPRODinparameter.ymlmeans no replacement happens, and production items keep development IDs. - A
find_valuethat does not match. The committed files contain a different GUID than the one inparameter.yml, often after someone re-attached a Lakehouse in the development workspace. - Unfiltered replacements that change IDs in files they were never meant to touch.
- A SQL endpoint still provisioning in the target workspace, which fails the whole deployment when dynamic variables are used.
- Missing connection access for the deployment identity, which fails data pipeline items.
- Renaming or removing a Variable Library value set after the first deployment, which fails later deployments until it is fixed in the Fabric UI.
- Recreating an item with the same name too quickly. Fabric reserves deleted item names for a few minutes.
- Throttling. Large workspaces can hit
429 Too Many Requests; retry later or split very large deployments. - Selective deployments by default, which leave dependencies unresolved.
- Expecting data or Warehouse schemas to move with the items.
When something fails, enable debug logging with change_log_level() (or set SYSTEM_DEBUG in Azure DevOps with the script above) and read the troubleshooting guide. Set FABRIC_CICD_FILE_LOGGING_ENABLED if you want the detailed fabric_cicd.error.log file.
Practical recommendations
- Treat Git as the only way into a deployed workspace. Development workspaces are Git-synced; everything else is a deployment target.
- Keep the deployment script small and generic. Environment, workspace ID and credential come from the pipeline.
- Use dynamic variables (
$workspace.$id,$items.<type>.<name>.$id,$sqlendpointid) instead of maintaining production GUIDs. - Filter every replacement by item type, item name or file path.
- Promote one commit through QA, STAGING and PROD, with approvals on the pipeline environments.
- Use a separate deployment identity and service connection per environment.
- Pin the fabric-cicd version and upgrade deliberately after reading the changelog.
- Deploy Warehouse schemas and create connections with their own tooling, before the items that depend on them.
- Validate the platform after every release, not just the pipeline result.
Done this way, fabric-cicd turns Fabric deployments into something boring: the same commit, the same script and the same checks in every environment. For the wider platform design that these environments sit in, see Designing a Scalable Microsoft Fabric Architecture.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetKQL for Microsoft Fabric Cheat SheetThe Kusto Query Language (KQL) a Microsoft Fabric data engineer uses most in Eventhouse and KQL querysets: filter, shape, sort, summarize, time windows with bin(), joins, nulls and dynamic fields, with a SQL-to-KQL mapping and seven practice exercises.Open cheat sheet
- ArticleGit Integration in Microsoft Fabric: A Practical Development WorkflowHow Fabric Git integration really works day to day: workspace and branch, commit and update, branching out, pull requests, conflicts, drift and permissions.Read article
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.