Microsoft Fabric
Pipelines vs Notebooks vs Dataflow Gen2 vs Copy Job in Microsoft Fabric
A practical decision guide for Fabric data integration: when to use a pipeline, a notebook, Dataflow Gen2 or a Copy job, how they combine, and what each means for CI/CD, debugging and capacity.
Dans cet article
A team new to Fabric opens the New item menu and finds four reasonable ways to get data from a source into a Lakehouse: a pipeline, a notebook, a Dataflow Gen2 and a Copy job. All four can move data. Three can transform it. All four can run on a schedule. Every one of them can sit inside a larger workflow.
So the team picks the one that matches its background. SQL people build pipelines, Python people write notebooks, BI people reach for Power Query. Six months later, the same business rule exists in a dataflow and a notebook, a notebook is orchestrating other notebooks with try/except blocks, and a pipeline contains forty activities that encode what is really a transformation.
The tools are not competitors. They are good at different parts of the problem:
- Pipelines orchestrate. They decide what runs, in what order, under which conditions, and what happens on failure.
- Notebooks process with code. Spark, PySpark and Spark SQL for transformations that need engineering: complex logic, large volumes, reusable libraries, tests.
- Dataflow Gen2 transforms with Power Query. A low-code, connector-rich way to shape data, familiar to BI developers and analysts.
- Copy jobs move data. Source-to-destination copies with built-in incremental and change data capture options, without building a pipeline.
This article is a decision guide, not a product tour. It compares the four on the things that matter in production (logic complexity, team skills, CI/CD, debugging and capacity), works through concrete scenarios, and shows how they combine. It assumes you know the Fabric basics.
Product capabilities were checked against Microsoft Learn in October 2026. Data Factory changes quickly; preview features are marked as such.
Why this choice is confusing
The overlap is real:
| Capability | Pipeline | Notebook | Dataflow Gen2 | Copy job |
|---|---|---|---|---|
| Move data | Yes (Copy activity) | Yes (Spark reads and writes) | Yes (connectors to destinations) | Yes, its main purpose |
| Transform data | Light (mapping, through other activities) | Yes, any logic you can code | Yes, Power Query transformations | Light (column mapping, source queries) |
| Run on a schedule | Yes | Yes | Yes | Yes, including multiple schedules |
| Run inside a pipeline | Yes (Invoke pipeline) | Yes (Notebook activity) | Yes (Dataflow activity) | Yes (Copy job activity) |
What differs is what each one is designed for. Every tool can stretch into the others’ territory, and stretching is where the trouble starts. A notebook can loop over files and copy them; a pipeline can chain enough activities to approximate business logic; a dataflow can grow into hundreds of steps. Each works on day one and becomes the hardest part of the platform to maintain.
The four tools in one table
| Pipeline | Notebook | Dataflow Gen2 | Copy job | |
|---|---|---|---|---|
| Best for | Orchestration: sequencing, branching, retries, parameters across steps | Code-heavy transformation and engineering at scale | Low-code transformation with Power Query | Straightforward data movement, including incremental and CDC copies |
| Main interface | Visual canvas of activities | Code cells (PySpark, Spark SQL, Scala, R) | Power Query editor | Guided wizard and settings |
| Logic complexity | Control flow; not row-level logic | Unlimited | Moderate; complex logic gets hard to maintain | Minimal by design |
| Transformation depth | Delegates to other activities | Deep | Moderate to deep, within Power Query | Column mapping and source queries |
| Data movement | Copy activity, fully configurable | Through Spark | Through connectors and destinations | Native: full, incremental (watermark) and CDC-based copies |
| Orchestration | Native | Possible in code, rarely wise | No; it is orchestrated | No; it is orchestrated or scheduled |
| Code required | No (expressions for dynamic content) | Yes | No (M is available) | No |
| Low-code support | Yes | No | Yes | Yes |
| Git / CI/CD | Supported | Supported | Supported; deployed changes need a publish before they run | Supported |
| Debugging | Per-activity run history, inputs and outputs | Cell output, Spark UI, your own logging | Refresh history per query | Run history per table |
| Reusability | Parameters, Invoke pipeline | Functions, modules, libraries, parameters | Parameters; logic tied to the dataflow | One job, many tables |
| Typical team | Data engineers, platform team | Data engineers | BI developers, analysts | Anyone who needs to land data |
| Common mistake | Encoding business rules in activities | Using it to copy files or orchestrate | Pushing heavy engineering logic into Power Query | Building a pipeline when a Copy job would do, or the reverse |
Pipelines
A pipeline is Fabric Data Factory’s orchestrator. Its value is control, not processing:
- Activities do the work. Data movement (Copy data, Copy job), transformation (Notebook, Spark job definition, Dataflow Gen2, Stored procedure, Script) and control flow (Invoke pipeline, ForEach, If condition, Switch, Until, Lookup, Get metadata, Fail, Wait, Web, Teams and more).
- Dependencies connect activities on success, failure, completion or skip, which is how you express “validate only if the load succeeded” or “notify on any failure”.
- Triggers start runs on a schedule or in response to events.
- Parameters and variables pass run-specific values, such as a load date, a window or an environment, into every activity.
- Retries and timeouts are activity settings: by default an activity times out after 12 hours, and you can enable retries with a count and interval.
A pipeline should read like a runbook: copy, transform, validate, notify. If you cannot tell what a pipeline does from its canvas, it is doing too much itself. Two published guides go further: Invoke Pipeline activity for parent-child pipelines, and notebook parameters for passing values from a pipeline into code.
Notebooks
A notebook runs code on Spark. It is the right tool when the transformation needs engineering:
- Complex logic: joins across many tables, window functions, deduplication, slowly changing dimensions, incremental
MERGE, parsing semi-structured data. - Large volumes, where Spark’s distributed execution matters.
- Reusable engineering logic: functions and modules shared across notebooks, custom libraries attached through Fabric environments, unit-testable code.
- Anything a visual tool cannot express cleanly, such as calls to APIs, custom validation frameworks or ML feature preparation.
Notebooks are overkill when the job is a plain copy, such as landing a table from SQL Server or moving files from one folder to another. You pay for a Spark session to start and run, you write and maintain code that a Copy job or Copy activity provides as configuration, and you lose built-in features such as connection management, incremental state and per-table run history.
For a single notebook, the Microsoft Fabric notebook tutorial goes from the first cell to production patterns.
Dataflow Gen2
Dataflow Gen2 is Power Query in Fabric: the same transformation editor BI developers know from Power BI, with Fabric destinations such as Lakehouse and Warehouse tables.
It is a strong choice when:
- The team thinks in Power Query. BI developers and analysts can build and maintain the transformation without learning Spark.
- The logic is shaping, not engineering. Cleaning columns, filtering, merging and appending queries, pivoting, type fixes, lookups.
- The source is connector-driven. Excel, SharePoint lists, SaaS applications, files with awkward layouts: Power Query’s connector library and data preparation are its strength.
- The output feeds reporting, for example preparing reference data or a curated table for a semantic model.
It is a weaker choice when the logic needs software engineering: deep branching, reusable libraries, extensive testing, or many steps that would be a few lines of PySpark. Power Query can do a great deal, but very large dataflows are hard to review and debug. Dataflow Gen2 refreshes run as background operations and can use staging items behind the scenes, so watch their capacity use as volumes grow.
Copy job
A Copy job is a Data Factory item dedicated to data movement, without building a pipeline:
- Copy modes: full copy, or incremental copy where the first run copies everything and later runs copy only changes.
- Change detection: watermark-based, on a column such as
ROWVERSION, a datetime or an integer, or CDC-based when CDC is enabled on a supported source. The Copy job tracks its own state and, after a failure, resumes from the end of the last successful run. - Write behavior: append by default, or merge (upsert on a key), overwrite, or SCD Type 2 for supported destinations. It can create destination tables automatically.
- Extras: many tables in one job, source queries to copy subsets, audit columns (extraction time, run ID, window bounds), multiple schedules, and connections through on-premises or virtual network gateways.
Some capabilities were in preview at the time of writing: CDC replication for some connectors, SCD Type 2 with CDC, and auto-partitioning for large tables. Check the current connector support for your source before designing around a specific mode.
A Copy job is often simpler than a pipeline for ingestion: there is no canvas to build, no watermark table to maintain, and per-table monitoring is built in. When the copy needs to be one step in a larger workflow, the pipeline Copy job activity runs it inside a pipeline.
The most common architecture
Most well-structured Fabric solutions use more than one of these tools, each in its lane:
- PipelineSchedule, parameters, dependencies, retries
- orchestratesCopy job or Copy activityLand source data in Bronze
- NotebookClean, conform, MERGE, aggregate
- Lakehouse or WarehouseSilver and Gold tables
- Semantic model and reportsDirect Lake or SQL consumers
The End-to-End Sales Analytics lab builds a version of this: notebooks for Bronze, Silver and Gold, orchestrated by one pipeline. Dataflow Gen2 slots in where a transformation is better owned in Power Query, usually feeding a Lakehouse or Warehouse table that the rest of the flow consumes.
Decision scenarios
Scenario A: copy 300 files every morning into Bronze
Primary tool: Copy job, or a pipeline Copy activity with a ForEach loop. This is data movement. If the files follow a predictable folder and naming pattern and the default behavior fits, a Copy job is the least to build. If you need metadata-driven control (a control table listing sources, per-file parameters, custom error handling), use a pipeline with Copy activities. A notebook only joins in after landing, to parse and transform. Do not write a notebook loop that copies files.
Scenario B: apply complex business rules to 100 million rows
Primary tool: notebook. Joins, deduplication, rules with many exceptions, incremental MERGE and large volumes are what Spark and code are for. A pipeline schedules the notebook and passes it the processing window; it does not implement the rules. For applying only the changes, see Incremental Loads in Microsoft Fabric.
Scenario C: a business analyst needs to clean Excel and CSV data with Power Query
Primary tool: Dataflow Gen2. The person who owns the logic knows Power Query, the sources are files with awkward layouts, and the transformations are shaping steps. Land the result in a Lakehouse or Warehouse table. If the dataflow becomes a step in a nightly process, a pipeline runs it with a Dataflow activity.
Scenario D: move tables from SQL Server to Fabric with minimal transformation
Primary tool: Copy job, or Mirroring if continuous replication of a supported source is what you need. A Copy job handles many tables, incremental copies by watermark or CDC, and merge writes, with an on-premises gateway for a SQL Server behind a firewall. Notebooks take over downstream if Silver and Gold need real transformation. ETL vs ELT vs CDC covers the integration patterns behind this choice. If those tables are loaded by SSIS packages today, Migrate SSIS Packages to Microsoft Fabric helps decide which packages to run as they are and which to rebuild.
Scenario E: run copy, then notebook, then validation, then notify
Primary tool: pipeline. This is orchestration by definition: ordered steps, a dependency on success, a branch on failure, and a notification. A Copy job activity or Copy activity lands the data, a Notebook activity transforms and validates it (failing deliberately if validation fails), and a Teams or Web activity notifies on the failure path. Keep the validation logic in the notebook and the decision about what happens next in the pipeline.
Scenario F: build a reusable transformation library with tests
Primary tool: notebooks and Spark code, packaged as functions or a library attached through a Fabric environment, and kept in Git. Testable, reusable logic is a software problem, and code is the only one of the four tools built for it. Pipelines call the notebooks that use the library.
When to combine tools
Composition is the normal case, not the exception:
- Pipeline + Notebook. The most common pairing: the pipeline owns scheduling, parameters, retries and branching, and the notebook owns the transformation. Pass values in through the notebook’s parameter cell, and return a small result (counts, status) for the pipeline to branch on.
- Pipeline + Copy job. Use a Copy job for ingestion with built-in incremental state, and run it from a pipeline when downstream steps depend on it. You keep the Copy job’s simplicity and gain orchestration.
- Pipeline + Dataflow Gen2. Run a dataflow as one step in a larger process, for example after reference files land and before a notebook that uses the shaped output.
- Pipeline + Notebook + validation. Load, transform, then a validation step (in the notebook or a separate one) that fails the run on bad data. The pipeline decides whether to continue, quarantine or alert. Validation that cannot stop the pipeline is just a report.
- Dataflow Gen2 feeding a Lakehouse or Warehouse. A BI-owned dataflow writes a curated table that notebooks or semantic models consume. Agree on the table contract, meaning columns, types and refresh timing, so the two teams do not break each other.
- Copy job on its own schedule. Not everything needs a pipeline. A standalone Copy job that lands data hourly, with a separate pipeline that processes whatever has landed, is a perfectly good design when the steps are loosely coupled.
Pipelines vs notebooks
This is the comparison teams get wrong most often, because both can technically do the other’s job.
Pipelines are the control plane; notebooks are the processing logic. A pipeline knows that a transformation ran, how long it took and whether it succeeded. A notebook knows what the transformation does. Keep each kind of knowledge in its own place.
When notebook orchestration becomes brittle:
- A “master” notebook calls other notebooks in sequence with
notebookutils.notebook.runorrunMultiple, wrapped in custom retry and error handling. It works, but scheduling, retries, history and alerting are now code you maintain. Failures surface as a single notebook error, and you cannot rerun just the failed step from the monitoring UI. - Credentials, connections and environment switches move into code, so deploying to another environment means editing notebooks.
- Copying data, calling APIs and waiting for external systems all happen in one Spark session that is mostly idle.
Code-based orchestration has legitimate uses: tightly coupled steps that share a session and its data, or fan-out within one processing job. It should not replace the platform’s orchestrator for a workflow.
When pipeline logic becomes too complex:
- Nested ForEach and If activities, dozens of Set variable steps and long expressions are implementing what is really business logic.
- The same expression appears in many activities, and nobody can test it except by running the pipeline.
- Data-dependent rules (thresholds, mappings, exceptions) live in activity settings instead of in code or configuration tables.
The fix is the same in both directions: move control flow to the pipeline and processing logic to code. A pipeline with ten readable activities calling well-named notebooks is easier to operate than either extreme.
Copy job vs pipeline Copy activity
They are distinct capabilities that share the same copy engine and connectors, and Microsoft documents them separately:
| Copy job | Copy activity (in a pipeline) | |
|---|---|---|
| What it is | A standalone Data Factory item for data movement | An activity inside a pipeline |
| Incremental state | Native: tracks watermark or CDC position, resumes after failure | You build it: Lookup the watermark, parameterize the query, persist the new value |
| Multiple tables | Many tables in one job, configured together | Typically a ForEach over a list, often metadata-driven |
| Write behavior | Append, merge, overwrite, SCD Type 2 (supported destinations) | Append, upsert and overwrite options depending on destination |
| Customization | Guided options plus advanced settings | Fully configurable per activity, with dynamic content everywhere |
| Orchestration | Own schedules; runs in pipelines through the Copy job activity | Part of the pipeline’s control flow |
| Best fit | Ingestion with standard incremental or CDC patterns and little surrounding logic | Metadata-driven ingestion frameworks, or copies tightly woven into custom control flow |
Microsoft’s own guidance is to start ingestion with a Copy job and use Copy activities when you need pipeline-level customization, such as data-object parameterization and metadata-driven ingestion. Inside a pipeline, either works. Pick the Copy job when its built-in incremental handling fits, and the Copy activity when you need control it does not expose.
Dataflow Gen2 vs notebook
Both transform data; the decision is mostly about people and maintainability:
- Team skill set. Power Query for BI developers and analysts; Python and Spark for data engineers. The best tool is often the one the owning team can maintain on a bad day.
- Maintainability. Small and medium dataflows are easy to read in the editor. Very large ones (many queries, deep step chains) become hard to review. Notebooks scale better as logic grows, if they are written with functions and structure rather than as long scripts.
- Version control. Both support Git integration. Notebook code diffs read naturally in a pull request; dataflow definitions are stored in Git, but reviewing Power Query changes in a diff is harder than reviewing Python.
- Transformation complexity. Shaping, cleaning and lookups suit Power Query. Complex rules, large joins, incremental
MERGEand reusable libraries suit notebooks. - Performance. Both can process substantial volumes, and both consume capacity. Neither is universally faster; it depends on the source, the transformations, query folding and data size. Measure your own workload rather than trusting a rule of thumb, including this article’s.
- Transparency and debugging. Dataflows show step-by-step previews while authoring and refresh history per query. Notebooks show cell output and the Spark UI, and record whatever logging you add, which is more work but can be made far more detailed.
Avoid splitting one transformation across both. If a rule exists in a dataflow and again in a notebook, they will drift apart.
CI/CD and source control
All four tools are supported by Fabric Git integration and deployment pipelines as of October 2026. The practical experience differs:
- Pipelines. The pipeline definition is stored in Git. Connections and environment-specific values need handling between environments, through parameters, variable libraries or deployment-time replacement.
- Notebooks. Code is stored in Git and reviews well in pull requests. The default Lakehouse binding and environment-specific parameters must change per environment; deployment rules or parameterization handle that. Git integration in Microsoft Fabric covers the development workflow.
- Dataflow Gen2. New dataflows are created with CI/CD and Git support by default, and deployment pipelines can move them between workspaces. One operational caveat: after syncing from Git or deploying, the updated dataflow must be published before refreshes use the change, either by saving it in the editor or through the publish API. Put that step in your deployment process.
- Copy job. Supported in Git and deployment pipelines, and variable libraries can parameterize its connections so the same job deploys to several environments.
Whatever you deploy with, test each item type through your full path once, from DEV to PROD, before committing to it. Microsoft Fabric CI/CD with fabric-cicd covers automated deployment, including checking which item types your tooling supports.
Capacity and compute considerations
Tool choice affects how work consumes capacity, without any tool being “cheap” or “expensive” in general:
- Notebooks consume Spark capacity for the whole session, including startup and idle time. Many small notebook activities each starting a session can cost more in overhead than in useful work; session sharing helps where it applies.
- Pipelines charge for activity runs and data movement. Orchestration overhead is usually small next to the work it triggers, but a pipeline that polls, loops or retries heavily adds up.
- Dataflow Gen2 consumes capacity for its evaluation, and can use staging items as well.
- Copy jobs and Copy activities are charged for data movement; incremental copies keep that proportional to the change.
Almost all of these are background operations, smoothed over 24 hours, so their effect is cumulative rather than spiky. Microsoft Fabric Capacity Explained covers how to see which tool is actually consuming your capacity. Measure it rather than assuming.
Debugging and operations
| Pipeline | Notebook | Dataflow Gen2 | Copy job | |
|---|---|---|---|---|
| Visibility | Run history per activity, with inputs, outputs and durations | Cell outputs, Spark UI, session details | Refresh history, per query and table | Run history per table, with rows and durations |
| Logs | Activity output and errors | Whatever you log, plus Spark logs | Refresh errors per query | Per-table run details |
| Retry behavior | Per-activity retries with count and interval | Your code, or the calling pipeline’s retry | Rerun the refresh, or the calling pipeline’s retry | Resumes from the last successful state |
| Failure isolation | One failed activity, visible in the graph | One failed notebook run; finer isolation is up to your code | One failed refresh, with the failing query | One failed table among many |
| Parameter inspection | Inputs shown per activity run | Parameter cell values in the run snapshot | Parameters per refresh | Configuration per job |
Two practices make every combination easier to operate. Correlate runs: a pipeline run ID passed into the notebooks it calls ties the evidence together. And keep an execution log that outlives the platform’s run history. The monitoring guide for pipelines and notebooks and Designing an Execution Log for Data Pipelines cover both.
Anti-patterns
- Notebooks that only copy files. A Spark session spent moving bytes that a Copy job or Copy activity moves as configuration, with connection handling and run history built in.
- All orchestration in notebook code. Master notebooks with custom retry loops and schedules duplicate the platform’s orchestrator, badly.
- Row-level business logic in a pipeline. Expressions and nested activities implementing rules that belong in code or SQL.
- Dataflow Gen2 for engineering-grade transformations. Complex incremental logic, large reusable rule sets and extensive testing are code problems.
- Dozens of tiny notebook activities without discipline. Each starts or shares a session, each needs monitoring, and nobody can see the overall flow. Group related steps and parameterize.
- The same logic in several tools. A business rule in a dataflow and again in a notebook guarantees they will drift. Pick one owner.
- A pipeline for every copy. Building a canvas, a watermark table and a ForEach where a Copy job would have done the job with built-in incremental state.
A simple decision tree
- What is the main job?
- orchestrationPipeline
- straightforward copyCopy job
- low-code shapingDataflow Gen2
- complex engineeringNotebook
- More than one?A pipeline orchestrates the others
Production checklist
- Primary responsibility identified for each step: orchestration, movement, low-code shaping or engineering.
- Orchestration separated from processing: pipelines decide, notebooks and dataflows transform, copies move.
- Team skill set considered: the owning team can maintain the tool it was given.
- Git and CI/CD checked for every item type in the path, including the Dataflow Gen2 publish step after deployment.
- Retry behavior defined: where retries happen (activity, Copy job state, code), and that reruns are safe.
- Monitoring defined: run correlation across pipeline, notebook and copy runs; an execution log for history.
- Capacity impact considered: session overhead, polling loops and refresh frequency reviewed.
- Parameters standardized: the same names and formats for dates, windows and environments across tools.
- Duplicate logic avoided: each business rule has exactly one home.
What you learned
- Pipelines orchestrate, notebooks process with code, Dataflow Gen2 transforms with Power Query, and Copy jobs move data. Most solutions combine them.
- A Copy job tracks incremental state for you; a pipeline Copy activity gives you full control and metadata-driven patterns at the cost of building that state yourself.
- Notebook orchestration and pipeline business logic are the two most common ways the boundary breaks. Move control flow to pipelines and logic to code.
- Dataflow Gen2 vs notebook is mostly a question of who owns the logic and how complex it will become.
- All four support Git integration and deployment pipelines, with different practical caveats. Test the full path for each item type.
- Tool choice changes how capacity is consumed. Measure it with the Capacity Metrics app instead of assuming.
Ce contenu vous a-t-il été utile ?
Que pourrions-nous améliorer ?
Merci — cela m’aide à améliorer Javi Builds.
Continuer à apprendre
- Aide-mémoireContenu en anglaisMicrosoft Fabric Workspace Settings Cheat SheetThe Microsoft Fabric workspace settings that change how a data engineering workspace behaves: Spark pools, environments and jobs, domain assignment, OneLake shortcut caching and diagnostics, and data workflow (Apache Airflow job) runtime pools, with what to verify for each.Ouvrir l’aide-mémoire
- Feuille de routeContenu en anglaisFabric in ProductionA practical decision guide for Fabric data integration: when to use a pipeline, a notebook, Dataflow Gen2 or a Copy job, how they combine, and what each means for CI/CD, debugging and capacity.Voir la feuille de route
Que devrais-je créer ensuite ?
Je développe Javi Builds autour des problèmes concrets rencontrés par les data engineers. Dites-moi quel sujet vous aimeriez voir ensuite.
Restez au courant
Recevez uniquement les nouvelles de Javi Builds qui vous intéressent.
Consultez votre boîte de réception
Je vous ai envoyé un lien de confirmation. Votre abonnement sera actif après confirmation.