Browse cheat sheets · Microsoft Fabric
Debug Fabric Pipelines
Troubleshoot Microsoft Fabric pipeline failures by isolating the failed activity, separating transient from deterministic errors, validating parameters and identities, and using retries, logging, and controlled reruns safely.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Start with the pipeline run, then drill into the activity
- 2. Find the first real failure, not every downstream failure
- 3. Record activity inputs before changing anything
- 4. Parameterize notebooks deliberately
- 5. Validate date parameters before using them
- 6. Validate source existence before expensive work
- 7. Add explicit notebook execution context
- 8. Add a pipeline run ID or batch ID to writes
- 9. Make notebook errors explicit
- 10. Distinguish transient errors from deterministic errors
- 11. Configure activity retries selectively
- 12. Use conditional retries where supported
- 13. Do not retry authentication failures automatically
- 14. Confirm which identity runs the notebook
- 15. Compare interactive notebook run vs pipeline run
- 16. Validate the default Lakehouse
- 17. Debug Notebook activity startup separately from notebook code
- 18. Check Spark stage/task failures for notebook errors
- 19. Watch for Delta concurrency conflicts
- 20. Check whether the failed run partially wrote data
- 21. Design activities to be idempotent
- 22. Check for duplicate batches before rerunning
- 23. Use a control table for operational state
- 24. Avoid concurrent blind appends to the same execution log without design
- 25. Fail a pipeline when business validation fails
- 26. Do not hard-code environment-specific paths
- 27. Debug Copy activity failures by layer
- 28. Validate source and sink row counts when available
- 29. Debug stored procedure activity failures separately
- 30. Set activity timeout deliberately
- 31. Differentiate timeout from performance regression
- 32. Use dependency conditions intentionally
- 33. Preserve the original error when adding failure handling
- 34. Pass only necessary error context to alerts
- 35. Use Secure Input / Secure Output for sensitive activities
- 36. Use workspace monitoring for recurring failure analysis
- 37. Query failed pipeline activity runs
- 38. Summarize failures by activity
- 39. Track duration regressions
- 40. Rerun the smallest safe scope
- What the result means
- What to look for
- Failure happens immediately
- Failure happens after long execution
- Only pipeline execution fails
- Failure disappears after retry
- Rerun succeeds but data duplicates
- Several downstream activities fail or skip
- Production notes
- Most activity retries should target transient faults
- Notebook activity execution context matters
- Pipeline runs are not one distributed transaction
- Logging should be durable
- Workspace Monitoring can centralize pipeline history
- Alerts should be actionable
- Retries consume time and capacity
- Common mistakes
- 1. Rerunning the whole pipeline immediately
- 2. Retrying every failure
- 3. Hiding notebook exceptions
- 4. Debugging skipped activities
- 5. Assuming interactive identity equals pipeline identity
- 6. Increasing timeout instead of investigating regression
- 7. Blind append on rerun
- 8. Adding retries before write safety
- 9. Logging secrets for debugging
- 10. Treating one successful rerun as proof the issue is solved
- Next step
- Related deep dives
Problem
A Microsoft Fabric pipeline fails, stalls, retries repeatedly, or produces incorrect output.
The failure can come from different layers:
- pipeline orchestration
- Notebook activity
- Copy activity
- stored procedure activity
- Dataflow activity
- credentials or workspace identity
- parameters
- Spark startup or runtime
- source connectivity
- destination write conflicts
- schema drift
- downstream dependency logic
The goal is to identify which layer failed and whether the failure is transient or deterministic before rerunning anything.
A production-safe debugging flow should answer:
- Which activity failed first?
- What did that activity receive as input?
- What identity did it run under?
- Was the error transient?
- Did the failed activity partially write data?
- Is rerunning the activity safe?
- Should the retry happen automatically or require intervention?
Quick answer
Use this sequence:
1. Open the failed pipeline run
2. Identify the first failed activity
3. Read the activity error, not only the pipeline status
4. Capture run ID, activity name, start/end time, and input parameters
5. Classify the error:
- transient
- authentication/permission
- parameter/configuration
- data/schema
- notebook/Spark
- destination/write conflict
6. Check whether the activity partially committed data
7. Fix the root cause
8. Rerun only when the activity is idempotent or safely recoverable
9. Add targeted retry only for transient failures
10. Persist operational metadata so the next incident is easier
Do not start with:
"Run the whole pipeline again."
until you know whether previous activities already wrote data.
Code / commands
1. Start with the pipeline run, then drill into the activity
In Fabric, use the pipeline run history / Output view to identify:
Pipeline run
↓
Failed activity
↓
Activity input
↓
Activity output/error
Capture at minimum:
pipeline name
pipeline run ID
activity name
activity type
start time
end time
duration
status
error code
error message
The pipeline-level status tells you that something failed.
The activity output usually tells you what failed.
2. Find the first real failure, not every downstream failure
A pipeline can show several failed or skipped activities because one upstream dependency failed.
Example:
Copy_Raw_Data Succeeded
Transform_Notebook Failed
Publish_Warehouse Skipped
Send_Success_Email Skipped
The debugging target is:
Transform_Notebook
not the skipped downstream activities.
Always identify the earliest failure in the dependency chain.
3. Record activity inputs before changing anything
For a failed activity, capture the effective values passed into it.
Examples:
source_path
target_table
start_date
end_date
environment
workspace
lakehouse
batch_id
run_id
Parameter bugs often look like data or connectivity bugs.
A notebook may be correct but receive:
2026/13/01
instead of:
2026-10-01
or point to the wrong Lakehouse.
4. Parameterize notebooks deliberately
A Fabric Notebook activity can pass parameters to a notebook.
Inside the notebook, define parameters explicitly in the designated parameter cell.
Example:
source_table = "bronze_sales"
target_table = "silver_sales"
run_date = "2026-10-04"
Then validate them early:
required = {
"source_table": source_table,
"target_table": target_table,
"run_date": run_date,
}
for name, value in required.items():
if value in (None, ""):
raise ValueError(
f"Missing required parameter: {name}"
)
Fail early with a clear message instead of allowing a vague downstream failure.
5. Validate date parameters before using them
from datetime import datetime
try:
parsed_run_date = datetime.strptime(
run_date,
"%Y-%m-%d",
).date()
except ValueError as exc:
raise ValueError(
f"run_date must be YYYY-MM-DD; got {run_date!r}"
) from exc
Do not allow malformed dates to fail later inside Spark filters or file paths.
6. Validate source existence before expensive work
Example with Spark tables:
if not spark.catalog.tableExists(source_table):
raise RuntimeError(
f"Source table does not exist: {source_table}"
)
For file-based workloads, validate the source path or expected partition before starting a large transformation.
Failing fast gives a clearer operational signal.
7. Add explicit notebook execution context
At the beginning of a production notebook:
print(
{
"source_table": source_table,
"target_table": target_table,
"run_date": run_date,
}
)
Do not print:
- passwords
- access tokens
- connection strings containing secrets
- confidential payloads
Operational logs should help debugging without leaking credentials.
8. Add a pipeline run ID or batch ID to writes
If the pipeline passes a run identifier:
from pyspark.sql import functions as F
output = (
output
.withColumn(
"pipeline_run_id",
F.lit(pipeline_run_id),
)
.withColumn(
"ingested_at_utc",
F.current_timestamp(),
)
)
This helps answer:
Which pipeline execution wrote this row?
It also helps detect replayed batches and partial reruns.
9. Make notebook errors explicit
Bad:
try:
do_work()
except Exception:
print("something failed")
This can hide failure from the orchestrator.
Better:
try:
do_work()
except Exception as exc:
print(
f"Notebook failed: {type(exc).__name__}: {exc}"
)
raise
Let the notebook activity fail when the transformation actually failed.
A pipeline that reports success with incomplete data is worse than a visible failure.
10. Distinguish transient errors from deterministic errors
Transient examples:
temporary service unavailability
HTTP 429 throttling
network timeout
temporary endpoint failure
temporary resource contention
Deterministic examples:
wrong column name
invalid SQL
missing table
permission denied
bad parameter
schema mismatch
duplicate business keys
invalid credentials
Retry transient problems.
Fix deterministic problems.
Do not use retries to hide broken logic.
11. Configure activity retries selectively
Most Fabric Data Factory pipeline activities support retries.
Use activity-level retry settings for failures that may recover automatically.
Conceptually:
Activity
→ General
→ Enable retries
→ Retry count
→ Retry interval
For brief transient problems, a fixed delay may be appropriate.
For throttling or unstable external services, increasing delay can reduce repeated pressure.
Do not set large retry counts by default.
12. Use conditional retries where supported
Fabric supports retry conditions for selected activity types.
Typical criteria include:
error code
error message
failure type
Example policy:
Retry:
HTTP 429
temporary system error
connection timeout
Do not retry:
authentication failure
bad schema
invalid SQL
missing source object
Conditional retry capability is feature/activity dependent, so verify support for the activity being used.
13. Do not retry authentication failures automatically
Example:
401
403
invalid credential
permission denied
Repeated retries usually do not fix:
- expired secret
- missing role
- incorrect workspace permission
- wrong identity
Stop retrying and fix access.
14. Confirm which identity runs the notebook
Notebook execution context matters.
Depending on how execution is triggered, the identity can differ.
For pipeline-triggered notebooks, validate the effective pipeline authentication/security context before assuming the notebook runs as the notebook owner.
When a notebook works interactively but fails in a pipeline, check:
identity
workspace access
Lakehouse access
external data permissions
API permissions
before changing notebook code.
15. Compare interactive notebook run vs pipeline run
If:
Notebook succeeds interactively
Pipeline Notebook activity fails
compare:
identity
parameters
default Lakehouse
Environment
Spark configuration
session mode
workspace
source credentials
The code may be identical while the execution context is not.
16. Validate the default Lakehouse
A notebook can fail if it expects tables from one Lakehouse but the activity/session uses another.
At notebook startup, log or validate the expected table namespace.
Prefer explicit table/path references for critical cross-Lakehouse workloads when ambiguity would be dangerous.
Do not assume the interactive notebook attachment exactly matches pipeline execution context.
17. Debug Notebook activity startup separately from notebook code
A Notebook activity can fail before useful Spark work starts.
Separate:
session acquisition
Spark startup
environment initialization
library setup
notebook code execution
If the failure happens before the first expected log line, investigate session/environment/platform setup rather than the transformation.
18. Check Spark stage/task failures for notebook errors
If the Notebook activity failed during Spark work, inspect:
failed stage
failed task
executor error
shuffle error
Delta conflict
out-of-memory condition
data parsing exception
The pipeline error is often only the outer wrapper.
The Spark job details contain the distributed execution cause.
19. Watch for Delta concurrency conflicts
Example scenario:
Notebook A → MERGE target table
Notebook B → MERGE same target table
at the same time.
Symptoms may include concurrent transaction/write conflict errors.
Do not solve this by increasing pipeline retries indefinitely.
Consider:
append to staging
→ serialize/reconcile merges
or isolate writers by partition/business domain where safe.
20. Check whether the failed run partially wrote data
Before rerunning, ask:
Did earlier activities succeed?
Did this activity commit anything?
Did the target contain the batch already?
For Delta writes, a failed Delta transaction does not partially commit that transaction, but earlier successful activities may already have committed data.
The pipeline as a whole is not one global transaction.
21. Design activities to be idempotent
A safe rerun should produce the same logical result.
Examples:
MERGE by business key
replace known partition
append with batch replay protection
write to staging then promote
Unsafe rerun pattern:
blind append
when the same source batch can be processed twice.
22. Check for duplicate batches before rerunning
If you persist batch metadata:
existing = (
spark.table("pipeline_execution_log")
.filter(
F.col("batch_id") == batch_id
)
)
if existing.count() > 0:
raise RuntimeError(
f"Batch already processed: {batch_id}"
)
In a real production design, avoid expensive count() checks when a more efficient keyed lookup or control-table design exists.
The point is to make replay state explicit.
23. Use a control table for operational state
Example logical schema:
pipeline_execution_log
----------------------
pipeline_name
pipeline_run_id
batch_id
activity_name
status
started_at
completed_at
rows_read
rows_written
error_type
error_message
Use this to answer:
Was this batch already processed?
Where did the last run stop?
How many rows were written?
Which activity failed?
Avoid storing secrets or huge stack traces directly in operational tables.
24. Avoid concurrent blind appends to the same execution log without design
An execution log can itself become a concurrency hotspot.
If multiple notebooks write simultaneously, design the logging layer for concurrency.
Possible approaches:
append-only per activity
unique run/activity key
controlled merge/upsert
central logging service/table
Do not let observability become the failure point.
25. Fail a pipeline when business validation fails
Example:
bad_rows = (
output
.filter(
F.col("merchant_id").isNull()
)
)
bad_count = bad_rows.count()
if bad_count > 0:
raise RuntimeError(
f"Validation failed: {bad_count} rows have null merchant_id"
)
A technically successful Spark job can still produce invalid business data.
Operational success should include quality checks where required.
26. Do not hard-code environment-specific paths
Bad:
source = "abfss://prod@..."
inside every notebook.
Prefer parameters or environment configuration:
source_path = source_path_param
This reduces:
- accidental production access
- deployment drift
- code changes between environments
27. Debug Copy activity failures by layer
For Copy activity, classify the error:
Source connection
Source query
Authentication
Network/connectivity
Schema mapping
Serialization
Sink connection
Sink constraint/write
Throughput/throttling
Do not treat every Copy failure as a connectivity issue.
Check whether the source read actually succeeded before investigating the sink.
28. Validate source and sink row counts when available
Record:
rows read
rows written
rows skipped/rejected
If:
rows read = 10,000,000
rows written = 0
the source is probably not the problem.
If:
rows read = 0
focus upstream first.
29. Debug stored procedure activity failures separately
A Stored Procedure activity may fail because of:
connection/authentication
missing procedure
invalid parameter
SQL exception
transaction/deadlock
timeout
business validation
Capture the SQL error number/message where available.
Do not automatically retry logic errors.
For deadlocks or transient connection failures, targeted retries may be reasonable.
30. Set activity timeout deliberately
Fabric activities support timeout configuration.
A timeout should represent:
the longest duration the activity is allowed to consume resources before being treated as abnormal
Do not set an extremely high timeout simply to avoid failures.
A timeout can reveal:
- stuck external calls
- runaway Spark jobs
- blocked SQL
- unexpected volume increases
Use historical duration to choose a defensible operational limit.
31. Differentiate timeout from performance regression
If a notebook usually takes:
20 minutes
and now hits a:
2-hour timeout
do not simply raise the timeout.
Investigate:
input volume
Spark plan
shuffle
skew
capacity
source latency
concurrency
A timeout increase can hide a real regression.
32. Use dependency conditions intentionally
Typical orchestration logic:
On success
On failure
On completion
Design failure paths explicitly.
Example:
Transform
├─ success → Publish
└─ failure → Log Failure
Do not let a diagnostic/notification activity accidentally convert the logical pipeline outcome into success.
33. Preserve the original error when adding failure handling
Failure-handling logic should record or alert on the error, but the original failed business activity should remain visible.
Avoid patterns where:
Activity fails
→ logging succeeds
→ pipeline appears successful
unless that is intentionally the contract.
34. Pass only necessary error context to alerts
Useful alert fields:
pipeline name
run ID
activity
environment
failure type
error code
short error message
monitoring link/context
Avoid sending:
- credentials
- tokens
- full sensitive payloads
- large data samples
35. Use Secure Input / Secure Output for sensitive activities
Fabric activity settings can hide sensitive input/output from run logs.
Use secure logging controls when activity payloads may include confidential values.
This reduces observability detail, so pair it with safe operational identifiers.
36. Use workspace monitoring for recurring failure analysis
When Fabric Workspace Monitoring is enabled, pipeline and activity logs can be analyzed centrally.
Pipeline-level events can be queried from the monitoring eventhouse.
Example KQL pattern:
ItemJobEventLogs
| where ItemKind == "Pipeline"
| summarize Runs = count() by JobStatus
Use activity-level logs for deeper investigation when available.
Central monitoring is more useful than manually opening individual runs when failures recur.
37. Query failed pipeline activity runs
A practical activity-level pattern is:
FabricDataPipelinesActivityRunsLogs
| where ActivityStatus == "Failed"
| project
PipelineName,
PipelineRunId,
ActivityName,
ActivityType,
ActivityStartTime,
ActivityEndTime,
FailureType
| order by ActivityStartTime desc
Column availability can vary with the monitoring schema/version, so inspect the actual table schema in your workspace.
Use this to identify:
- repeat offenders
- failure clusters
- failing activity types
- time-based patterns
38. Summarize failures by activity
FabricDataPipelinesActivityRunsLogs
| where ActivityStatus == "Failed"
| summarize Failures = count()
by PipelineName, ActivityName, ActivityType
| order by Failures desc
This turns operational history into prioritization.
A pipeline that failed once because of a temporary outage is different from an activity that fails every day.
39. Track duration regressions
Example concept:
FabricDataPipelinesActivityRunsLogs
| where ActivityStatus == "Succeeded"
| summarize
AvgDurationMs = avg(DurationMs),
MaxDurationMs = max(DurationMs)
by PipelineName, ActivityName
Use actual schema field names available in your monitoring table.
Do not invent universal performance thresholds.
Compare each activity against its own historical baseline.
40. Rerun the smallest safe scope
After fixing the root cause, prefer:
rerun failed activity / failed branch
when the platform/workflow supports it safely and upstream outputs are valid.
Otherwise rerun the whole pipeline only when:
- earlier activities are idempotent
- duplicates are prevented
- partial state is understood
The safest rerun scope depends on the pipeline design.
What the result means
| Signal | Likely direction |
|---|---|
| Notebook works interactively, fails in pipeline | Identity / parameters / environment / Lakehouse context |
| HTTP 429 / temporary service failure | Transient retry candidate |
| 401 / 403 | Authentication or permission |
| Invalid column / schema mismatch | Deterministic data or code issue |
| Spark stage failure | Notebook distributed-execution problem |
| Delta conflict | Concurrent write / merge problem |
| Timeout | Performance, blocking, dependency, or resource issue |
| Upstream succeeded, downstream skipped | Dependency behavior, not multiple independent failures |
| Rerun creates duplicates | Non-idempotent write design |
| Repeated same activity failure | Structural issue, not random incident |
What to look for
Failure happens immediately
Likely causes:
- invalid parameter
- missing object
- permission
- configuration
- wrong path
- syntax
Check setup before performance.
Failure happens after long execution
Likely causes:
- Spark stage failure
- timeout
- write conflict
- memory pressure
- source/sink instability
- large-volume edge case
Inspect runtime telemetry.
Only pipeline execution fails
Compare pipeline execution context to interactive execution.
Especially:
- identity
- parameters
- default Lakehouse
- workspace
- Environment
- external permissions
Failure disappears after retry
That supports a transient-failure hypothesis.
Still inspect frequency.
If retries are constantly required, the dependency may be unhealthy or overloaded.
Rerun succeeds but data duplicates
The retry/rerun policy is unsafe.
Fix idempotency before increasing retry counts.
Several downstream activities fail or skip
Trace the dependency graph back to the first genuine failure.
Do not debug every downstream symptom independently.
Production notes
Most activity retries should target transient faults
Fabric Data Factory supports activity retries and retry intervals.
Current retry capabilities also include increasing-delay behavior and conditional retry support for selected activity types.
Use those features for recoverable infrastructure/service failures, not deterministic code errors.
Notebook activity execution context matters
Pipeline-triggered notebook execution can use a different security context from an interactive notebook run.
A notebook that works manually can therefore fail from the pipeline because access conditions differ.
Pipeline runs are not one distributed transaction
One activity can succeed and commit before a later activity fails.
Rerun safety must be designed per activity.
Logging should be durable
Notebook print output is useful for immediate troubleshooting.
For operations, persist:
- run IDs
- batch IDs
- row counts
- status
- timestamps
- safe error metadata
so incidents can be compared over time.
Workspace Monitoring can centralize pipeline history
Fabric Workspace Monitoring can expose pipeline-level and activity-level execution logs through a monitoring eventhouse/KQL database.
This is much better for recurring operational analysis than opening individual run pages manually.
Alerts should be actionable
Fabric supports pipeline/activity alerting patterns.
Alert on failures that need intervention, not every transient event that self-recovers within the expected retry policy.
Retries consume time and capacity
A retry is not free.
Excessive retries can:
- increase pipeline latency
- increase capacity consumption
- increase load on an already unhealthy dependency
- duplicate work if writes are not idempotent
Use retry counts deliberately.
Common mistakes
1. Rerunning the whole pipeline immediately
Earlier activities may already have committed data.
Check partial state first.
2. Retrying every failure
Authentication, bad SQL, invalid schema, and incorrect parameters will not improve because you waited 30 seconds.
3. Hiding notebook exceptions
Printing an error without re-raising can make the notebook appear successful.
Fail visibly when the workload failed.
4. Debugging skipped activities
Skipped downstream activities are usually consequences.
Find the earliest actual failure.
5. Assuming interactive identity equals pipeline identity
Execution context can differ.
Always compare identity and permissions.
6. Increasing timeout instead of investigating regression
A much slower activity may indicate volume growth, skew, blocking, or resource pressure.
7. Blind append on rerun
This is a common way to create duplicate data.
Design idempotency.
8. Adding retries before write safety
Automatic retries are dangerous when the activity can partially or repeatedly write non-idempotent data.
9. Logging secrets for debugging
Never expose credentials or tokens in pipeline output.
Use secure input/output controls where appropriate.
10. Treating one successful rerun as proof the issue is solved
A successful rerun suggests transience but does not explain repeated failures.
Trend the failure over time.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.