DevOps & Observability
Designing an Execution Log for Microsoft Fabric Pipelines and Notebooks
Design a production execution log for Microsoft Fabric pipelines and notebooks using run IDs, correlation IDs, status, row counts, watermarks, errors and rerun-safe patterns.
On this page
Fabric’s run history tells you what the platform executed: a pipeline run with an ID, its activities, their durations and errors. It does not tell you what the run meant. Which business date was it loading? Which window of source changes did it read? How many rows did it insert, update and reject? Did it advance the watermark? Was it the original run or a rerun of last night’s failure?
Those questions come up in every production incident, and they are the job of an execution log: a small table your pipelines and notebooks write to, one row per execution, that records the workload’s business context alongside its operational outcome.
An execution log does not replace Fabric’s monitoring. It complements it. The Monitoring hub stays the place for activity-level detail and Spark logs; the execution log is the durable, queryable record that ties those runs to business operations, outlives the platform’s run-history retention, and supports reruns and incremental loads. Monitoring Microsoft Fabric Pipelines and Notebooks in Production shows how monitoring uses it day to day. This article is about designing it.
Fabric behavior referenced here was checked against Microsoft Learn in October 2026.
The identifiers, and what each one is for
Most execution logs go wrong by mixing up identifiers. There are several, and each answers a different question:
| Identifier | Answers | Comes from |
|---|---|---|
Platform run ID (pipeline_run_id) |
Which Fabric pipeline run was this? | @pipeline().RunId, generated by Fabric |
Execution ID (execution_id) |
Which single execution record is this? | Your logging code: the pipeline run ID for pipeline records, a new UUID for everything else |
Parent execution ID (parent_execution_id) |
What called this execution? | Passed down by the caller |
Correlation ID (correlation_id) |
Which business operation does this belong to? | Created once by the top-level orchestrator, passed to every child |
Business date and window (business_date, window_start, window_end) |
What data was this execution responsible for? | Computed by the orchestrator, passed as parameters |
The distinction that matters most: execution_id identifies one record; parent_execution_id describes the hierarchy; correlation_id groups the whole business operation, including reruns. A notebook failure is hard to diagnose without all three, because the notebook alone knows neither who called it nor which day’s data it was processing.
The execution model
Every workload that does meaningful work writes one execution record: the top-level pipeline, each child pipeline or Copy job, each notebook, and each validation step.
- correlation_idBusiness operationsales-daily:2026-10-24
- execution_idParent pipeline= @pipeline().RunId
- Child pipeline or Copy jobparent_execution_id = parent pipeline
- Notebookparent_execution_id = parent pipeline
- Validationparent_execution_id = parent pipeline
In tree form, for one daily load:
correlation_id = sales-daily:2026-10-24
└── pl_daily_sales execution_id = 9f1c… (pipeline run ID)
├── pl_load_orders execution_id = 47ab… parent = 9f1c…
│ └── nb_silver_orders execution_id = 3d02… parent = 47ab…
├── nb_validate_sales execution_id = 8e71… parent = 9f1c…
└── nb_publish_gold execution_id = c5f9… parent = 9f1c…
Platform run IDs and your execution IDs serve different purposes. The run ID lets you jump to the Monitoring hub for activity detail. The execution ID lets you join your own records to each other, including executions that are not pipeline runs at all.
The canonical schema
This is the schema the rest of the Javi Builds Fabric articles use. Field names are snake_case, timestamps are UTC.
Identity
execution_id: unique per execution recordparent_execution_id: the caller’sexecution_id; null for the top-level runcorrelation_id: the business operationrerun_of_execution_id: for a manual rerun, the failed execution it replaces
Workload
workspace,environment: where it ran (dev,test,prod)item_type:pipeline,notebook,copy_job,validationpipeline_name,notebook_name: the item that ranpipeline_run_id: Fabric’s run ID for pipeline recordsspark_activity_id: the notebook session’s Livy job ID, from the runtime contexttrigger_type:scheduled,manual,rerun,event
Timing
start_time,end_time,duration_ms
Status
status:started,succeeded,failed,cancelled,timed_outretry_count: automatic retries inside this execution
Business context
business_date,source_system,target_object
Incremental context
window_start,window_end: the bounded change window, as in Incremental Loadswatermark_advanced: whether this execution moved the watermark
Data movement
rows_read,rows_written,rows_rejectedrows_inserted,rows_updated,rows_deleted: fromMERGEmetricsfiles_discovered,files_processed,files_skipped,files_failed,bytes_processed: file workloads only
Errors
error_code,error_stage,error_message
timed_out earns its place next to failed because a timeout usually has a different cause (capacity, a hung source, a missing data volume limit) and a different fix. Keep the status list that short: everything else belongs in error_code or the event table.
Required vs optional fields
Not every field applies to every execution. A schema where every row is mostly null is a sign of one table trying to be everything; marking fields by workload keeps it honest.
| Field | Required? | Purpose | Example |
|---|---|---|---|
execution_id |
All | Identifies the record | 3d02e1c8-… |
parent_execution_id |
All except top-level | Hierarchy | 9f1c… |
correlation_id |
All | Business operation | sales-daily:2026-10-24 |
item_type, environment |
All | What and where | notebook, prod |
status, start_time |
All | Outcome and timing | succeeded, 2026-10-24 05:00:12 |
end_time, duration_ms |
All finished executions | Duration and baselines | 2026-10-24 05:04:40, 268000 |
pipeline_name, pipeline_run_id, trigger_type |
Pipelines | Link to Fabric run history | pl_daily_sales, 9f1c…, scheduled |
notebook_name, spark_activity_id |
Notebooks | Link to Spark details | nb_silver_orders |
business_date, target_object |
Data-moving executions | What data it was responsible for | 2026-10-24, silver_sales_orders |
rows_read, rows_written |
Data-moving executions | Volume and zero-row detection | 1240, 1217 |
rows_inserted, rows_updated, rows_deleted |
MERGE-based loads |
What actually changed | 37, 1180, 0 |
rows_rejected |
Executions with validation or quarantine | Data quality | 23 |
window_start, window_end, watermark_advanced |
Incremental loads | Rerun safety | 09:50, 11:00, true |
files_*, bytes_processed |
File ingestion | File-level health | 300, 298, 0, 2 |
retry_count, rerun_of_execution_id |
When applicable | Retries and reruns | 1, 7a40… |
error_code, error_stage, error_message |
Failed executions | Diagnosis | Py4JJavaError, merge_silver, … |
Where to store it
The log is written in small, frequent operations: a row at the start of every execution and an update or second row at the end. That write pattern decides where it should live.
- SQL database in Fabric (or Azure SQL Database) is the natural fit for a T-SQL implementation. It is an operational engine built for row-level
INSERTandUPDATE, it supports indexes and stored procedures, pipelines can write to it with the Script activity, and Fabric replicates its data to OneLake so you can analyze the log alongside everything else. - A Lakehouse Delta table fits teams whose workloads are mostly notebooks. Written append-only from Spark, as shown below, it is simple and conflict-free.
- Fabric Warehouse works but is not ideal. Microsoft’s own guidance is to avoid trickle
INSERT,UPDATEandDELETEtransactions in Warehouse, because each one writes new Parquet files and fragments the table. An execution log is exactly that kind of workload. - The SQL analytics endpoint cannot be the target. It is read-only over Lakehouse tables; you can query a Delta log through it, but not write to one.
Pick one store for the whole platform. Two half-logs in different places are worse than one.
SQL implementation (SQL database in Fabric)
CREATE TABLE dbo.execution_log (
log_id bigint IDENTITY(1,1) NOT NULL,
execution_id varchar(64) NOT NULL,
parent_execution_id varchar(64) NULL,
correlation_id varchar(200) NOT NULL,
rerun_of_execution_id varchar(64) NULL,
workspace varchar(128) NULL,
environment varchar(16) NOT NULL,
item_type varchar(20) NOT NULL,
pipeline_name varchar(200) NULL,
notebook_name varchar(200) NULL,
pipeline_run_id varchar(64) NULL,
spark_activity_id varchar(64) NULL,
trigger_type varchar(20) NULL,
status varchar(16) NOT NULL,
retry_count int NOT NULL CONSTRAINT df_execution_log_retry DEFAULT (0),
start_time datetime2(3) NOT NULL,
end_time datetime2(3) NULL,
duration_ms bigint NULL,
business_date date NULL,
source_system varchar(100) NULL,
target_object varchar(200) NULL,
window_start datetime2(3) NULL,
window_end datetime2(3) NULL,
watermark_advanced bit NULL,
rows_read bigint NULL,
rows_written bigint NULL,
rows_inserted bigint NULL,
rows_updated bigint NULL,
rows_deleted bigint NULL,
rows_rejected bigint NULL,
files_discovered int NULL,
files_processed int NULL,
files_skipped int NULL,
files_failed int NULL,
bytes_processed bigint NULL,
error_code varchar(100) NULL,
error_stage varchar(100) NULL,
error_message nvarchar(2000) NULL,
CONSTRAINT pk_execution_log PRIMARY KEY CLUSTERED (log_id),
CONSTRAINT uq_execution_log_execution UNIQUE (execution_id),
CONSTRAINT ck_execution_log_status CHECK (status IN ('started','succeeded','failed','cancelled','timed_out'))
);
CREATE INDEX ix_execution_log_correlation ON dbo.execution_log (correlation_id, start_time);
CREATE INDEX ix_execution_log_status ON dbo.execution_log (status, start_time);
CREATE INDEX ix_execution_log_workload ON dbo.execution_log (pipeline_name, notebook_name, start_time);A pipeline writes its own start and finish records with Script activities (NonQuery) at the beginning and on its success and failure paths, using dynamic content such as @pipeline().RunId, @pipeline().PipelineName and @pipeline().TriggerTime. Wrapping the two statements in stored procedures keeps the pipeline expressions short and the SQL in one reviewed place:
-- Start
INSERT INTO dbo.execution_log (execution_id, parent_execution_id, correlation_id, environment, item_type,
pipeline_name, pipeline_run_id, trigger_type, status, start_time, business_date)
VALUES (@execution_id, @parent_execution_id, @correlation_id, @environment, 'pipeline',
@pipeline_name, @pipeline_run_id, @trigger_type, 'started', SYSUTCDATETIME(), @business_date);
-- Finish: only an open record can be closed
UPDATE dbo.execution_log
SET status = @final_status, end_time = SYSUTCDATETIME(),
duration_ms = DATEDIFF_BIG(millisecond, start_time, SYSUTCDATETIME()),
error_code = @error_code, error_message = @error_message
WHERE execution_id = @execution_id AND status = 'started';The AND status = 'started' guard enforces the status rules below in the database itself: a finished record cannot be reopened or flipped.
Delta implementation (Lakehouse, notebook-first teams)
In a Lakehouse, write the log append-only: one row when the execution starts, one when it ends. A view returns the latest row per execution_id. This avoids running UPDATE or MERGE against a table that many notebooks write at the same time, where Delta’s optimistic concurrency could make two writers conflict. Concurrent blind appends do not conflict with each other.
-- Latest state per execution (Spark SQL view over the append-only table)
CREATE OR REPLACE VIEW execution_log_current AS
SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY execution_id ORDER BY event_time DESC) AS rn
FROM execution_log_events_raw
) WHERE rn = 1;Here the physical table carries an event_time column (when the row was appended) in addition to the canonical fields, and the view presents the canonical shape. Frequent tiny appends also produce many small files, so enable auto compaction on the log table; Delta Table Maintenance explains why.
The start/finish lifecycle in a notebook
The sequence is the same whatever the store:
- Create the execution ID.
- Write a
startedrecord. - Run the workload.
- On success, write
succeededwith the metrics. - On failure, write
failedwith a sanitized error. - Re-raise the exception.
import uuid
from datetime import datetime, timezone
# Parameters passed by the pipeline (see "Capturing Fabric run IDs")
# correlation_id, parent_execution_id, business_date, window_start, window_end, environment
ctx = notebookutils.runtime.context
execution_id = str(uuid.uuid4())
start_time = datetime.now(timezone.utc)
def log(status, **fields):
now = datetime.now(timezone.utc)
row = {
"execution_id": execution_id,
"parent_execution_id": parent_execution_id,
"correlation_id": correlation_id,
"environment": environment,
"item_type": "notebook",
"notebook_name": ctx["currentNotebookName"],
"spark_activity_id": ctx["activityId"],
"status": status,
"start_time": start_time,
"end_time": None if status == "started" else now,
"duration_ms": None if status == "started" else int((now - start_time).total_seconds() * 1000),
"business_date": business_date,
"window_start": window_start,
"window_end": window_end,
"event_time": now,
**fields,
}
(spark.createDataFrame([row])
.write.format("delta").mode("append").option("mergeSchema", "true")
.saveAsTable("execution_log_events_raw"))
log("started")
try:
metrics = run_silver_merge(window_start, window_end) # returns rows_read, rows_inserted, ...
log("succeeded", target_object="silver_sales_orders", **metrics)
except Exception as e:
log("failed", error_code=type(e).__name__, error_stage="merge_silver", error_message=str(e)[:2000])
raiseThe final raise is the most important line. A notebook that logs the failure and then exits normally turns a red activity into a green one, so the pipeline’s failure path, retries and alerts never fire. Logging is evidence, not error handling.
Capturing Fabric run IDs
Fabric provides some identifiers automatically; the rest you pass explicitly.
In pipelines, these system variables are available in dynamic content:
@pipeline().RunId: the current pipeline run. Use it as the pipeline record’sexecution_idandpipeline_run_id.@pipeline().PipelineName,@pipeline().TriggerTime,@pipeline().TriggerName.@pipeline()?.TriggeredByPipelineRunIdandTriggeredByPipelineName: the calling pipeline’s run, when this pipeline was started by an Invoke pipeline activity. Null otherwise; note the?.
In notebooks, notebookutils.runtime.context reports the notebook name, workspace, default Lakehouse, whether the run is for a pipeline (isForPipeline), and the session’s activityId. Its currentRunId, parentRunId and rootRunId are only populated for notebook reference runs (notebooks started with notebookutils.notebook.run or runMultiple). They do not contain the calling pipeline’s run ID.
So the reliable pattern is explicit: the pipeline passes its identity into each notebook as Base parameters:
| Notebook parameter | Value |
|---|---|
parent_execution_id |
@pipeline().RunId |
correlation_id |
@pipeline().parameters.correlation_id (see next section) |
business_date, window_start, window_end |
computed by the pipeline |
environment |
from a pipeline parameter or variable library |
Notebook parameters covers declaring and validating them, and Invoke Pipeline covers passing values from a parent pipeline to a child.
For notebooks orchestrating other notebooks through reference runs, the runtime context’s run IDs are useful for the internal tree; still pass the business identifiers explicitly.
Generating the correlation ID
The top-level orchestrator creates one correlation ID per business operation and passes it to every child. Children never generate their own.
A deterministic correlation ID built from the workload and its business key works best:
@concat('sales-daily:', formatDateTime(pipeline().parameters.business_date, 'yyyy-MM-dd'))
Because it is derived from what the operation is, not from a particular run, a rerun of the same business date gets the same correlation ID. One query then shows the original failed attempt, the rerun and everything they called. A correlation ID derived from @pipeline().RunId would split every rerun into a separate, unrelated operation.
Child pipelines receive it as a parameter, and pass it on to their own children. If a child is ever run on its own, it should require the parameter rather than invent a value.
Incremental loads
For incremental workloads, the log is part of the correctness story. The Incremental Loads article keeps the watermark in its own control table; the execution log records what each execution did with it, so you can answer:
- What was the previous watermark, and what upper bound did this run select? (
window_start,window_end) - How many rows were read, inserted, updated and deleted?
- Did validation pass?
- Was the watermark advanced? (
watermark_advanced)
Watermark safety
The safe sequence fails in a recoverable way:
Silver and Gold writes commit
↓
the run fails before persisting the watermark
↓
execution record: status = failed, watermark_advanced = false
↓
the next run reads the old watermark and replays the window safely
The dangerous sequence fails silently:
watermark advances first
↓
the downstream write fails
↓
the next run starts after the new watermark: that window is skipped for good
The execution log makes both visible. A failed record with watermark_advanced = false is expected and recoverable; a watermark that moved without a matching succeeded record is an incident. The monitoring article includes the query that detects it.
Reruns
Never overwrite a failed record. The failed execution is evidence: when it failed, where, with which inputs. A rerun gets its own record, with its own execution_id, status and timing:
correlation_id: the same, so the attempts appear together.rerun_of_execution_id: points to the failed execution it replaces.trigger_type = 'rerun'.
rerun_of_execution_id is worth its column: it lets you count how often a workload needs manual intervention, and it answers “did anyone rerun this?” without reading tickets. An attempt_number adds little once you have that link and the correlation ID.
Retry vs rerun
The two words are often used interchangeably. In the log, they are different things:
- Retry: one execution automatically repeats a failed activity. A pipeline activity with retries enabled, or retry logic inside a notebook, makes several attempts within the same execution. Record it as
retry_counton that execution’s record. - Rerun: a new execution of the workload after a previous one finished or failed. A scheduled catch-up, an operator pressing Rerun in the Monitoring hub, a backfill. It gets a new record, linked by
rerun_of_execution_idandcorrelation_id.
A run that succeeded after two retries is healthy but worth trending; a workload that needs a rerun every week is not healthy, even if every rerun succeeds.
Idempotency: the log must not block legitimate reruns
A tempting shortcut is to make workloads skip work that already has a log record: “if a succeeded row exists for this date, do nothing.” It looks like protection against duplicate processing. In practice it turns the log into hidden state that blocks the reruns you need after a bad load, a code fix or a backfill.
Keep the two concerns separate. Idempotency comes from how the workload writes data: deterministic keys, guarded MERGE, overwriting a scoped slice. The log records what happened; it does not decide whether work may happen. If a workload genuinely must run at most once per key (for example, sending an external notification), make that rule explicit, with its own key and a documented override, rather than a side effect of the log.
Status transitions
| From | To | Allowed? |
|---|---|---|
| (new) | started |
Yes, when the execution begins |
started |
succeeded, failed, cancelled, timed_out |
Yes, once, when it ends |
failed, cancelled, timed_out |
succeeded |
No: write a new execution record instead |
succeeded |
anything | No |
A record that stays started long after it should have finished means the process died without writing its final status: a killed session, a capacity rejection, a crash in the logging itself. Detect those with a query (below) and close them as failed with error_code = 'abandoned' by a scheduled check, rather than leaving them open forever.
Error handling
When an execution fails, log enough to start diagnosing without opening five screens:
error_code: the exception type or a stable code.error_stage: which step failed (extract,merge_silver,validate).error_message: a concise, truncated message.- The identifiers and the time, which the record already has.
Keep full stack traces in the Spark logs or the event table, not in the narrow execution table. Link to them by spark_activity_id.
Never log secrets. No passwords, keys, tokens, connection strings or SAS URLs, and no full payloads that might contain personal data. Error messages from drivers sometimes include connection details, so sanitize before writing. In pipelines, mark activities whose inputs or outputs contain sensitive values as secure input and secure output so they are not shown in run history either.
Row counts: be specific
A single row_count column becomes ambiguous the first time someone asks “row count of what?”. Name counts by what they measure:
rows_read: rows read from the source or window.rows_inserted,rows_updated,rows_deleted: what the write changed. Delta records these asMERGEoperation metrics in the table history, so you can read them instead of running extra counts.rows_rejected: rows sent to quarantine or dropped by validation.rows_written: total rows written, for append or overwrite workloads where insert/update does not apply.
A final target row count is occasionally useful for reconciliation, but it costs a full count, so collect it only where it earns its place.
File ingestion fields
For file-based workloads, add the file counters: files_discovered, files_processed, files_skipped, files_failed and bytes_processed. Leave them null for database workloads; they are optional by design. Per-file detail (which file failed, its checksum, the reason) belongs in the processed-file table or the event table, not as columns here.
One row per execution, not per record
The execution log records executions, not data. One row per pipeline, notebook or validation run. Logging a row per source record, per file line or per loop iteration turns a small operational table into a large, slow one that nobody can query during an incident.
When you do need finer detail, such as milestones within a long notebook, per-file outcomes or intermediate metrics, add an optional second table.
The optional event table
CREATE TABLE dbo.execution_event (
event_id bigint IDENTITY(1,1) NOT NULL PRIMARY KEY,
execution_id varchar(64) NOT NULL,
event_time datetime2(3) NOT NULL,
event_type varchar(30) NOT NULL, -- milestone, metric, warning, error
stage varchar(100) NULL,
message nvarchar(2000) NULL,
metric_name varchar(100) NULL,
metric_value decimal(38,6) NULL
);
CREATE INDEX ix_execution_event_execution ON dbo.execution_event (execution_id, event_time);It is useful when a single execution has distinct phases worth timing (extract, transform, write), when file-level outcomes matter, or when you want warnings that do not fail the run. Keep it bounded: a handful of events per execution, not a stream. If you are writing hundreds of events per run, you want a tracing or logging tool, not a table.
- correlation_idBusiness operationOne per business date or batch
- execution_idExecution table
- parent_execution_id
- status and timing
- rows and watermark
- error summary
- execution_idEvent table (optional)
- milestones
- metrics
- warnings
- error detail
Operational queries
These run on the SQL database implementation; the Delta version is the same against execution_log_current.
-- 1. Everything that happened for one business operation, including reruns
SELECT item_type, pipeline_name, notebook_name, status, start_time, duration_ms,
rows_read, rows_inserted, rows_updated, retry_count, rerun_of_execution_id, error_message
FROM dbo.execution_log
WHERE correlation_id = 'sales-daily:2026-10-24'
ORDER BY start_time;
-- 2. Failed executions in the last day
SELECT start_time, item_type, COALESCE(notebook_name, pipeline_name) AS workload,
error_code, error_stage, error_message, correlation_id
FROM dbo.execution_log
WHERE status IN ('failed', 'timed_out') AND start_time >= DATEADD(day, -1, SYSUTCDATETIME())
ORDER BY start_time DESC;
-- 3. Executions still open long after they should have finished (abandoned)
SELECT execution_id, COALESCE(notebook_name, pipeline_name) AS workload, start_time
FROM dbo.execution_log
WHERE status = 'started' AND start_time < DATEADD(hour, -6, SYSUTCDATETIME()); -- example cutoff
-- 4. Duration against recent history, per workload
WITH recent AS (
SELECT COALESCE(notebook_name, pipeline_name) AS workload, duration_ms,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY duration_ms)
OVER (PARTITION BY COALESCE(notebook_name, pipeline_name)) AS p50_ms
FROM dbo.execution_log
WHERE status = 'succeeded' AND start_time >= DATEADD(day, -28, SYSUTCDATETIME())
)
SELECT workload, MAX(p50_ms) / 60000.0 AS median_minutes, MAX(duration_ms) / 60000.0 AS worst_minutes
FROM recent GROUP BY workload ORDER BY worst_minutes DESC;
-- 5. Failures by workload over 30 days
SELECT COALESCE(notebook_name, pipeline_name) AS workload, COUNT(*) AS failures,
SUM(CASE WHEN rerun_of_execution_id IS NOT NULL THEN 1 ELSE 0 END) AS reruns
FROM dbo.execution_log
WHERE start_time >= DATEADD(day, -30, SYSUTCDATETIME())
AND (status IN ('failed', 'timed_out') OR rerun_of_execution_id IS NOT NULL)
GROUP BY COALESCE(notebook_name, pipeline_name)
ORDER BY failures DESC;
-- 6. Suspicious incremental runs: watermark advanced by a run that did not succeed
SELECT execution_id, notebook_name, status, window_start, window_end, start_time
FROM dbo.execution_log
WHERE watermark_advanced = 1 AND status <> 'succeeded';The six-hour cutoff in query 3 is only an example: base it on each workload’s longest normal duration. The monitoring article adds the complementary check against the watermark table itself.
Retention
There is no universal retention period. It depends on:
- Troubleshooting needs: how far back you compare when a workload slows down.
- Rerun and backfill windows: you need the log at least as long as you might rerun.
- Audit and regulatory requirements: some data flows must prove what ran, when.
- Volume: a few hundred executions a day is tiny; many thousands with detailed events is not.
A common pattern is to keep the execution table for much longer than the event table, because one row per execution stays small for years, while event detail is useful mostly for recent incidents. Retain aggregated summaries (daily duration percentiles, failure counts) longer still if you need long trends.
Physical design
The query patterns are predictable: look up by execution_id, group by correlation_id, filter by status and time, trend by workload and time. How you support them depends on the store:
- SQL database in Fabric / Azure SQL: standard SQL Server design applies. A narrow clustered key on an increasing
log_idkeeps inserts cheap; a unique constraint onexecution_id; nonclustered indexes on(correlation_id, start_time),(status, start_time)and the workload columns, as in the schema above. Avoid clustering on a random UUID. - Lakehouse Delta: there are no indexes. The table stays small enough that this rarely matters; keep files compacted, and if it ever grows large, cluster or partition by date rather than by ID. Delta Tables for SQL Server Engineers explains why index habits do not carry over.
- Fabric Warehouse: no user-created indexes; the engine manages statistics and compaction, and data clustering is the available layout control. The trickle-write concern above matters more than indexing.
How it fits with monitoring
The two Javi Builds articles divide the work deliberately:
- The execution log is durable workload context: what each execution was asked to do, what it did, and how it relates to the rest of the operation.
- Monitoring is the operational use of that context together with Fabric’s run history, validation results and capacity history: alerts, baselines, investigations.
Other related pieces:
- Pipelines vs Notebooks vs Dataflow Gen2 vs Copy Job explains which tool owns each step, which decides who writes which records.
- Microsoft Fabric Capacity Explained is the companion when an execution is slow for reasons outside its own code, and How I Diagnosed a Microsoft Fabric Capacity Saturation Problem shows the log being used to explain a capacity spike.
- Microsoft Fabric CI/CD with fabric-cicd covers deploying the logging code itself; keep the log store and its parameters environment-specific.
- Using session tags in Fabric pipelines is relevant when you log
spark_activity_id: tagged notebooks can share a session, so several executions may report the same session while keeping distinct execution IDs. - The Fabric monitoring cheat sheet and pipeline debugging cheat sheet are quick references for using the log during an incident, and the data platform monitoring checklist covers the wider review.
Production checklist
- One store chosen for the whole platform, suited to small frequent writes.
- Every execution writes a record: pipelines, child pipelines, Copy jobs, notebooks, validation.
- Identifiers separated:
execution_idper record,parent_execution_idfor the tree, onecorrelation_idper business operation. - Correlation ID created once by the top-level orchestrator and passed down; deterministic, so reruns share it.
- Run IDs passed explicitly from pipelines to notebooks as parameters.
- Start and finish records written; exceptions re-raised after logging.
- Status transitions enforced: finished records never change; reruns get new records.
- Retries and reruns distinguished:
retry_countvsrerun_of_execution_id. - Specific row counts and, for incremental loads, window bounds and
watermark_advanced. - Abandoned executions detected and closed.
- No secrets in errors or parameters; sensitive activity I/O marked secure.
- One row per execution, with an optional, bounded event table for detail.
- Retention decided per table, from troubleshooting, rerun and audit needs.
With the log in place, the next step in this path puts it to work: Debugging Microsoft Fabric Pipelines and Notebooks in Production uses these identifiers to trace a failed, slow or wrong run back to its cause.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Continue learning
- Cheat sheetRead and Write Delta TablesRead and write Delta tables with PySpark using safe append, overwrite, merge, schema-evolution, partitioning, and maintenance patterns for production data pipelines.Open cheat sheet
- ArticleMonitoring a Data Platform: What Should You Actually Measure?A practical monitoring model connecting clients, applications, APIs, report engines, data platforms, compute and storage.Read article
What should I build next?
I'm building Javi Builds around practical problems data engineers actually face. Tell me what you'd like to see next.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.