DevOps y observabilidad
Data Quality Checks for Data Pipelines: Rules, Severity, Quarantine and Evidence
How to design data-quality checks that actually protect a pipeline: row-level and dataset-level rules, severity and actions, quarantine with reason codes, thresholds from baselines, recorded evidence, tests for the checks themselves, and Microsoft Fabric's built-in options.
En este artículo
A data-quality check that runs after the data is already in the report is a monitor, not a control. A check that fails on every load because its threshold was a guess gets muted within a week. A check that drops bad rows without recording them hides the problem it found. This guide answers one question:
How do I stop bad data from reaching consumers, and prove which data passed?
Three principles shape everything below:
- A check is more than a query. It is a rule, a scope, a threshold, a severity, an action and an owner. The query is the easy part.
- Check before you publish. Land the batch somewhere consumers can’t see it, check it, then publish, quarantine or hold.
- Every run leaves evidence. Which rules ran, what they observed, what passed, and what was decided, recorded with the run.
The Data Quality Checks cheat sheet is the catalogue of individual checks (schema, nulls, keys, ranges, dates, referential integrity, freshness, reconciliation). This guide is the layer around them: how checks are organised, what they decide, and how you know they work. Testing a Production Data Platform explains where data-quality checks sit among the other kinds of tests.
Where checks run: write, audit, publish
The pattern that makes checks a control instead of a report is simple: the batch is checked before consumers can see it.
- LandRaw batch, as received
- StageTyped and transformed, not yet visible
- AuditRow rules and dataset checks
- Quarantine bad rows
- Gate the batch
- Publish or holdConsumers see only audited data
- Record evidenceResults, counts, decision, run ID
“Staging” can be a staging table, a separate schema, a Delta table that the publish step MERGEs from, or a DataFrame that hasn’t been written yet. What matters is the order: audit before publish. A check that runs after fact_orders has already been overwritten can only tell you that consumers have been reading bad data.
There are three places the same rules run, and they answer different questions:
| Where | Data | What it proves |
|---|---|---|
| Unit tests in CI | Hand-written rows | The rule logic is correct (the check catches what it should, and only that) |
| Test environment | Representative data | The thresholds are calibrated: clean data passes, known-bad data fails |
| Every production load | The real batch | This batch is fit to publish |
Row rules and dataset checks
The most useful distinction in data-quality design, and the one most frameworks blur:
- Row ruleEach row, on its own
- amount >= 0customer present, known currency
- QuarantineFailing rowsWith reason codes
- Valid rows continue
- Dataset checkThe batch as a whole
- Unique keysFreshness, volume, reconciliation
- Hold or warnBatch fails the checkNo single row to remove
- Batch decision
| Row rule | Dataset check | |
|---|---|---|
| Examples | Not null, range, allowed values, pattern, valid date | Unique key, freshness, volume, reconciliation, distribution, referential integrity at scale |
| Unit of judgement | One row | The batch, or a table |
| Natural action | Quarantine the row, keep the rest | Publish with a warning, or hold the batch |
| Typical owner | Producer of the data | Producer, with business input on thresholds |
Duplicate keys are the classic trap: they look like a row problem, but which of the two rows is wrong? Quarantining both loses a real order; keeping both doubles it. Deduplication by a business rule belongs in the transformation (PySpark unit testing shows how to test it); a duplicate that survives the transformation is a dataset-level failure.
Anatomy of a rule
Before any code, write down what each rule is:
| Field | Example | Why it matters |
|---|---|---|
| Name | amount_non_negative |
Appears in quarantine reasons, results and alerts; stable over time |
| Scope | silver.orders, column amount |
What is checked |
| Level | Row | Decides the possible actions |
| Condition | amount >= 0 |
True for valid data |
| Threshold | 0 failing rows | What “pass” means |
| Severity | Error | Error quarantines or holds; warn records and alerts |
| Action | Quarantine row | What happens on failure |
| Owner | Orders data product owner | Who decides when the rule fires, and whether it changes |
| Rationale | “Refunds are a separate order type” | Why the rule exists, so it isn’t deleted the first time it’s inconvenient |
Keeping rules as data (a list in code, a YAML file, a table) rather than scattered if statements makes them reviewable in a pull request, reusable across tables, and reportable: the results table can join back to the rule definitions.
Severity and actions
| Severity | Row rule action | Dataset check action |
|---|---|---|
| Error | Quarantine the row with its reasons; publish the rest | Hold the batch; consumers keep the last good version |
| Warn | Keep the row; count and report it | Publish; alert the owner |
| Info | Record only | Record only (useful while calibrating a new rule) |
One more rule turns row-level quarantine into a batch-level decision: if too many rows are quarantined, the problem isn’t the rows, it’s the batch. A source that suddenly sends 30% of rows without a customer ID has changed; publishing the other 70% as if nothing happened produces a report that is complete-looking and wrong. Set a maximum quarantine ratio per table, and hold the batch above it.
A PySpark implementation
A deliberately small layer: row rules that quarantine, dataset checks that gate, one decision. It’s a pattern to adapt, not a framework to install.
Row rules with reason codes
# src/dq/checks.py
"""A small data-quality layer: row rules that quarantine, dataset checks that gate, one decision."""
from dataclasses import dataclass
from datetime import datetime
from pyspark.sql import DataFrame
from pyspark.sql import functions as F
@dataclass(frozen=True)
class RowRule:
"""A rule every row must satisfy. `condition` is a SQL expression that is true for a valid row."""
name: str
condition: str
severity: str = "error" # error: quarantine the row · warn: keep it, count it
@dataclass(frozen=True)
class CheckResult:
check: str
level: str # "row" or "dataset"
severity: str # "error" or "warn"
observed: float
threshold: str
passed: bool
def _fails(rule: RowRule):
# A condition that evaluates to NULL (e.g. amount >= 0 on a null amount) counts as a failure.
return ~F.coalesce(F.expr(rule.condition), F.lit(False))
def apply_row_rules(df: DataFrame, rules: list[RowRule]):
"""Split a batch into valid rows and quarantined rows, each quarantined row with its reasons."""
errors = [r for r in rules if r.severity == "error"]
reasons = F.array_compact(F.array(*[F.when(_fails(r), F.lit(r.name)) for r in errors]))
tagged = df.withColumn("dq_failed_rules", reasons)
valid = tagged.where(F.size("dq_failed_rules") == 0).drop("dq_failed_rules")
quarantined = tagged.where(F.size("dq_failed_rules") > 0)
counts = df.agg(*[F.sum(_fails(r).cast("int")).alias(r.name) for r in rules]).first()
results = [
CheckResult(r.name, "row", r.severity, float(counts[r.name] or 0), "0 rows", (counts[r.name] or 0) == 0)
for r in rules
]
return valid, quarantined, results
The rules for the orders example:
RULES = [
RowRule("customer_id_present", "customer_id IS NOT NULL"),
RowRule("amount_non_negative", "amount >= 0"),
RowRule("currency_known", "currency IN ('CAD', 'USD', 'EUR')"),
RowRule("amount_below_review_limit", "amount < 10000", severity="warn"),
]
Three design choices in apply_row_rules are worth copying:
- Every failing rule is recorded, not just the first. A row with a negative amount and an unknown currency carries both reasons in
dq_failed_rules, so the person investigating fixes both. - The split is total: every row lands in exactly one output. Valid and quarantined are derived from the same tagged DataFrame, so their counts add up to the input. That invariant is what makes reconciliation possible later.
- NULL is a failure. This is the subtle one.
array_compact is available from Spark 3.4. Warn-level rules are counted in the results but never quarantine.
Dataset checks
def duplicate_keys(df: DataFrame, keys: list[str], severity: str = "error") -> CheckResult:
duplicated = df.groupBy(*keys).count().where(F.col("count") > 1).count()
return CheckResult(f"unique_{'_'.join(keys)}", "dataset", severity, float(duplicated), "0 keys", duplicated == 0)
def freshness(df: DataFrame, column: str, as_of: datetime, max_lag_minutes: int, severity: str = "error") -> CheckResult:
"""`as_of` is the run's reference time, passed in: the check never reads the wall clock."""
newest = df.agg(F.max(column)).first()[0]
lag = float("inf") if newest is None else (as_of - newest).total_seconds() / 60
return CheckResult(f"fresh_{column}", "dataset", severity, lag, f"<= {max_lag_minutes} min", lag <= max_lag_minutes)
def volume_within_history(rows: int, history: list[int], tolerance: float = 0.5, severity: str = "warn") -> CheckResult:
"""Row count within +/- tolerance of the median of comparable past runs (e.g. same weekday)."""
if not history:
return CheckResult("volume", "dataset", severity, float(rows), "no baseline yet", True)
ordered = sorted(history)
middle = len(ordered) // 2
median = ordered[middle] if len(ordered) % 2 else (ordered[middle - 1] + ordered[middle]) / 2
low, high = median * (1 - tolerance), median * (1 + tolerance)
return CheckResult("volume", "dataset", severity, float(rows), f"{low:.0f}-{high:.0f}", low <= rows <= high)
def reconciles(source_rows: int, published_rows: int, quarantined_rows: int, duplicate_rows: int,
severity: str = "error") -> CheckResult:
"""Every source row is accounted for: published, quarantined or removed as a duplicate."""
gap = source_rows - published_rows - quarantined_rows - duplicate_rows
return CheckResult("reconciliation", "dataset", severity, float(gap), "0 rows unaccounted", gap == 0)
Each check returns the same CheckResult shape: what it observed, the threshold, and whether it passed. Three details:
- Freshness takes the run time as an argument. A check that reads the wall clock can’t be tested deterministically and gives different answers when a load is rerun hours later. The run’s reference time comes from the orchestrator.
- An empty batch fails freshness. No newest timestamp means an infinite lag, not “nothing to check”.
- Reconciliation accounts for every row, not just counts.
source = published + quarantined + deduplicatedis the rule. Exactsource = targetequality is wrong as soon as anything is filtered or deduplicated, which the flagship covers in why exact equality is sometimes the wrong rule.
One decision
def decide(results: list[CheckResult], quarantined_rows: int, total_rows: int,
max_quarantine_ratio: float = 0.01) -> str:
"""publish · publish_with_warnings · hold. Too many quarantined rows is a batch problem, not a row problem."""
if any(r.severity == "error" and r.level == "dataset" and not r.passed for r in results):
return "hold"
if total_rows and quarantined_rows / total_rows > max_quarantine_ratio:
return "hold"
if quarantined_rows or any(not r.passed for r in results):
return "publish_with_warnings"
return "publish"
The orchestration around it, in the notebook or job:
valid, quarantined, results = apply_row_rules(staged, RULES)
results.append(duplicate_keys(valid, ["order_id"]))
results.append(freshness(valid, "event_timestamp", as_of=run_started_at, max_lag_minutes=120))
results.append(volume_within_history(staged_rows, history=same_weekday_counts))
decision = decide(results, quarantined_rows=quarantined.count(), total_rows=staged_rows)
# publish: MERGE valid into silver.orders; append quarantined to the quarantine table
# publish_with_warnings: the same, then alert the owner
# hold: write nothing to silver.orders; alert; the run fails visibly
run_started_at, staged_rows and same_weekday_counts come from the run context and the results history; the last section shows where that history lives. That orchestration snippet is illustrative; the functions it calls are the tested ones above.
Thresholds: where the numbers come from
There are no universal data-quality thresholds. A rule’s threshold comes from one of three places, and should say which:
| Kind | Example | Use when |
|---|---|---|
| Zero tolerance | 0 null customer IDs; 0 duplicate order IDs | The rule is a hard invariant of the data model |
| Agreed ratio | At most 0.5% of rows with an unknown postcode | Some failure is normal and accepted; the number comes from the data owner |
| Baseline-relative | Row count within ±50% of the median of the last eight Mondays | “Normal” varies; compare with comparable history |
Baseline thresholds need comparable history. Compare Mondays with Mondays and month-ends with month-ends, use the median (one bad day doesn’t move it), and start new rules as warn until the baseline has enough runs. The tolerance in volume_within_history is a parameter, not a recommendation: calibrate it on your own results history.
A mis-specified rule is worse than no rule. A volume check that fires every public holiday teaches people to ignore it, and then it doesn’t matter that it also fires the day the source silently truncates a file.
Quarantine done properly
A quarantine table is only useful if someone can act on it:
- Keep the original row, unchanged, plus the reasons, the run ID and the time. Don’t “fix” values on the way in.
- Store reasons as rule names, the same names the results table uses.
- Give it an owner and a review cadence. A quarantine table nobody looks at is a slower way of deleting data.
- Decide how rows come back. Usually the source resends a corrected record, which flows through the normal load; occasionally an owner approves a manual release. Either way, the path is designed, not improvised.
- Watch its growth. Quarantined rows per run is itself a metric worth alerting on.
Record the evidence
Every run writes one row per check:
def results_frame(spark, results: list[CheckResult], run_id: str, table: str, checked_at: datetime) -> DataFrame:
"""One row per check per run: the evidence that monitoring, audits and baselines read."""
rows = [(run_id, table, r.check, r.level, r.severity, r.observed, r.threshold, r.passed, checked_at) for r in results]
schema = ("run_id string, table_name string, check_name string, level string, severity string, "
"observed double, threshold string, passed boolean, checked_at timestamp")
return spark.createDataFrame(rows, schema)
Appended to a results table, this one structure serves four purposes:
- Audit: for any published batch, which rules ran and what they found.
- Monitoring: alerts on failed error-level checks, on warnings that persist, on quarantine growth.
- Baselines: the
observedhistory is where volume and distribution thresholds come from. - Correlation:
run_idjoins to the pipeline’s execution log, so a slow or failed run and its data-quality results appear together. Designing an Execution Log for Microsoft Fabric Pipelines and Notebooks covers that log.
The same pattern in T-SQL
In a Warehouse-centred platform, the checks are SQL. These scripts were not executed for this guide. They use only varchar, bigint, float, bit, decimal and datetime2(6), which Fabric Warehouse supports for tables (Fabric Warehouse data types); test them on your target before relying on them.
The results and quarantine tables:
CREATE TABLE dq.check_result (
run_id varchar(64) NOT NULL,
table_name varchar(256) NOT NULL,
check_name varchar(128) NOT NULL,
check_level varchar(16) NOT NULL, -- row | dataset
severity varchar(16) NOT NULL, -- error | warn
observed float NULL,
threshold varchar(64) NULL,
passed bit NOT NULL,
checked_at datetime2(6) NOT NULL
);
CREATE TABLE dq.orders_quarantine (
run_id varchar(64) NOT NULL,
order_id bigint NULL,
customer_id bigint NULL,
amount decimal(18,2) NULL,
currency varchar(3) NULL,
event_timestamp datetime2(6) NULL,
failed_rules varchar(512) NOT NULL,
quarantined_at datetime2(6) NOT NULL
);Row rules with reason codes. Each CASE treats NULL as a failure by testing for the valid case explicitly; CONCAT_WS skips the NULLs, so only failing rule names remain:
DECLARE @run_id varchar(64) = 'run-2026-11-10-0600';
DECLARE @now datetime2(6) = SYSUTCDATETIME();
INSERT INTO dq.orders_quarantine
SELECT @run_id, order_id, customer_id, amount, currency, event_timestamp, failed_rules, @now
FROM (
SELECT s.*,
CONCAT_WS(',',
CASE WHEN s.customer_id IS NOT NULL THEN NULL ELSE 'customer_id_present' END,
CASE WHEN s.amount >= 0 THEN NULL ELSE 'amount_non_negative' END,
CASE WHEN s.currency IN ('CAD', 'USD', 'EUR') THEN NULL ELSE 'currency_known' END
) AS failed_rules
FROM stg.orders AS s
) AS checked
WHERE failed_rules <> '';CASE WHEN s.amount >= 0 THEN NULL ELSE 'amount_non_negative' END is NULL-safe by construction: a NULL amount doesn’t satisfy WHEN, so it falls to ELSE and is reported. Writing it the other way round (CASE WHEN s.amount < 0 THEN 'amount_non_negative' END) silently passes NULL amounts, the same trap as in PySpark.
A dataset check, recorded:
INSERT INTO dq.check_result
SELECT @run_id, 'stg.orders', 'unique_order_id', 'dataset', 'error',
COUNT(*), '0 keys', CASE WHEN COUNT(*) = 0 THEN 1 ELSE 0 END, @now
FROM (
SELECT order_id
FROM stg.orders
GROUP BY order_id
HAVING COUNT(*) > 1
) AS duplicated;In Fabric Warehouse this check isn’t optional: primary-key and unique constraints exist only as NOT ENFORCED, so they document intent but don’t stop duplicates (table constraints). The publish step (an INSERT or MERGE from staging into the published table, excluding quarantined keys) runs only when no error-level dataset check failed for the run.
Test the checks themselves
A data-quality rule is code, and it can be wrong in both directions: it can miss bad data, or flag good data. Both are tested the same way as any transformation, with small DataFrames and pytest (PySpark DataFrame Unit Testing with pytest covers the setup). From this guide’s suite:
def test_rows_failing_error_rules_are_quarantined_with_reasons(spark):
df = batch(spark, [
(1, 10, Decimal("100.00"), "CAD", T),
(2, None, Decimal("50.00"), "CAD", T), # missing customer
(3, 30, Decimal("-5.00"), "XYZ", T), # two failures
(4, 40, Decimal("12000.00"), "USD", T), # warn only: stays valid
])
valid, quarantined, results = apply_row_rules(df, RULES)
assert sorted(r.order_id for r in valid.collect()) == [1, 4]
reasons = {r.order_id: sorted(r.dq_failed_rules) for r in quarantined.collect()}
assert reasons == {2: ["customer_id_present"], 3: ["amount_non_negative", "currency_known"]}
by_name = {r.check: r for r in results}
assert by_name["amount_below_review_limit"].observed == 1
assert by_name["amount_below_review_limit"].severity == "warn"
def test_a_null_condition_counts_as_a_failure_not_a_pass(spark):
# amount >= 0 is NULL for a null amount. Without coalesce, the rule would silently pass the row.
df = batch(spark, [(1, 10, None, "CAD", T)])
valid, quarantined, _ = apply_row_rules(df, RULES)
assert valid.count() == 0
assert quarantined.first().dq_failed_rules == ["amount_non_negative"]
def test_every_row_lands_in_exactly_one_output(spark):
df = batch(spark, [(i, 10 if i % 3 else None, Decimal(i), "CAD", T) for i in range(1, 31)])
valid, quarantined, _ = apply_row_rules(df, RULES)
assert valid.count() + quarantined.count() == df.count()
@pytest.mark.parametrize(("rows", "passed"), [(1000, True), (1400, True), (1600, False), (400, False)])
def test_volume_against_same_weekday_history(rows, passed):
history = [980, 1010, 1000, 1050] # median 1005
assert volume_within_history(rows, history, tolerance=0.5).passed is passed
def test_decision():
ok = CheckResult("unique_order_id", "dataset", "error", 0, "0 keys", True)
bad = CheckResult("unique_order_id", "dataset", "error", 4, "0 keys", False)
warn = CheckResult("volume", "dataset", "warn", 1600, "502-1508", False)
assert decide([ok], quarantined_rows=0, total_rows=1000) == "publish"
assert decide([ok, warn], quarantined_rows=0, total_rows=1000) == "publish_with_warnings"
assert decide([ok], quarantined_rows=5, total_rows=1000) == "publish_with_warnings"
assert decide([ok], quarantined_rows=50, total_rows=1000) == "hold" # 5% quarantined: systemic
assert decide([bad], quarantined_rows=0, total_rows=1000) == "hold"
(batch builds a DataFrame with the orders schema; T and RUN_AT are fixed timestamps, as in the PySpark testing guide.)
I checked that these tests can fail. Removing the coalesce from _fails, in a copy of the module, makes test_a_null_condition_counts_as_a_failure_not_a_pass fail, which is exactly the regression it exists to catch. Interestingly, the “exactly one output” invariant did not catch that change, because the array-based split never loses rows; it misclassifies them. Invariants and specific behaviour tests protect against different mistakes, so keep both.
Then calibrate in a test environment: run the rule set against representative data that is known to be clean (no error-level failures expected) and against a batch with planted defects (each one must be caught). A rule that fires on clean data is mis-calibrated; a rule that misses a planted defect is broken.
Microsoft Fabric’s built-in options
Fabric gives you some of this declaratively. Use it where it fits, and know its boundaries:
| Mechanism | What it does | Fits |
|---|---|---|
| Materialized lake view row constraints | CONSTRAINT … CHECK (…) ON MISMATCH DROP or FAIL; FAIL is the default and stops the refresh at the first violation; the lineage view shows dropped-record counts |
Row rules on MLV-built layers, where dropping or failing is the right action |
| Materialized lake view column-level suites | Deequ-based checks (completeness, uniqueness, size, ranges…) with FAIL or WARN; a failed FAIL check blocks the dependent views downstream, and results appear in a Data quality tab and the run history |
Dataset checks on MLVs, with a built-in mandatory-versus-advisory split |
| Delta Lake constraints | NOT NULL and CHECK constraints that reject a write when any row violates them |
Invariants that must never be written, where failing the whole write is acceptable |
| Fabric Warehouse keys | PRIMARY KEY, UNIQUE and FOREIGN KEY exist only as NOT ENFORCED |
Documentation and optimisation only: uniqueness and referential integrity still need checks |
Sources: data quality in materialized lake views, Delta Lake constraints, Warehouse table constraints. Check the release status of the materialized lake view features on Microsoft Learn before depending on them in production.
What the built-ins don’t give you on their own: quarantine with reason codes that someone can work through, reconciliation against the source, baseline-relative thresholds from your own history, and one results table across Lakehouse, Warehouse and pipelines. That’s the layer this guide builds; the two combine well. Materialized Lake Views in Microsoft Fabric covers when MLVs fit at all.
Ownership and the rule lifecycle
Rules change. A healthy rule set has a lifecycle:
- Proposed: written with a rationale and an owner, reviewed in a pull request.
- Advisory: runs at
warn(orinfo) while its threshold is calibrated against real history. - Mandatory: promoted to
erroronce it has proven it catches real problems without false alarms. - Retired: removed, with the reason recorded, when the data model changes and the rule no longer means anything.
The owner is the person who answers “the rule fired; is this batch acceptable?” For most rules that is the data owner, not the engineer who wrote the query. Testing a Production Data Platform covers how quality ownership splits between engineering, platform and business.
Common mistakes
| Mistake | Do this instead |
|---|---|
| Checking after publishing | Write, audit, publish: check staging, then publish |
| Treating NULL results as passes | coalesce to false in Spark; test the valid case explicitly in CASE |
| Dropping bad rows silently | Quarantine with reason codes, and record counts |
| Quarantining duplicates | Deduplicate by a business rule in the transformation; fail the batch if duplicates survive |
| Guessed thresholds | Zero tolerance, an agreed ratio, or a baseline from comparable history; say which |
| Every rule at error from day one | Start advisory, promote when calibrated |
| Results nobody reads | One results table, alerts on errors and persistent warnings, an owner per rule |
| Exact source-equals-target reconciliation | Account for every row: published, quarantined, deduplicated |
| Checks that read the wall clock | Pass the run’s reference time in |
| Untested rules | Unit-test each rule with rows that must pass and rows that must fail |
Checklist
- Batches are checked before consumers can see them
- Every rule has a name, level, condition, threshold, severity, action, owner and rationale
- Row rules quarantine with reason codes; the split is total (valid + quarantined = input)
- NULL results count as failures
- Dataset checks cover keys, freshness, volume and reconciliation for critical tables
- A maximum quarantine ratio turns systemic row failures into a batch hold
- Thresholds come from invariants, agreements or baselines, never guesses
- Every run writes one result row per check, joined to the execution log by run ID
- Each rule has unit tests (must pass, must fail) and has been calibrated on representative data
- New rules start advisory; promotion and retirement are deliberate
What to read next
- Testing a Production Data Platform: where data-quality checks sit among tests and CI/CD gates.
- PySpark DataFrame Unit Testing with pytest: the test setup used for the rules above.
- Data Quality Checks cheat sheet: the catalogue of individual checks.
- Designing an Execution Log for Microsoft Fabric Pipelines and Notebooks: where
run_idcomes from, and how results join to runs. - Incremental Loads in Microsoft Fabric: watermarks,
MERGEand periodic reconciliation. - Testing Microsoft Fabric Pipelines and Notebooks: running these checks as pipeline tests, and gating promotion on their results.
Sources
- Microsoft Fabric: Data quality in materialized lake views, Primary, foreign and unique keys in Warehouse, Data types in Fabric Data Warehouse
- Delta Lake: Constraints
- PySpark:
pyspark.sql.functions.array_compact, Testing PySpark - pytest: Parametrizing tests
Last verified: November 2026. The PySpark code and its 15 tests were executed with PySpark 3.5.9, pytest 9.2.0, Python 3.12.10 and Java 21; the T-SQL examples were not executed. The rule anatomy, severity model, thresholds guidance and lifecycle are Javi Builds guidance; product behaviour comes from the sources above.
¿Te resultó útil este contenido?
¿Qué podríamos mejorar?
Gracias — esto me ayuda a mejorar Javi Builds.
Sigue aprendiendo
- Guía rápidaLeer y escribir tablas DeltaLee y escribe tablas Delta con PySpark usando patrones seguros de append, overwrite, merge, evolución del esquema, particionamiento y mantenimiento para pipelines de datos en producción.Abrir guía rápida
- RecursoContenido en inglésFabric Notebook Performance ChecklistA practical pre-run, read, transform, write and orchestration checklist for faster and more reliable Microsoft Fabric notebooks.Ver recurso
¿Qué debería crear después?
Estoy construyendo Javi Builds alrededor de problemas reales que enfrentan los ingenieros de datos. Dime qué tema te gustaría ver después.
Mantente al día
Recibe solo las novedades de Javi Builds que realmente te interesan.
Revisa tu bandeja de entrada
Te he enviado un enlace de confirmación. Tu suscripción se activará cuando la confirmes.