Browse cheat sheets · Data Engineering
Medallion Architecture Quick Reference
Design Bronze, Silver, and Gold data layers with clear ownership, replayability, quality gates, incremental processing, and serving contracts without turning medallion architecture into rigid folder naming.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Bronze: preserve source fidelity
- 2. Bronze: keep raw payload when parsing may evolve
- 3. Bronze: capture source identity
- 4. Bronze: do not overwrite source evidence casually
- 5. Bronze: quarantine malformed input separately
- 6. Silver: define canonical grain first
- 7. Silver: enforce required keys
- 8. Silver: normalize types
- 9. Silver: deduplicate deterministically
- 10. Do not use arbitrary dropDuplicates() for business dedupe
- 11. Silver: validate source uniqueness before MERGE
- 12. Silver: MERGE mutable entities
- 13. Silver: update only with newer versions when needed
- 14. Silver: keep late-arriving logic explicit
- 15. Silver: use reference data carefully
- 16. Silver: distinguish missing reference from invalid reference
- 17. Silver: create canonical business fields
- 18. Silver: keep business meaning stable
- 19. Gold: model for a consumer use case
- 20. Gold: define the output grain explicitly
- 21. Gold: reconcile totals
- 22. Gold can be detailed, not only aggregated
- 23. Gold can be a warehouse model
- 24. Gold can stay in the Lakehouse
- 25. Do not force every layer into separate physical systems
- 26. Bronze should usually be replayable
- 27. Preserve source versions when needed
- 28. Silver should be reproducible
- 29. Version critical reference data
- 30. Gold should have a consumer contract
- 31. Use quality gates between layers
- 32. Add run metadata
- 33. Watermark should advance after successful durable work
- 34. Bronze ingestion can be append-only
- 35. Silver does not have to be append-only
- 36. Gold may be rebuilt or incrementally maintained
- 37. Do not make every layer incremental just because the source is
- 38. Backfill should have explicit boundaries
- 39. Backfill and live processing must coexist safely
- 40. Separate live checkpoint from backfill state
- 41. Streaming fits medallion architecture
- 42. Batch fits the same architecture
- 43. Hybrid is normal
- 44. Do not duplicate transformation logic in batch and streaming paths
- 45. Use source-aligned Bronze names
- 46. Use canonical Silver names
- 47. Use consumer-oriented Gold names
- 48. Do not copy every source table into all three layers
- 49. Not every dataset needs all three layers
- 50. Bronze can keep source schema drift
- 51. Silver should fail or quarantine breaking schema changes
- 52. Add stable data quality rule IDs
- 53. Keep quarantine outside trusted tables
- 54. Quarantine needs remediation metadata
- 55. Silver can preserve rejected-source lineage
- 56. Gold should not bypass Silver casually
- 57. Avoid consumer logic in Bronze
- 58. Avoid source-specific quirks leaking into Gold
- 59. Keep business metrics centralized
- 60. Gold may have multiple consumer models
- 61. Avoid one giant “enterprise Gold” table
- 62. Use dimensional modeling where appropriate
- 63. Gold can serve APIs
- 64. Precompute expensive report logic when useful
- 65. Do not precompute every possible query
- 66. Gold freshness should be explicit
- 67. Data lineage should cross layers
- 68. Use stable business keys
- 69. SCD logic belongs where canonical history is defined
- 70. Separate event history from current-state entities
- 71. Store business time and ingestion time separately
- 72. Late data policy should be layer-specific
- 73. Gold corrections should be deterministic
- 74. Partitioning can differ by layer
- 75. Avoid overpartitioning Bronze
- 76. Avoid overpartitioning Silver/Gold
- 77. Compact small files where needed
- 78. Schema evolution should be governed per layer
- 79. Breaking changes need versioning/migration
- 80. Use table contracts for Gold
- 81. Bronze retention should match replay needs
- 82. Silver retention may differ
- 83. Gold retention follows consumer/legal needs
- 84. Security can differ by layer
- 85. Do not use layer names as security controls
- 86. Mask/tokenize where the trusted model requires it
- 87. Avoid storing secrets in Bronze unless required
- 88. Do not let Bronze become permanent junk storage
- 89. Monitor each layer differently
- 90. Use layer-specific success criteria
- 91. Avoid chain reactions from one bad source batch
- 92. Promotion is a contract transition
- 93. Do not promote by copying files blindly
- 94. Use views when physical duplication is unnecessary
- 95. Materialize when it provides value
- 96. Do not equate folder count with architectural maturity
- 97. Define ownership by layer and dataset
- 98. Keep code modular by responsibility
- 99. Test replay explicitly
- 100. Use a repeatable architecture checklist
- What the result means
- What to look for
- Bronze cannot rebuild Silver
- Silver has duplicate business keys
- Gold does not reconcile to Silver
- Gold logic contains source-specific quirks
- Every source table exists in all three layers with no transformation
- Backfills interfere with live jobs
- Production notes
- Medallion architecture is a responsibility model
- Replayability is one of the strongest design tests
- Quality belongs between layers
- Retention and security vary by layer
- Gold is a data product
- Incremental design should remain recoverable
- Common mistakes
- 1. Treating Bronze as ungoverned dumping ground
- 2. Doing business transformations in Bronze
- 3. Using arbitrary dedupe in Silver
- 4. Letting Gold bypass Silver routinely
- 5. Assuming Gold must be aggregated
- 6. Copying every table into all three layers
- 7. Advancing watermarks before quality gates succeed
- 8. Ignoring late-arriving data
- 9. Using layer names as security
- 10. Having no replay/backfill path
- Next step
- Related deep dives
Problem
A data platform needs clear boundaries between raw ingestion, trusted transformation, and consumer-ready data.
Teams often adopt Bronze, Silver, and Gold labels, but the labels alone do not solve the real engineering questions:
- What is allowed in each layer?
- Which layer is authoritative for replay?
- Where should deduplication happen?
- Where should business rules live?
- How are schema drift and bad records handled?
- How do incremental loads and backfills work?
- Can streaming and batch coexist?
- When should Gold be a lakehouse table versus a warehouse/reporting model?
- How do you prevent the same transformation from being reimplemented differently in multiple layers?
The goal is to use the medallion pattern as a responsibility model, not as a decorative folder hierarchy.
Quick answer
Use this mental model:
Bronze
→ preserve source truth and replayability
Silver
→ create trusted, cleaned, conformed data at a defined business grain
Gold
→ publish consumer-ready data with stable business semantics
A practical flow:
source
→ Bronze: ingest + metadata + minimal normalization
→ Silver: validate + dedupe + conform + enrich
→ Gold: aggregate/model/serve for consumer use
Good default principle:
Bronze answers:
"What did the source send?"
Silver answers:
"What does the trusted canonical data mean?"
Gold answers:
"What should a consumer use?"
Code / commands
1. Bronze: preserve source fidelity
Typical Bronze responsibilities:
append source records
preserve raw fields
capture source file/event metadata
capture ingestion timestamp
capture batch/run identifier
avoid destructive business transformations
Example PySpark pattern:
from pyspark.sql import functions as F
bronze = (
source_df
.withColumn(
"_ingested_at",
F.current_timestamp(),
)
.withColumn(
"_source_file",
F.input_file_name(),
)
.withColumn(
"_batch_id",
F.lit(batch_id),
)
)
Then write append-oriented data:
(
bronze.write
.format("delta")
.mode("append")
.saveAsTable("bronze.orders_raw")
)
Use append semantics when the source represents immutable arrivals.
2. Bronze: keep raw payload when parsing may evolve
For event/API payloads:
parsed fields
+
raw_payload
can be valuable.
Example:
bronze = source_df.withColumn(
"_raw_payload",
F.to_json(F.struct("*")),
)
Do not use this blindly if the payload contains secrets or regulated data.
Preserve only what your retention/security policy allows.
3. Bronze: capture source identity
Useful metadata:
_source_system
_source_file
_source_event_id
_ingested_at
_batch_id
_pipeline_run_id
_source_partition
These fields support:
- replay
- duplicate detection
- lineage
- incident analysis
4. Bronze: do not overwrite source evidence casually
Bad pattern:
read source
→ cast invalid values to null
→ discard original
Better:
raw field
+
parsed field
+
parse status/reason
This keeps evidence for remediation.
5. Bronze: quarantine malformed input separately
Example:
valid = parsed.filter(
F.col("_parse_error").isNull()
)
invalid = parsed.filter(
F.col("_parse_error").isNotNull()
)
Write invalid rows with:
reason
source location
batch
ingestion timestamp
Do not silently drop malformed records.
6. Silver: define canonical grain first
Before transforming Silver, state the row meaning.
Example:
one row per order_id
or:
one row per order line
Do not let joins or retries silently change that grain.
7. Silver: enforce required keys
Example:
silver_candidate = bronze.filter(
F.col("order_id").isNotNull()
)
If null keys are invalid, route them to quarantine rather than quietly discarding them without trace.
8. Silver: normalize types
Example:
silver_candidate = (
bronze
.withColumn(
"order_date",
F.to_date("order_date_raw"),
)
.withColumn(
"amount",
F.col("amount_raw").cast("decimal(18,2)"),
)
)
Validate conversion failures explicitly.
9. Silver: deduplicate deterministically
If the source can resend the same business record, choose a documented winner.
Example:
from pyspark.sql.window import Window
w = (
Window
.partitionBy("order_id")
.orderBy(
F.col("source_updated_at").desc(),
F.col("_ingested_at").desc(),
F.col("_source_event_id").desc(),
)
)
deduped = (
silver_candidate
.withColumn(
"_rn",
F.row_number().over(w),
)
.filter(F.col("_rn") == 1)
.drop("_rn")
)
Tie-breakers must be deterministic.
10. Do not use arbitrary dropDuplicates() for business dedupe
Bad:
df.dropDuplicates(["order_id"])
This does not encode which version wins.
Use ordered window logic or source versioning.
11. Silver: validate source uniqueness before MERGE
duplicates = (
deduped
.groupBy("order_id")
.count()
.filter(F.col("count") > 1)
)
if duplicates.limit(1).count() > 0:
raise ValueError(
"Silver source is not unique by order_id"
)
This protects the target grain.
12. Silver: MERGE mutable entities
from delta.tables import DeltaTable
target = DeltaTable.forName(
spark,
"silver.orders",
)
(
target.alias("t")
.merge(
deduped.alias("s"),
"t.order_id = s.order_id",
)
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)
Use explicit column mappings when you need tighter schema control.
13. Silver: update only with newer versions when needed
Conceptual condition:
source.source_updated_at > target.source_updated_at
Example:
(
target.alias("t")
.merge(
deduped.alias("s"),
"t.order_id = s.order_id",
)
.whenMatchedUpdateAll(
condition=(
"s.source_updated_at > t.source_updated_at"
)
)
.whenNotMatchedInsertAll()
.execute()
)
This avoids older replayed records overwriting newer state.
14. Silver: keep late-arriving logic explicit
Example:
order created on Oct 1
arrives Oct 4
Decide whether Silver:
- updates historical partition
- appends an event
- changes current entity state
- sends to a reconciliation path
Late data behavior belongs in the contract.
15. Silver: use reference data carefully
Example:
orders = (
orders.alias("o")
.join(
customers.alias("c"),
on="customer_id",
how="left",
)
)
Validate that the dimension/reference key is unique before the join.
Otherwise Silver can multiply rows unexpectedly.
16. Silver: distinguish missing reference from invalid reference
A missing customer could mean:
bad key
late-arriving customer
reference refresh lag
Do not classify every miss as corrupt without understanding source timing.
17. Silver: create canonical business fields
Examples:
normalized country code
canonical currency
trusted event timestamp
business status
conformed merchant ID
Avoid consumer-specific presentation logic here unless it is genuinely canonical.
18. Silver: keep business meaning stable
If one Silver table means:
one row per active account
do not later change it to:
one row per account event
without a deliberate contract/version change.
19. Gold: model for a consumer use case
Gold is not merely:
Silver with prettier names
Gold should answer a consumer/business need.
Examples:
daily sales by merchant
customer 360
monthly finance summary
report-ready fact/dimension model
feature table
serving table for API
20. Gold: define the output grain explicitly
Example:
one row per:
report_date + merchant_id + currency
Then aggregate:
gold = (
silver_orders
.groupBy(
"order_date",
"merchant_id",
"currency",
)
.agg(
F.count("*").alias("order_count"),
F.sum("amount").alias("gross_amount"),
)
)
Validate uniqueness at that grain.
21. Gold: reconcile totals
Before publishing:
silver_total = (
silver_orders
.filter(
(F.col("order_date") >= "2026-10-01") &
(F.col("order_date") < "2026-11-01")
)
.agg(
F.sum("amount").alias("amount")
)
)
gold_total = (
gold
.filter(
(F.col("order_date") >= "2026-10-01") &
(F.col("order_date") < "2026-11-01")
)
.agg(
F.sum("gross_amount").alias("amount")
)
)
Gold should reconcile to the trusted upstream semantics for the same scope.
22. Gold can be detailed, not only aggregated
Gold may contain:
consumer-ready detailed fact table
if the consumer needs row-level data.
Do not force aggregation merely because the layer is called Gold.
23. Gold can be a warehouse model
Good fit:
Lakehouse Bronze/Silver
→ Warehouse Gold
when consumers need:
- SQL serving
- dimensional models
- BI/reporting
- stored procedures or warehouse semantics where supported
Do not duplicate data into a warehouse unless the serving value justifies it.
24. Gold can stay in the Lakehouse
Good fit when consumers use:
- Spark
- notebooks
- direct lake access
- lakehouse SQL surfaces
- downstream engineering workloads
Architecture should follow consumer needs.
25. Do not force every layer into separate physical systems
You may have:
Bronze Lakehouse
Silver Lakehouse
Gold Lakehouse
or:
Bronze/Silver Lakehouse
Gold Warehouse
The medallion concept is about responsibility boundaries, not mandatory products.
26. Bronze should usually be replayable
A key test:
Can I rebuild Silver if transformation logic changes?
If not, Bronze may not be preserving enough source history.
27. Preserve source versions when needed
If source sends mutable snapshots only, consider whether you need:
arrival history
snapshot date
source version
for audit/replay.
Do not assume current source state is enough to rebuild historical logic.
28. Silver should be reproducible
Given:
same Bronze scope
same transformation code/version
same reference data/version
you should be able to reproduce the same Silver result as far as practical.
Hidden mutable dependencies reduce replayability.
29. Version critical reference data
If Silver depends on:
merchant hierarchy
currency mapping
country mapping
and those change over time, preserve effective dates or versions where historical correctness matters.
30. Gold should have a consumer contract
Document:
grain
keys
metrics
dimensions
refresh cadence
freshness expectation
late-data behavior
ownership
This makes Gold a product, not merely another table.
31. Use quality gates between layers
Example:
Bronze → Silver gate:
required schema
required key present
parse success
duplicate policy
Silver → Gold gate:
trusted grain
referential integrity
metric reconciliation
freshness
Do not push all validation to the final layer.
32. Add run metadata
Useful metadata per pipeline execution:
run_id
batch_id
source watermark
target watermark
row count
status
code version
started_at
completed_at
This supports lineage and replay.
33. Watermark should advance after successful durable work
Correct:
read
→ transform
→ validate
→ write
→ reconcile
→ advance watermark
Bad:
advance watermark
→ write fails
The latter can skip data.
34. Bronze ingestion can be append-only
Useful when source is:
events
files
CDC records
API snapshots with batch metadata
Append preserves history.
Later Silver logic decides current/canonical state.
35. Silver does not have to be append-only
Mutable canonical entities often need:
MERGE
update
delete
SCD handling
The correct write mode follows the entity semantics.
36. Gold may be rebuilt or incrementally maintained
Full rebuild can be better when:
- Gold is small enough
- logic changes often
- correctness simplicity matters
Incremental Gold can be better when:
- table is large
- daily changes are small
- rebuild cost is high
Choose deliberately.
37. Do not make every layer incremental just because the source is
Sometimes:
Bronze append
Silver MERGE
Gold full rebuild
is the simplest correct design.
Execution mode can differ by layer.
38. Backfill should have explicit boundaries
Example:
backfill:
2026-09-01
through
2026-09-07
Do not use vague:
rebuild recent data
for production corrections.
39. Backfill and live processing must coexist safely
Potential conflict:
live job updates Oct 4
backfill rewrites Sep 1–Oct 4
Define:
- overlap rules
- merge keys
- job coordination
- retry behavior
- concurrency expectations
40. Separate live checkpoint from backfill state
Streaming/live state should not be moved backward by historical repair.
Keep backfill controls independent.
41. Streaming fits medallion architecture
Example:
event stream
→ Bronze durable event table
→ streaming Silver transformation
→ Gold aggregates
The layers still represent responsibilities.
42. Batch fits the same architecture
Example:
daily file
→ Bronze raw load
→ Silver clean/merge
→ Gold report table
Medallion is independent of execution cadence.
43. Hybrid is normal
Example:
Bronze ingested continuously
Silver updated every few minutes
Gold rebuilt hourly
That can be entirely valid if it matches SLAs.
44. Do not duplicate transformation logic in batch and streaming paths
If possible, centralize business transformation rules so:
live processing
and
historical replay
produce consistent results.
45. Use source-aligned Bronze names
Examples:
bronze.salesforce_accounts
bronze.orders_api_events
bronze.erp_invoice_files
Bronze is usually source-centric.
46. Use canonical Silver names
Examples:
silver.customer
silver.order
silver.merchant
Silver often moves from source naming toward business concepts.
47. Use consumer-oriented Gold names
Examples:
gold.daily_merchant_sales
gold.finance_monthly_summary
gold.customer_360
Gold naming should reveal intended use.
48. Do not copy every source table into all three layers
Bad pattern:
bronze.table_a
silver.table_a
gold.table_a
for every table regardless of need.
Create Gold only when there is a real consumer contract.
49. Not every dataset needs all three layers
A tiny reference dataset may go:
source
→ Silver
with raw audit elsewhere.
A temporary staging feed may not need Gold.
Use the responsibilities, not rigid ceremony.
50. Bronze can keep source schema drift
Depending on platform:
new fields
unknown fields
raw payload
may be accepted into Bronze.
Silver should decide whether the new schema is compatible.
51. Silver should fail or quarantine breaking schema changes
Examples:
required key removed
decimal becomes malformed string
timestamp meaning changes
These are not harmless drift.
Fail safely or quarantine according to rule severity.
52. Add stable data quality rule IDs
Example:
DQ_ORDER_001
→ order_id required
DQ_ORDER_002
→ one Silver row per order_id
DQ_ORDER_003
→ currency must be supported
This makes layer gates observable.
53. Keep quarantine outside trusted tables
Invalid records should not silently mix into:
silver.orders
if Silver is advertised as trusted.
Use a dedicated:
quarantine.orders_invalid
or equivalent.
54. Quarantine needs remediation metadata
Include:
rule_id
reason
source_file
batch_id
ingested_at
business key if available
This turns failures into actionable work.
55. Silver can preserve rejected-source lineage
For corrected rows, retain:
_source_system
_source_event_id
_source_file
where useful.
Canonical does not mean lineage-free.
56. Gold should not bypass Silver casually
Bad:
Gold reads raw Bronze directly
because the report is urgent
This creates inconsistent semantics.
Only bypass Silver when the architecture explicitly defines a different trusted path.
57. Avoid consumer logic in Bronze
Bad Bronze transformation:
rename fields to report labels
apply dashboard filters
calculate KPI definitions
Those belong later.
Bronze should remain source-oriented.
58. Avoid source-specific quirks leaking into Gold
Gold should not require consumers to understand:
source system flag X means active
source status 7 means cancelled
Silver should normalize that meaning first.
59. Keep business metrics centralized
If gross_revenue is defined in Gold, document the formula once.
Do not let each report redefine it differently.
60. Gold may have multiple consumer models
Example:
gold.sales_daily
gold.finance_monthly
gold.marketing_attribution
Each can be valid even if derived from the same Silver data.
Gold is consumer/use-case oriented.
61. Avoid one giant “enterprise Gold” table
A huge table attempting to satisfy every consumer usually becomes:
- hard to govern
- hard to optimize
- semantically ambiguous
Prefer purpose-built stable models.
62. Use dimensional modeling where appropriate
Gold can use:
fact tables
dimension tables
star schema
when BI/reporting is the consumer.
Medallion architecture and dimensional modeling are complementary.
63. Gold can serve APIs
Example:
Silver canonical data
→ Gold serving table
→ report/API service
This can isolate expensive transformation from request-time serving.
64. Precompute expensive report logic when useful
Example:
Gold daily aggregate
can avoid recomputing the same expensive joins for every API request.
Use only for stable repeated query patterns.
65. Do not precompute every possible query
If consumer dimensions/filters are highly dynamic, a more flexible serving model may be better.
Balance:
precomputation
vs
query flexibility
66. Gold freshness should be explicit
Example:
refresh every 15 minutes
or:
available by 06:00 UTC daily
Consumers need to know expected freshness.
67. Data lineage should cross layers
At minimum, be able to answer:
Which source/batch created this Silver row?
Which Silver data produced this Gold result?
Which pipeline/code version was used?
Exact lineage tooling varies by platform.
68. Use stable business keys
Surrogate keys may be useful for warehouse models.
Still preserve the source/business identifiers needed for traceability.
69. SCD logic belongs where canonical history is defined
For dimensions:
Silver
or
Gold
can be valid depending on architecture.
The important point is consistency.
Do not implement SCD Type 2 differently in multiple consumer layers.
70. Separate event history from current-state entities
Example:
silver.order_events
→ append history
silver.orders
→ current canonical state
Both can be valuable.
Do not force one table to represent both semantics ambiguously.
71. Store business time and ingestion time separately
Useful columns:
event_time
source_updated_at
ingested_at
processed_at
These answer different questions.
72. Late data policy should be layer-specific
Bronze:
accept/preserve late arrival
Silver:
apply canonical correction
Gold:
recompute impacted aggregates
This is often cleaner than pretending late arrivals do not exist.
73. Gold corrections should be deterministic
If one late Silver record affects:
merchant/day aggregate
rebuild or update that exact Gold scope.
Avoid broad unexplained rewrites.
74. Partitioning can differ by layer
Example:
Bronze
→ ingestion_date
Silver
→ event_date
Gold
→ report_month
This can be valid because each layer has different access patterns.
Do not reuse partition strategy mechanically.
75. Avoid overpartitioning Bronze
High-cardinality source identifiers can create tiny files/directories.
Use coarse operational partitions such as date/source where justified.
76. Avoid overpartitioning Silver/Gold
Use partition columns that align with:
- common filters
- backfills
- retention
- maintenance
not every dimension.
77. Compact small files where needed
For Delta, platform-supported maintenance can include:
OPTIMIZE silver.orders;
Use based on observed file fragmentation.
Do not schedule expensive maintenance without evidence.
78. Schema evolution should be governed per layer
Bronze may tolerate additive drift.
Silver should enforce canonical schema.
Gold should protect consumer contracts strongly.
Different layers can have different evolution policies.
79. Breaking changes need versioning/migration
Example:
Gold metric definition changes
Do not silently alter historical semantics if consumers rely on the old definition.
Options:
- new version/table
- coordinated migration
- documented backfill
80. Use table contracts for Gold
Document:
owner
SLA
grain
keys
refresh
schema
metrics
retention
breaking-change policy
This makes Gold reliable for consumers.
81. Bronze retention should match replay needs
If Bronze is the only replay source and you retain only three days, you cannot rebuild six months of Silver later.
Retention is an architecture decision.
82. Silver retention may differ
Current-state Silver tables may not need the same retention model as append-only Bronze history.
Define each explicitly.
83. Gold retention follows consumer/legal needs
Examples:
13 months reporting
7 years finance
90 days operational dashboard
Do not inherit retention automatically from Silver.
84. Security can differ by layer
Example:
Bronze
→ highly restricted raw PII
Silver
→ masked/conformed
Gold
→ least-privilege consumer access
Medallion can support progressive data exposure.
85. Do not use layer names as security controls
A table being called:
gold.*
does not make it safe.
Use actual permissions, masking, row-level security, and policy.
86. Mask/tokenize where the trusted model requires it
Silver may be the correct point to:
- normalize sensitive identifiers
- tokenize
- mask
- remove unnecessary raw secrets
depending on governance.
87. Avoid storing secrets in Bronze unless required
Raw source fidelity does not justify retaining:
passwords
access tokens
private keys
Remove or protect sensitive data according to policy.
88. Do not let Bronze become permanent junk storage
Preserving raw data does not mean:
no schema
no metadata
no retention
no ownership
Bronze still needs governance.
89. Monitor each layer differently
Bronze:
arrival
schema
file/event count
ingestion latency
Silver:
dedupe
quality
referential integrity
watermark
merge health
Gold:
reconciliation
freshness
consumer SLA
query performance
90. Use layer-specific success criteria
A pipeline can succeed technically while Gold is stale.
Define success per layer, not only “job completed.”
91. Avoid chain reactions from one bad source batch
Where possible:
quarantine bad Bronze batch
prevent Silver promotion
keep unrelated datasets running
Do not let one malformed file stop an entire platform if isolation is possible.
92. Promotion is a contract transition
Conceptually:
Bronze → Silver
means:
source-aligned data passed canonical quality rules
and:
Silver → Gold
means:
trusted canonical data passed consumer/business validation
93. Do not promote by copying files blindly
Layer transition should correspond to transformation/validation responsibility.
If Bronze and Silver are byte-for-byte identical forever, question whether both layers add value.
94. Use views when physical duplication is unnecessary
A logical Gold view over trusted Silver may be enough for some consumers.
Do not materialize every layer/table if there is no performance or isolation benefit.
95. Materialize when it provides value
Reasons:
performance
consumer isolation
stable snapshot
security
simplified model
repeated expensive logic
Materialization should solve a concrete problem.
96. Do not equate folder count with architectural maturity
This:
/bronze/
/silver/
/gold/
does not guarantee:
- replayability
- quality
- contracts
- observability
- idempotency
The engineering behaviors matter.
97. Define ownership by layer and dataset
For every production table, know:
who owns ingestion?
who owns canonical semantics?
who owns consumer metric definitions?
Unclear ownership produces stale and contradictory data.
98. Keep code modular by responsibility
Useful separation:
ingest
validate
normalize
dedupe
merge
aggregate
reconcile
publish
This supports reuse and testing.
99. Test replay explicitly
Take a known Bronze interval:
2026-10-01
rebuild Silver/Gold in a test target and compare expected results.
Do not assume replay works because the code is “deterministic.”
100. Use a repeatable architecture checklist
Before adding a dataset:
1. What is the source truth?
2. What must Bronze preserve?
3. What is the Silver business grain?
4. What dedupe/version rule applies?
5. What quality gates are blocking?
6. What late-data policy applies?
7. What Gold consumer contract is needed?
8. What is the replay/backfill path?
9. What are retention/security rules?
10. What metrics prove the dataset is healthy?
What the result means
| Layer | Main responsibility | Typical write pattern |
|---|---|---|
| Bronze | Preserve source-aligned, replayable data | Append / source snapshots |
| Silver | Produce trusted canonical data | MERGE, dedupe, normalize, validate |
| Gold | Publish consumer-ready semantics | Aggregate, model, serve, rebuild/incremental |
The central idea is:
Bronze
→ evidence
Silver
→ trust
Gold
→ consumption
What to look for
Bronze cannot rebuild Silver
You may not be preserving enough history/source metadata.
Silver has duplicate business keys
Check source retries, dedupe ordering, join cardinality, and MERGE input uniqueness.
Gold does not reconcile to Silver
Check:
- grain
- filters
- joins
- currency/sign rules
- duplicated/dropped rows
- late data
Gold logic contains source-specific quirks
Silver is probably not canonical enough.
Every source table exists in all three layers with no transformation
The architecture may be ceremonial rather than functional.
Backfills interfere with live jobs
Check overlap, checkpoint isolation, write concurrency, and target scope.
Production notes
Medallion architecture is a responsibility model
Do not treat Bronze/Silver/Gold as mandatory physical storage tiers.
Replayability is one of the strongest design tests
If a defect is discovered, the platform should have a credible way to rebuild affected trusted data.
Quality belongs between layers
Layer transitions should correspond to explicit checks, not just successful notebook execution.
Retention and security vary by layer
Raw source retention and access may need to be much stricter than Gold.
Gold is a data product
Stable semantics, freshness, ownership, and change management matter as much as query performance.
Incremental design should remain recoverable
Watermarks/checkpoints are part of correctness.
Treat them as production state.
Common mistakes
1. Treating Bronze as ungoverned dumping ground
Raw still needs ownership, metadata, retention, and security.
2. Doing business transformations in Bronze
This reduces replay/source fidelity.
3. Using arbitrary dedupe in Silver
Canonical state requires a deterministic winner.
4. Letting Gold bypass Silver routinely
This creates conflicting business semantics.
5. Assuming Gold must be aggregated
Detailed consumer-ready Gold tables can be valid.
6. Copying every table into all three layers
Not every dataset needs every layer.
7. Advancing watermarks before quality gates succeed
This risks skipped data.
8. Ignoring late-arriving data
Historical Silver/Gold corrections must be planned.
9. Using layer names as security
Permissions/policies are still required.
10. Having no replay/backfill path
Eventually defects or logic changes will require historical repair.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.