Browse cheat sheets · Data Engineering

Technology: Data EngineeringCategory: ArchitectureDifficulty: IntermediateUpdated

Medallion Architecture Quick Reference

Design Bronze, Silver, and Gold data layers with clear ownership, replayability, quality gates, incremental processing, and serving contracts without turning medallion architecture into rigid folder naming.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Bronze: preserve source fidelity
  5. 2. Bronze: keep raw payload when parsing may evolve
  6. 3. Bronze: capture source identity
  7. 4. Bronze: do not overwrite source evidence casually
  8. 5. Bronze: quarantine malformed input separately
  9. 6. Silver: define canonical grain first
  10. 7. Silver: enforce required keys
  11. 8. Silver: normalize types
  12. 9. Silver: deduplicate deterministically
  13. 10. Do not use arbitrary dropDuplicates() for business dedupe
  14. 11. Silver: validate source uniqueness before MERGE
  15. 12. Silver: MERGE mutable entities
  16. 13. Silver: update only with newer versions when needed
  17. 14. Silver: keep late-arriving logic explicit
  18. 15. Silver: use reference data carefully
  19. 16. Silver: distinguish missing reference from invalid reference
  20. 17. Silver: create canonical business fields
  21. 18. Silver: keep business meaning stable
  22. 19. Gold: model for a consumer use case
  23. 20. Gold: define the output grain explicitly
  24. 21. Gold: reconcile totals
  25. 22. Gold can be detailed, not only aggregated
  26. 23. Gold can be a warehouse model
  27. 24. Gold can stay in the Lakehouse
  28. 25. Do not force every layer into separate physical systems
  29. 26. Bronze should usually be replayable
  30. 27. Preserve source versions when needed
  31. 28. Silver should be reproducible
  32. 29. Version critical reference data
  33. 30. Gold should have a consumer contract
  34. 31. Use quality gates between layers
  35. 32. Add run metadata
  36. 33. Watermark should advance after successful durable work
  37. 34. Bronze ingestion can be append-only
  38. 35. Silver does not have to be append-only
  39. 36. Gold may be rebuilt or incrementally maintained
  40. 37. Do not make every layer incremental just because the source is
  41. 38. Backfill should have explicit boundaries
  42. 39. Backfill and live processing must coexist safely
  43. 40. Separate live checkpoint from backfill state
  44. 41. Streaming fits medallion architecture
  45. 42. Batch fits the same architecture
  46. 43. Hybrid is normal
  47. 44. Do not duplicate transformation logic in batch and streaming paths
  48. 45. Use source-aligned Bronze names
  49. 46. Use canonical Silver names
  50. 47. Use consumer-oriented Gold names
  51. 48. Do not copy every source table into all three layers
  52. 49. Not every dataset needs all three layers
  53. 50. Bronze can keep source schema drift
  54. 51. Silver should fail or quarantine breaking schema changes
  55. 52. Add stable data quality rule IDs
  56. 53. Keep quarantine outside trusted tables
  57. 54. Quarantine needs remediation metadata
  58. 55. Silver can preserve rejected-source lineage
  59. 56. Gold should not bypass Silver casually
  60. 57. Avoid consumer logic in Bronze
  61. 58. Avoid source-specific quirks leaking into Gold
  62. 59. Keep business metrics centralized
  63. 60. Gold may have multiple consumer models
  64. 61. Avoid one giant “enterprise Gold” table
  65. 62. Use dimensional modeling where appropriate
  66. 63. Gold can serve APIs
  67. 64. Precompute expensive report logic when useful
  68. 65. Do not precompute every possible query
  69. 66. Gold freshness should be explicit
  70. 67. Data lineage should cross layers
  71. 68. Use stable business keys
  72. 69. SCD logic belongs where canonical history is defined
  73. 70. Separate event history from current-state entities
  74. 71. Store business time and ingestion time separately
  75. 72. Late data policy should be layer-specific
  76. 73. Gold corrections should be deterministic
  77. 74. Partitioning can differ by layer
  78. 75. Avoid overpartitioning Bronze
  79. 76. Avoid overpartitioning Silver/Gold
  80. 77. Compact small files where needed
  81. 78. Schema evolution should be governed per layer
  82. 79. Breaking changes need versioning/migration
  83. 80. Use table contracts for Gold
  84. 81. Bronze retention should match replay needs
  85. 82. Silver retention may differ
  86. 83. Gold retention follows consumer/legal needs
  87. 84. Security can differ by layer
  88. 85. Do not use layer names as security controls
  89. 86. Mask/tokenize where the trusted model requires it
  90. 87. Avoid storing secrets in Bronze unless required
  91. 88. Do not let Bronze become permanent junk storage
  92. 89. Monitor each layer differently
  93. 90. Use layer-specific success criteria
  94. 91. Avoid chain reactions from one bad source batch
  95. 92. Promotion is a contract transition
  96. 93. Do not promote by copying files blindly
  97. 94. Use views when physical duplication is unnecessary
  98. 95. Materialize when it provides value
  99. 96. Do not equate folder count with architectural maturity
  100. 97. Define ownership by layer and dataset
  101. 98. Keep code modular by responsibility
  102. 99. Test replay explicitly
  103. 100. Use a repeatable architecture checklist
  104. What the result means
  105. What to look for
  106. Bronze cannot rebuild Silver
  107. Silver has duplicate business keys
  108. Gold does not reconcile to Silver
  109. Gold logic contains source-specific quirks
  110. Every source table exists in all three layers with no transformation
  111. Backfills interfere with live jobs
  112. Production notes
  113. Medallion architecture is a responsibility model
  114. Replayability is one of the strongest design tests
  115. Quality belongs between layers
  116. Retention and security vary by layer
  117. Gold is a data product
  118. Incremental design should remain recoverable
  119. Common mistakes
  120. 1. Treating Bronze as ungoverned dumping ground
  121. 2. Doing business transformations in Bronze
  122. 3. Using arbitrary dedupe in Silver
  123. 4. Letting Gold bypass Silver routinely
  124. 5. Assuming Gold must be aggregated
  125. 6. Copying every table into all three layers
  126. 7. Advancing watermarks before quality gates succeed
  127. 8. Ignoring late-arriving data
  128. 9. Using layer names as security
  129. 10. Having no replay/backfill path
  130. Next step
  131. Related deep dives

Problem

A data platform needs clear boundaries between raw ingestion, trusted transformation, and consumer-ready data.

Teams often adopt Bronze, Silver, and Gold labels, but the labels alone do not solve the real engineering questions:

  • What is allowed in each layer?
  • Which layer is authoritative for replay?
  • Where should deduplication happen?
  • Where should business rules live?
  • How are schema drift and bad records handled?
  • How do incremental loads and backfills work?
  • Can streaming and batch coexist?
  • When should Gold be a lakehouse table versus a warehouse/reporting model?
  • How do you prevent the same transformation from being reimplemented differently in multiple layers?

The goal is to use the medallion pattern as a responsibility model, not as a decorative folder hierarchy.

Quick answer

Use this mental model:

Bronze
→ preserve source truth and replayability

Silver
→ create trusted, cleaned, conformed data at a defined business grain

Gold
→ publish consumer-ready data with stable business semantics

A practical flow:

source
→ Bronze: ingest + metadata + minimal normalization
→ Silver: validate + dedupe + conform + enrich
→ Gold: aggregate/model/serve for consumer use

Good default principle:

Bronze answers:
"What did the source send?"

Silver answers:
"What does the trusted canonical data mean?"

Gold answers:
"What should a consumer use?"

Code / commands

1. Bronze: preserve source fidelity

Typical Bronze responsibilities:

append source records
preserve raw fields
capture source file/event metadata
capture ingestion timestamp
capture batch/run identifier
avoid destructive business transformations

Example PySpark pattern:

from pyspark.sql import functions as F

bronze = (
    source_df
    .withColumn(
        "_ingested_at",
        F.current_timestamp(),
    )
    .withColumn(
        "_source_file",
        F.input_file_name(),
    )
    .withColumn(
        "_batch_id",
        F.lit(batch_id),
    )
)

Then write append-oriented data:

(
    bronze.write
    .format("delta")
    .mode("append")
    .saveAsTable("bronze.orders_raw")
)

Use append semantics when the source represents immutable arrivals.


2. Bronze: keep raw payload when parsing may evolve

For event/API payloads:

parsed fields
+
raw_payload

can be valuable.

Example:

bronze = source_df.withColumn(
    "_raw_payload",
    F.to_json(F.struct("*")),
)

Do not use this blindly if the payload contains secrets or regulated data.

Preserve only what your retention/security policy allows.


3. Bronze: capture source identity

Useful metadata:

_source_system
_source_file
_source_event_id
_ingested_at
_batch_id
_pipeline_run_id
_source_partition

These fields support:

  • replay
  • duplicate detection
  • lineage
  • incident analysis

4. Bronze: do not overwrite source evidence casually

Bad pattern:

read source
→ cast invalid values to null
→ discard original

Better:

raw field
+
parsed field
+
parse status/reason

This keeps evidence for remediation.


5. Bronze: quarantine malformed input separately

Example:

valid = parsed.filter(
    F.col("_parse_error").isNull()
)

invalid = parsed.filter(
    F.col("_parse_error").isNotNull()
)

Write invalid rows with:

reason
source location
batch
ingestion timestamp

Do not silently drop malformed records.


6. Silver: define canonical grain first

Before transforming Silver, state the row meaning.

Example:

one row per order_id

or:

one row per order line

Do not let joins or retries silently change that grain.


7. Silver: enforce required keys

Example:

silver_candidate = bronze.filter(
    F.col("order_id").isNotNull()
)

If null keys are invalid, route them to quarantine rather than quietly discarding them without trace.


8. Silver: normalize types

Example:

silver_candidate = (
    bronze
    .withColumn(
        "order_date",
        F.to_date("order_date_raw"),
    )
    .withColumn(
        "amount",
        F.col("amount_raw").cast("decimal(18,2)"),
    )
)

Validate conversion failures explicitly.


9. Silver: deduplicate deterministically

If the source can resend the same business record, choose a documented winner.

Example:

from pyspark.sql.window import Window

w = (
    Window
    .partitionBy("order_id")
    .orderBy(
        F.col("source_updated_at").desc(),
        F.col("_ingested_at").desc(),
        F.col("_source_event_id").desc(),
    )
)

deduped = (
    silver_candidate
    .withColumn(
        "_rn",
        F.row_number().over(w),
    )
    .filter(F.col("_rn") == 1)
    .drop("_rn")
)

Tie-breakers must be deterministic.


10. Do not use arbitrary dropDuplicates() for business dedupe

Bad:

df.dropDuplicates(["order_id"])

This does not encode which version wins.

Use ordered window logic or source versioning.


11. Silver: validate source uniqueness before MERGE

duplicates = (
    deduped
    .groupBy("order_id")
    .count()
    .filter(F.col("count") > 1)
)

if duplicates.limit(1).count() > 0:
    raise ValueError(
        "Silver source is not unique by order_id"
    )

This protects the target grain.


12. Silver: MERGE mutable entities

from delta.tables import DeltaTable

target = DeltaTable.forName(
    spark,
    "silver.orders",
)

(
    target.alias("t")
    .merge(
        deduped.alias("s"),
        "t.order_id = s.order_id",
    )
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute()
)

Use explicit column mappings when you need tighter schema control.


13. Silver: update only with newer versions when needed

Conceptual condition:

source.source_updated_at > target.source_updated_at

Example:

(
    target.alias("t")
    .merge(
        deduped.alias("s"),
        "t.order_id = s.order_id",
    )
    .whenMatchedUpdateAll(
        condition=(
            "s.source_updated_at > t.source_updated_at"
        )
    )
    .whenNotMatchedInsertAll()
    .execute()
)

This avoids older replayed records overwriting newer state.


14. Silver: keep late-arriving logic explicit

Example:

order created on Oct 1
arrives Oct 4

Decide whether Silver:

  • updates historical partition
  • appends an event
  • changes current entity state
  • sends to a reconciliation path

Late data behavior belongs in the contract.


15. Silver: use reference data carefully

Example:

orders = (
    orders.alias("o")
    .join(
        customers.alias("c"),
        on="customer_id",
        how="left",
    )
)

Validate that the dimension/reference key is unique before the join.

Otherwise Silver can multiply rows unexpectedly.


16. Silver: distinguish missing reference from invalid reference

A missing customer could mean:

bad key
late-arriving customer
reference refresh lag

Do not classify every miss as corrupt without understanding source timing.


17. Silver: create canonical business fields

Examples:

normalized country code
canonical currency
trusted event timestamp
business status
conformed merchant ID

Avoid consumer-specific presentation logic here unless it is genuinely canonical.


18. Silver: keep business meaning stable

If one Silver table means:

one row per active account

do not later change it to:

one row per account event

without a deliberate contract/version change.


19. Gold: model for a consumer use case

Gold is not merely:

Silver with prettier names

Gold should answer a consumer/business need.

Examples:

daily sales by merchant
customer 360
monthly finance summary
report-ready fact/dimension model
feature table
serving table for API

20. Gold: define the output grain explicitly

Example:

one row per:
report_date + merchant_id + currency

Then aggregate:

gold = (
    silver_orders
    .groupBy(
        "order_date",
        "merchant_id",
        "currency",
    )
    .agg(
        F.count("*").alias("order_count"),
        F.sum("amount").alias("gross_amount"),
    )
)

Validate uniqueness at that grain.


21. Gold: reconcile totals

Before publishing:

silver_total = (
    silver_orders
    .filter(
        (F.col("order_date") >= "2026-10-01") &
        (F.col("order_date") <  "2026-11-01")
    )
    .agg(
        F.sum("amount").alias("amount")
    )
)

gold_total = (
    gold
    .filter(
        (F.col("order_date") >= "2026-10-01") &
        (F.col("order_date") <  "2026-11-01")
    )
    .agg(
        F.sum("gross_amount").alias("amount")
    )
)

Gold should reconcile to the trusted upstream semantics for the same scope.


22. Gold can be detailed, not only aggregated

Gold may contain:

consumer-ready detailed fact table

if the consumer needs row-level data.

Do not force aggregation merely because the layer is called Gold.


23. Gold can be a warehouse model

Good fit:

Lakehouse Bronze/Silver
→ Warehouse Gold

when consumers need:

  • SQL serving
  • dimensional models
  • BI/reporting
  • stored procedures or warehouse semantics where supported

Do not duplicate data into a warehouse unless the serving value justifies it.


24. Gold can stay in the Lakehouse

Good fit when consumers use:

  • Spark
  • notebooks
  • direct lake access
  • lakehouse SQL surfaces
  • downstream engineering workloads

Architecture should follow consumer needs.


25. Do not force every layer into separate physical systems

You may have:

Bronze Lakehouse
Silver Lakehouse
Gold Lakehouse

or:

Bronze/Silver Lakehouse
Gold Warehouse

The medallion concept is about responsibility boundaries, not mandatory products.


26. Bronze should usually be replayable

A key test:

Can I rebuild Silver if transformation logic changes?

If not, Bronze may not be preserving enough source history.


27. Preserve source versions when needed

If source sends mutable snapshots only, consider whether you need:

arrival history
snapshot date
source version

for audit/replay.

Do not assume current source state is enough to rebuild historical logic.


28. Silver should be reproducible

Given:

same Bronze scope
same transformation code/version
same reference data/version

you should be able to reproduce the same Silver result as far as practical.

Hidden mutable dependencies reduce replayability.


29. Version critical reference data

If Silver depends on:

merchant hierarchy
currency mapping
country mapping

and those change over time, preserve effective dates or versions where historical correctness matters.


30. Gold should have a consumer contract

Document:

grain
keys
metrics
dimensions
refresh cadence
freshness expectation
late-data behavior
ownership

This makes Gold a product, not merely another table.


31. Use quality gates between layers

Example:

Bronze → Silver gate:
required schema
required key present
parse success
duplicate policy

Silver → Gold gate:
trusted grain
referential integrity
metric reconciliation
freshness

Do not push all validation to the final layer.


32. Add run metadata

Useful metadata per pipeline execution:

run_id
batch_id
source watermark
target watermark
row count
status
code version
started_at
completed_at

This supports lineage and replay.


33. Watermark should advance after successful durable work

Correct:

read
→ transform
→ validate
→ write
→ reconcile
→ advance watermark

Bad:

advance watermark
→ write fails

The latter can skip data.


34. Bronze ingestion can be append-only

Useful when source is:

events
files
CDC records
API snapshots with batch metadata

Append preserves history.

Later Silver logic decides current/canonical state.


35. Silver does not have to be append-only

Mutable canonical entities often need:

MERGE
update
delete
SCD handling

The correct write mode follows the entity semantics.


36. Gold may be rebuilt or incrementally maintained

Full rebuild can be better when:

  • Gold is small enough
  • logic changes often
  • correctness simplicity matters

Incremental Gold can be better when:

  • table is large
  • daily changes are small
  • rebuild cost is high

Choose deliberately.


37. Do not make every layer incremental just because the source is

Sometimes:

Bronze append
Silver MERGE
Gold full rebuild

is the simplest correct design.

Execution mode can differ by layer.


38. Backfill should have explicit boundaries

Example:

backfill:
2026-09-01
through
2026-09-07

Do not use vague:

rebuild recent data

for production corrections.


39. Backfill and live processing must coexist safely

Potential conflict:

live job updates Oct 4
backfill rewrites Sep 1–Oct 4

Define:

  • overlap rules
  • merge keys
  • job coordination
  • retry behavior
  • concurrency expectations

40. Separate live checkpoint from backfill state

Streaming/live state should not be moved backward by historical repair.

Keep backfill controls independent.


41. Streaming fits medallion architecture

Example:

event stream
→ Bronze durable event table
→ streaming Silver transformation
→ Gold aggregates

The layers still represent responsibilities.


42. Batch fits the same architecture

Example:

daily file
→ Bronze raw load
→ Silver clean/merge
→ Gold report table

Medallion is independent of execution cadence.


43. Hybrid is normal

Example:

Bronze ingested continuously
Silver updated every few minutes
Gold rebuilt hourly

That can be entirely valid if it matches SLAs.


44. Do not duplicate transformation logic in batch and streaming paths

If possible, centralize business transformation rules so:

live processing
and
historical replay

produce consistent results.


45. Use source-aligned Bronze names

Examples:

bronze.salesforce_accounts
bronze.orders_api_events
bronze.erp_invoice_files

Bronze is usually source-centric.


46. Use canonical Silver names

Examples:

silver.customer
silver.order
silver.merchant

Silver often moves from source naming toward business concepts.


47. Use consumer-oriented Gold names

Examples:

gold.daily_merchant_sales
gold.finance_monthly_summary
gold.customer_360

Gold naming should reveal intended use.


48. Do not copy every source table into all three layers

Bad pattern:

bronze.table_a
silver.table_a
gold.table_a

for every table regardless of need.

Create Gold only when there is a real consumer contract.


49. Not every dataset needs all three layers

A tiny reference dataset may go:

source
→ Silver

with raw audit elsewhere.

A temporary staging feed may not need Gold.

Use the responsibilities, not rigid ceremony.


50. Bronze can keep source schema drift

Depending on platform:

new fields
unknown fields
raw payload

may be accepted into Bronze.

Silver should decide whether the new schema is compatible.


51. Silver should fail or quarantine breaking schema changes

Examples:

required key removed
decimal becomes malformed string
timestamp meaning changes

These are not harmless drift.

Fail safely or quarantine according to rule severity.


52. Add stable data quality rule IDs

Example:

DQ_ORDER_001
→ order_id required

DQ_ORDER_002
→ one Silver row per order_id

DQ_ORDER_003
→ currency must be supported

This makes layer gates observable.


53. Keep quarantine outside trusted tables

Invalid records should not silently mix into:

silver.orders

if Silver is advertised as trusted.

Use a dedicated:

quarantine.orders_invalid

or equivalent.


54. Quarantine needs remediation metadata

Include:

rule_id
reason
source_file
batch_id
ingested_at
business key if available

This turns failures into actionable work.


55. Silver can preserve rejected-source lineage

For corrected rows, retain:

_source_system
_source_event_id
_source_file

where useful.

Canonical does not mean lineage-free.


56. Gold should not bypass Silver casually

Bad:

Gold reads raw Bronze directly
because the report is urgent

This creates inconsistent semantics.

Only bypass Silver when the architecture explicitly defines a different trusted path.


57. Avoid consumer logic in Bronze

Bad Bronze transformation:

rename fields to report labels
apply dashboard filters
calculate KPI definitions

Those belong later.

Bronze should remain source-oriented.


58. Avoid source-specific quirks leaking into Gold

Gold should not require consumers to understand:

source system flag X means active
source status 7 means cancelled

Silver should normalize that meaning first.


59. Keep business metrics centralized

If gross_revenue is defined in Gold, document the formula once.

Do not let each report redefine it differently.


60. Gold may have multiple consumer models

Example:

gold.sales_daily
gold.finance_monthly
gold.marketing_attribution

Each can be valid even if derived from the same Silver data.

Gold is consumer/use-case oriented.


61. Avoid one giant “enterprise Gold” table

A huge table attempting to satisfy every consumer usually becomes:

  • hard to govern
  • hard to optimize
  • semantically ambiguous

Prefer purpose-built stable models.


62. Use dimensional modeling where appropriate

Gold can use:

fact tables
dimension tables
star schema

when BI/reporting is the consumer.

Medallion architecture and dimensional modeling are complementary.


63. Gold can serve APIs

Example:

Silver canonical data
→ Gold serving table
→ report/API service

This can isolate expensive transformation from request-time serving.


64. Precompute expensive report logic when useful

Example:

Gold daily aggregate

can avoid recomputing the same expensive joins for every API request.

Use only for stable repeated query patterns.


65. Do not precompute every possible query

If consumer dimensions/filters are highly dynamic, a more flexible serving model may be better.

Balance:

precomputation
vs
query flexibility

66. Gold freshness should be explicit

Example:

refresh every 15 minutes

or:

available by 06:00 UTC daily

Consumers need to know expected freshness.


67. Data lineage should cross layers

At minimum, be able to answer:

Which source/batch created this Silver row?
Which Silver data produced this Gold result?
Which pipeline/code version was used?

Exact lineage tooling varies by platform.


68. Use stable business keys

Surrogate keys may be useful for warehouse models.

Still preserve the source/business identifiers needed for traceability.


69. SCD logic belongs where canonical history is defined

For dimensions:

Silver
or
Gold

can be valid depending on architecture.

The important point is consistency.

Do not implement SCD Type 2 differently in multiple consumer layers.


70. Separate event history from current-state entities

Example:

silver.order_events
→ append history

silver.orders
→ current canonical state

Both can be valuable.

Do not force one table to represent both semantics ambiguously.


71. Store business time and ingestion time separately

Useful columns:

event_time
source_updated_at
ingested_at
processed_at

These answer different questions.


72. Late data policy should be layer-specific

Bronze:

accept/preserve late arrival

Silver:

apply canonical correction

Gold:

recompute impacted aggregates

This is often cleaner than pretending late arrivals do not exist.


73. Gold corrections should be deterministic

If one late Silver record affects:

merchant/day aggregate

rebuild or update that exact Gold scope.

Avoid broad unexplained rewrites.


74. Partitioning can differ by layer

Example:

Bronze
→ ingestion_date

Silver
→ event_date

Gold
→ report_month

This can be valid because each layer has different access patterns.

Do not reuse partition strategy mechanically.


75. Avoid overpartitioning Bronze

High-cardinality source identifiers can create tiny files/directories.

Use coarse operational partitions such as date/source where justified.


76. Avoid overpartitioning Silver/Gold

Use partition columns that align with:

  • common filters
  • backfills
  • retention
  • maintenance

not every dimension.


77. Compact small files where needed

For Delta, platform-supported maintenance can include:

OPTIMIZE silver.orders;

Use based on observed file fragmentation.

Do not schedule expensive maintenance without evidence.


78. Schema evolution should be governed per layer

Bronze may tolerate additive drift.

Silver should enforce canonical schema.

Gold should protect consumer contracts strongly.

Different layers can have different evolution policies.


79. Breaking changes need versioning/migration

Example:

Gold metric definition changes

Do not silently alter historical semantics if consumers rely on the old definition.

Options:

  • new version/table
  • coordinated migration
  • documented backfill

80. Use table contracts for Gold

Document:

owner
SLA
grain
keys
refresh
schema
metrics
retention
breaking-change policy

This makes Gold reliable for consumers.


81. Bronze retention should match replay needs

If Bronze is the only replay source and you retain only three days, you cannot rebuild six months of Silver later.

Retention is an architecture decision.


82. Silver retention may differ

Current-state Silver tables may not need the same retention model as append-only Bronze history.

Define each explicitly.


83. Gold retention follows consumer/legal needs

Examples:

13 months reporting
7 years finance
90 days operational dashboard

Do not inherit retention automatically from Silver.


84. Security can differ by layer

Example:

Bronze
→ highly restricted raw PII

Silver
→ masked/conformed

Gold
→ least-privilege consumer access

Medallion can support progressive data exposure.


85. Do not use layer names as security controls

A table being called:

gold.*

does not make it safe.

Use actual permissions, masking, row-level security, and policy.


86. Mask/tokenize where the trusted model requires it

Silver may be the correct point to:

  • normalize sensitive identifiers
  • tokenize
  • mask
  • remove unnecessary raw secrets

depending on governance.


87. Avoid storing secrets in Bronze unless required

Raw source fidelity does not justify retaining:

passwords
access tokens
private keys

Remove or protect sensitive data according to policy.


88. Do not let Bronze become permanent junk storage

Preserving raw data does not mean:

no schema
no metadata
no retention
no ownership

Bronze still needs governance.


89. Monitor each layer differently

Bronze:

arrival
schema
file/event count
ingestion latency

Silver:

dedupe
quality
referential integrity
watermark
merge health

Gold:

reconciliation
freshness
consumer SLA
query performance

90. Use layer-specific success criteria

A pipeline can succeed technically while Gold is stale.

Define success per layer, not only “job completed.”


91. Avoid chain reactions from one bad source batch

Where possible:

quarantine bad Bronze batch
prevent Silver promotion
keep unrelated datasets running

Do not let one malformed file stop an entire platform if isolation is possible.


92. Promotion is a contract transition

Conceptually:

Bronze → Silver

means:

source-aligned data passed canonical quality rules

and:

Silver → Gold

means:

trusted canonical data passed consumer/business validation

93. Do not promote by copying files blindly

Layer transition should correspond to transformation/validation responsibility.

If Bronze and Silver are byte-for-byte identical forever, question whether both layers add value.


94. Use views when physical duplication is unnecessary

A logical Gold view over trusted Silver may be enough for some consumers.

Do not materialize every layer/table if there is no performance or isolation benefit.


95. Materialize when it provides value

Reasons:

performance
consumer isolation
stable snapshot
security
simplified model
repeated expensive logic

Materialization should solve a concrete problem.


96. Do not equate folder count with architectural maturity

This:

/bronze/
/silver/
/gold/

does not guarantee:

  • replayability
  • quality
  • contracts
  • observability
  • idempotency

The engineering behaviors matter.


97. Define ownership by layer and dataset

For every production table, know:

who owns ingestion?
who owns canonical semantics?
who owns consumer metric definitions?

Unclear ownership produces stale and contradictory data.


98. Keep code modular by responsibility

Useful separation:

ingest
validate
normalize
dedupe
merge
aggregate
reconcile
publish

This supports reuse and testing.


99. Test replay explicitly

Take a known Bronze interval:

2026-10-01

rebuild Silver/Gold in a test target and compare expected results.

Do not assume replay works because the code is “deterministic.”


100. Use a repeatable architecture checklist

Before adding a dataset:

1. What is the source truth?
2. What must Bronze preserve?
3. What is the Silver business grain?
4. What dedupe/version rule applies?
5. What quality gates are blocking?
6. What late-data policy applies?
7. What Gold consumer contract is needed?
8. What is the replay/backfill path?
9. What are retention/security rules?
10. What metrics prove the dataset is healthy?

What the result means

Layer Main responsibility Typical write pattern
Bronze Preserve source-aligned, replayable data Append / source snapshots
Silver Produce trusted canonical data MERGE, dedupe, normalize, validate
Gold Publish consumer-ready semantics Aggregate, model, serve, rebuild/incremental

The central idea is:

Bronze
→ evidence

Silver
→ trust

Gold
→ consumption

What to look for

Bronze cannot rebuild Silver

You may not be preserving enough history/source metadata.

Silver has duplicate business keys

Check source retries, dedupe ordering, join cardinality, and MERGE input uniqueness.

Gold does not reconcile to Silver

Check:

  • grain
  • filters
  • joins
  • currency/sign rules
  • duplicated/dropped rows
  • late data

Gold logic contains source-specific quirks

Silver is probably not canonical enough.

Every source table exists in all three layers with no transformation

The architecture may be ceremonial rather than functional.

Backfills interfere with live jobs

Check overlap, checkpoint isolation, write concurrency, and target scope.

Production notes

Medallion architecture is a responsibility model

Do not treat Bronze/Silver/Gold as mandatory physical storage tiers.

Replayability is one of the strongest design tests

If a defect is discovered, the platform should have a credible way to rebuild affected trusted data.

Quality belongs between layers

Layer transitions should correspond to explicit checks, not just successful notebook execution.

Retention and security vary by layer

Raw source retention and access may need to be much stricter than Gold.

Gold is a data product

Stable semantics, freshness, ownership, and change management matter as much as query performance.

Incremental design should remain recoverable

Watermarks/checkpoints are part of correctness.

Treat them as production state.

Common mistakes

1. Treating Bronze as ungoverned dumping ground

Raw still needs ownership, metadata, retention, and security.

2. Doing business transformations in Bronze

This reduces replay/source fidelity.

3. Using arbitrary dedupe in Silver

Canonical state requires a deterministic winner.

4. Letting Gold bypass Silver routinely

This creates conflicting business semantics.

5. Assuming Gold must be aggregated

Detailed consumer-ready Gold tables can be valid.

6. Copying every table into all three layers

Not every dataset needs every layer.

7. Advancing watermarks before quality gates succeed

This risks skipped data.

8. Ignoring late-arriving data

Historical Silver/Gold corrections must be planned.

9. Using layer names as security

Permissions/policies are still required.

10. Having no replay/backfill path

Eventually defects or logic changes will require historical repair.

Next step

Use the layer symptom to choose the next action.

  • Need raw replay/audit → strengthen Bronze source fidelity and metadata.
  • Silver has duplicates → fix grain, dedupe ordering, or join cardinality.
  • Silver schema keeps drifting → define governed compatibility rules.
  • Gold does not reconcile → compare grain and business totals with Silver.
  • Consumers need lower latency → change execution cadence, not layer responsibilities.
  • Backfills conflict with live processing → isolate state/checkpoints and target scopes.
  • Gold serves BI/reporting → evaluate Warehouse/dimensional modeling where useful.
  • Gold serves Spark/data science → Lakehouse may remain the right surface.
  • Every layer feels redundant → collapse unnecessary physical copies while keeping responsibility boundaries.
  • Need implementation detail → move to Delta Lake Read and Write Patterns or Incremental Load Patterns.

← All cheat sheets