Browse cheat sheets · Data Engineering

Technology: Data EngineeringCategory: ArchitectureDifficulty: IntermediateUpdated

Batch vs Streaming

Choose between batch and streaming data processing by comparing latency, correctness, operational complexity, cost, replay, state, and failure recovery for real production workloads.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Batch processing pattern
  5. 2. Streaming processing pattern
  6. 3. Micro-batch is still streaming
  7. 4. Batch is not necessarily “once per day”
  8. 5. Define the latency requirement before choosing architecture
  9. 6. Bounded batch window
  10. 7. Add a tie-breaker when timestamps are not unique
  11. 8. Batch watermark example
  12. 9. Never advance a batch watermark before success
  13. 10. Streaming checkpoint example
  14. 11. Keep source events durable when possible
  15. 12. Separate ingestion from business transformation
  16. 13. Streaming does not mean “no storage”
  17. 14. Event-time processing
  18. 15. Late-arriving data
  19. 16. Watermarks in streaming are not the same as batch checkpoints
  20. 17. Do not choose an arbitrary lateness threshold
  21. 18. Windowed streaming aggregation
  22. 19. Streaming joins require state
  23. 20. Static-to-stream joins are simpler
  24. 21. Deduplicate streaming events by durable event ID
  25. 22. Idempotent sinks reduce retry risk
  26. 23. Exactly-once claims need careful scope
  27. 24. At-least-once delivery is common
  28. 25. Batch reruns also need idempotency
  29. 26. Full reload batch
  30. 27. Incremental batch
  31. 28. Change Data Capture can feed either model
  32. 29. Event source does not require streaming consumer
  33. 30. Files do not require batch-only architecture
  34. 31. Use trigger-based batch when latency allows it
  35. 32. Streaming is useful for operational decisions
  36. 33. Batch is often enough for analytics
  37. 34. Do not use streaming just for dashboards without a latency requirement
  38. 35. Batch is easier to replay
  39. 36. Backfill must coexist with streaming
  40. 37. Use separate backfill controls
  41. 38. Batch failures are naturally bounded
  42. 39. Streaming failures can accumulate lag
  43. 40. Lag is a core streaming health metric
  44. 41. Throughput must exceed sustained arrival rate
  45. 42. Burst handling matters
  46. 43. Batch has peak-window pressure too
  47. 44. Streaming can move load downstream continuously
  48. 45. Micro-batching can protect downstream systems
  49. 46. Avoid one external API call per event at high volume
  50. 47. Streaming state can become large
  51. 48. Batch also has state, but it is often explicit
  52. 49. Track batch execution metadata
  53. 50. Track streaming execution metadata
  54. 51. Prefer deterministic transformations
  55. 52. Record processing time separately from event time
  56. 53. Separate business freshness from pipeline latency
  57. 54. Event ordering may not be guaranteed globally
  58. 55. Partition key affects streaming order and scale
  59. 56. Avoid one hot key
  60. 57. Batch partitioning also affects performance
  61. 58. Streaming schemas evolve too
  62. 59. Batch schema drift is easier to quarantine
  63. 60. Use dead-letter/quarantine patterns carefully
  64. 61. Do not let one bad record stop the whole stream unnecessarily
  65. 62. Streaming costs are not only compute
  66. 63. Batch costs are not automatically lower
  67. 64. Cost follows work performed
  68. 65. Batch and streaming can share transformations
  69. 66. Build a replay path for streaming logic
  70. 67. Lambda architecture adds duplicate logic
  71. 68. Kappa-style thinking emphasizes replayable streams
  72. 69. Avoid architecture labels as design goals
  73. 70. Use hybrid architecture when requirements differ
  74. 71. Use streaming for early signal, batch for reconciliation
  75. 72. Reconciliation should be deliberate
  76. 73. Define source-of-truth semantics
  77. 74. Data contracts matter more in streaming
  78. 75. Streaming requires stronger operational monitoring
  79. 76. Batch monitoring should track completeness
  80. 77. Define recovery before production
  81. 78. Test failure scenarios
  82. 79. Streaming is not always “real-time”
  83. 80. Choose the simplest architecture that meets the requirement
  84. What the result means
  85. What to look for
  86. Streaming job is running but data is stale
  87. Batch finishes but data is incomplete
  88. Streaming state keeps growing
  89. Backfill creates duplicates
  90. Pipeline repeatedly falls behind during peaks
  91. Frequent batches are meeting the SLA comfortably
  92. Production notes
  93. Latency is a business requirement, not a technology feature
  94. Replayability is critical
  95. Streaming increases operational state
  96. Downstream systems may constrain the choice
  97. Cost and complexity should be measured together
  98. Hybrid is normal
  99. Common mistakes
  100. 1. Choosing streaming because the source is Kafka/Event Hub
  101. 2. Calling hourly batch “not real-time enough” without an SLA
  102. 3. Assuming streaming eliminates duplicates
  103. 4. Assuming streaming gives snapshot consistency
  104. 5. Advancing batch watermarks before successful writes
  105. 6. Using timestamps without tie-breakers
  106. 7. Ignoring late/out-of-order events
  107. 8. Letting streaming state grow without bounds
  108. 9. Treating streaming checkpoints as disposable temp files
  109. 10. Building streaming without a replay/backfill plan
  110. Next step
  111. Related deep dives

Problem

You need to move or transform data, but the architecture decision is unclear:

Should this run every few minutes or hours as a batch?
Or should events be processed continuously as they arrive?

The wrong choice can create unnecessary:

  • latency
  • operational complexity
  • cost
  • duplicate processing
  • state-management problems
  • recovery difficulty
  • pressure on downstream systems

The goal is to choose the simplest processing model that meets the real business latency and correctness requirements.

Quick answer

Use this decision model:

Minutes or hours of latency are acceptable
→ batch is usually simpler

Data must be processed continuously with low latency
→ streaming may be appropriate

Need easy replay and deterministic recomputation
→ batch is usually easier

Need immediate alerts, fraud detection, telemetry, or live operational state
→ streaming may be justified

Need both historical correctness and low latency
→ combine streaming ingestion with durable storage and replayable batch/backfill paths

A useful rule:

Do not choose streaming because the source emits events.

Choose streaming because the business needs continuous processing.

Code / commands

1. Batch processing pattern

Typical flow:

source
→ land raw data
→ process a bounded dataset
→ write curated output
→ checkpoint completion

Example pseudo-workflow:

start = get_last_successful_watermark()
end = get_current_cutoff()

df = read_source(
    start_time=start,
    end_time=end,
)

clean = transform(df)

validate(clean)

write_output(clean)

save_watermark(end)

The important property is that the input range is bounded.


2. Streaming processing pattern

Typical flow:

event source
→ continuously ingest
→ process records/micro-batches
→ maintain state/checkpoints
→ write output continuously

Conceptual Spark Structured Streaming example:

events = (
    spark.readStream
    .format("delta")
    .table("bronze_events")
)

clean = (
    events
    .filter("event_type IS NOT NULL")
    .select(
        "event_id",
        "event_time",
        "event_type",
        "customer_id",
    )
)

query = (
    clean.writeStream
    .format("delta")
    .option("checkpointLocation", "/checkpoints/silver_events")
    .outputMode("append")
    .toTable("silver_events")
)

Streaming requires durable checkpoint/state design, not only a continuously running loop.


3. Micro-batch is still streaming

Many production streaming engines process events as repeated small batches.

Conceptually:

events arrive continuously
→ every short interval, engine processes new data
→ checkpoint advances

This is still a streaming architecture because the pipeline is continuously incremental.


4. Batch is not necessarily “once per day”

Batch can run:

every 5 minutes
every 15 minutes
hourly
nightly
on file arrival
on demand

If five-minute latency satisfies the requirement, a small frequent batch may be simpler than a full streaming platform.


5. Define the latency requirement before choosing architecture

Write the requirement explicitly:

New transaction visible in analytics within:
15 minutes

or:

Fraud alert generated within:
5 seconds

Do not start with:

We should use Kafka/Spark Streaming because it is modern.

Architecture should follow the business SLA.


6. Bounded batch window

A robust batch usually defines:

[start, end)

Example:

WHERE event_time >= @start_time
  AND event_time <  @end_time

Half-open windows avoid overlap between adjacent batches when timestamps are the incremental boundary.


7. Add a tie-breaker when timestamps are not unique

Bad incremental state:

last_processed_timestamp

if thousands of rows can share the same timestamp.

Better state:

last_processed_timestamp
+
last_processed_id

or use a source-provided sequence/version.


8. Batch watermark example

Checkpoint:

{
  "lastSuccessfulEnd": "2026-10-04T14:00:00Z"
}

Next batch:

start = 2026-10-04T14:00:00Z
end   = 2026-10-04T15:00:00Z

Advance the watermark only after durable processing succeeds.


9. Never advance a batch watermark before success

Bad:

save watermark
→ process data
→ pipeline fails

The failed interval may be skipped on retry.

Safer:

read bounded input
→ transform
→ validate
→ write durably
→ persist watermark

10. Streaming checkpoint example

A streaming engine checkpoint typically tracks:

source offsets
progress
state-store metadata
sink commit information

Checkpoint storage should be durable and isolated per logical stream.

Do not point unrelated streaming jobs at the same checkpoint location.


11. Keep source events durable when possible

Useful pattern:

event bus / source
→ durable Bronze/raw layer
→ streaming transformation

Durable raw data makes it possible to:

  • replay
  • backfill
  • debug
  • rebuild downstream state

Streaming without replayability makes production recovery much harder.


12. Separate ingestion from business transformation

A strong architecture often uses:

Source events
→ durable ingestion
→ transformation
→ serving

Do not put every business rule into the ingestion layer if it prevents replaying raw source data later.


13. Streaming does not mean “no storage”

Streaming systems still need durable state for:

  • offsets
  • checkpoints
  • deduplication
  • windows
  • joins
  • aggregation state
  • replay/backfill

Continuous computation increases the importance of state management.


14. Event-time processing

Streaming logic should often use:

event_time

rather than only:

processing_time

Example:

event_time
→ when the business event actually happened

processing_time
→ when the engine received/processed it

They can differ significantly.


15. Late-arriving data

Example:

event happened at 10:00
event arrived at 10:12

A streaming system must decide:

accept late event?
update prior result?
drop it?
route to quarantine?

This is a business correctness decision.


16. Watermarks in streaming are not the same as batch checkpoints

In event-time streaming, a watermark often expresses how far behind event time the engine expects late data may arrive.

Conceptually:

max event time seen
-
allowed lateness

The engine can use that boundary to clean old state.

Exact semantics depend on the streaming engine.


17. Do not choose an arbitrary lateness threshold

Use source and business evidence.

Example questions:

How late can events arrive?
How late are they in real production?
Can old windows be updated?
What downstream consumers expect corrections?

Do not invent a universal five-minute or one-hour watermark.


18. Windowed streaming aggregation

Conceptual PySpark example:

from pyspark.sql import functions as F

agg = (
    events
    .withWatermark("event_time", "30 minutes")
    .groupBy(
        F.window("event_time", "5 minutes"),
        "merchant_id",
    )
    .agg(
        F.count("*").alias("event_count")
    )
)

Use only when a 30-minute lateness policy is actually appropriate to the source and business.

The number above is illustrative, not a universal recommendation.


19. Streaming joins require state

Joining two unbounded streams can require maintaining historical state.

Example:

orders stream
JOIN
payments stream

Without bounded time/state rules, state can grow indefinitely.

Use:

  • event-time constraints
  • watermarks
  • bounded matching windows

where the engine requires/supports them.


20. Static-to-stream joins are simpler

Common pattern:

event stream
JOIN
small reference/dimension table

This is often simpler than stream-to-stream joins.

But decide how reference changes become visible:

  • reload
  • refresh
  • broadcast update
  • versioned table

21. Deduplicate streaming events by durable event ID

Conceptually:

event_id = globally stable source event identifier

Then deduplication can reason about replay/retry.

Avoid generating a new random event ID every time an event is reprocessed.


22. Idempotent sinks reduce retry risk

A pipeline is easier to recover when writing the same logical record twice does not corrupt state.

Examples:

MERGE by business/event key
upsert by stable ID
transactional append with dedupe downstream

Do not depend on “the event will only arrive once.”


23. Exactly-once claims need careful scope

“Exactly once” may refer to:

source offset processing
state update
sink commit
business side effect

These are not automatically identical.

For example:

stream writes row exactly once

does not mean:

email sent exactly once

Design external side effects separately.


24. At-least-once delivery is common

A retry may deliver/process the same event again.

Downstream logic should tolerate this where possible.

Use stable identifiers and idempotency.


25. Batch reruns also need idempotency

Batch is not automatically safe.

Bad rerun:

INSERT entire interval again

can duplicate output.

Safer options:

overwrite bounded partition
MERGE by key
delete/rebuild controlled interval
append with deterministic batch/event IDs

Choose according to table semantics.


26. Full reload batch

Pattern:

read complete source
→ rebuild complete target

Useful when:

  • dataset is small enough
  • logic changes often
  • simple correctness matters more than incremental optimization

Do not implement complex CDC if a safe full rebuild is inexpensive.


27. Incremental batch

Pattern:

watermark
→ read only changed/new data
→ merge into target

Useful when full reload becomes too expensive.

Incremental design adds state and recovery complexity.


28. Change Data Capture can feed either model

CDC does not automatically mean streaming.

CDC records can be processed:

continuously

or:

in periodic batches

Choose consumption mode based on latency requirement.


29. Event source does not require streaming consumer

Example:

Kafka / Event Hub / Kinesis source

can still be consumed into durable storage and processed periodically.

Do not let source technology dictate downstream architecture automatically.


30. Files do not require batch-only architecture

File-arrival events can trigger near-real-time processing.

The architecture can be:

file arrives
→ event trigger
→ bounded processing

This is event-driven batch, not necessarily continuous streaming.


31. Use trigger-based batch when latency allows it

Example:

new file lands
→ orchestrator triggers pipeline
→ process only that file/batch

This can deliver low latency without continuously running a streaming engine.


32. Streaming is useful for operational decisions

Good candidates:

fraud detection
device telemetry alerts
security events
live personalization
real-time operational dashboards
inventory signals

These use cases lose value when processed too late.


33. Batch is often enough for analytics

Good candidates:

daily finance reports
hourly business aggregates
warehouse transformations
regulatory extracts
historical model features
backfills

If the consumer does not need immediate data, batch usually reduces operational burden.


34. Do not use streaming just for dashboards without a latency requirement

Ask:

Does this dashboard truly need 2-second freshness?

If 5–15 minute freshness is acceptable, frequent incremental batch may be simpler and cheaper.


35. Batch is easier to replay

Replay pattern:

choose historical interval
→ rerun transformation
→ compare output

This is valuable when:

  • fixing defects
  • changing business logic
  • rebuilding history

Streaming systems should preserve equivalent replay capability if historical correction matters.


36. Backfill must coexist with streaming

Common production requirement:

live stream processes today
while
backfill repairs last month

Design for:

  • write conflicts
  • duplicate keys
  • partition overlap
  • downstream updates
  • resource contention

Do not assume backfills will never happen.


37. Use separate backfill controls

Example:

live checkpoint
!=
backfill checkpoint

A historical backfill should not move the live stream’s checkpoint backward or overwrite its state accidentally.


38. Batch failures are naturally bounded

If hourly batch 14:00–15:00 fails:

rerun 14:00–15:00

The failure scope is easy to reason about.

Streaming recovery may involve:

  • offsets
  • state
  • checkpoint compatibility
  • event replay
  • late data

39. Streaming failures can accumulate lag

If processing stops:

events continue arriving
→ backlog grows

Recovery requires enough capacity to:

process new arrivals
+
catch up backlog

Monitor consumer lag/backlog explicitly.


40. Lag is a core streaming health metric

Useful concept:

latest source position
-
consumer processed position

or time-based equivalent.

A running streaming job can still be unhealthy if lag keeps increasing.


41. Throughput must exceed sustained arrival rate

If source produces:

10,000 events/sec

and the pipeline sustainably processes:

8,000 events/sec

lag grows indefinitely.

Do not size streaming only for average traffic if real workloads have sustained peaks.


42. Burst handling matters

A stream may normally receive little traffic but occasionally spike.

Options include:

  • buffering in durable broker/storage
  • autoscaling
  • partition scaling
  • bounded backpressure
  • catch-up capacity

43. Batch has peak-window pressure too

A nightly batch may compete with:

  • warehouse users
  • backups
  • maintenance
  • downstream reports

Batch simplifies processing state, but scheduling/resource contention still matters.


44. Streaming can move load downstream continuously

A downstream database that handled:

one bulk hourly merge

may behave differently under:

continuous small updates

Evaluate downstream write patterns.

Streaming upstream does not guarantee downstream systems are suited for constant writes.


45. Micro-batching can protect downstream systems

Instead of writing every event individually:

buffer small interval
→ write batch

This can reduce per-request/transaction overhead.

Choose interval based on latency and sink capacity.


46. Avoid one external API call per event at high volume

Pattern:

event
→ HTTP request
→ event
→ HTTP request

can overload downstream services.

Consider:

  • batching
  • queueing
  • rate limiting
  • async workers
  • deduplicated side-effect processing

47. Streaming state can become large

Stateful operations include:

  • windows
  • deduplication
  • stream-stream joins
  • sessionization

If state is not bounded, memory/storage pressure can grow continuously.

Use appropriate retention/watermark semantics.


48. Batch also has state, but it is often explicit

Examples:

watermark table
processed file manifest
batch ID
high-water mark

Persist state transactionally where possible.


49. Track batch execution metadata

Useful fields:

batch_id
start_watermark
end_watermark
start_time
end_time
row_count
status
source_version
code_version

This helps replay and incident analysis.


50. Track streaming execution metadata

Useful signals:

checkpoint
source offset
processed rows/sec
input rows/sec
lag
state size
batch duration
failed batch
sink commit

Exact metrics depend on the platform.


51. Prefer deterministic transformations

Given the same input:

same code
+
same reference data/version
→ same output

This makes replay safer.

Avoid hidden dependencies on current time or mutable lookup state unless intentional.


52. Record processing time separately from event time

Useful columns:

event_time
ingestion_time
processed_time

These help answer:

When did event happen?
When did platform receive it?
When did transformation complete?

53. Separate business freshness from pipeline latency

Example:

pipeline processed event in 5 seconds

but the source itself delivered it 20 minutes late.

Your pipeline latency is low, but business data freshness is not.

Monitor both.


54. Event ordering may not be guaranteed globally

Distributed sources may preserve order only:

  • within a partition
  • within a key
  • within a producer session

Do not assume global ordering unless the source guarantees it.


55. Partition key affects streaming order and scale

A partition key controls:

distribution
parallelism
per-key ordering
hot partitions

A poor key can create skew.


56. Avoid one hot key

Example:

partition_key = "all_events"

forces all events through one partition/key.

Choose a key aligned with:

  • required ordering
  • workload distribution
  • consumer semantics

57. Batch partitioning also affects performance

A batch can be inefficient if all work lands in:

  • one file
  • one partition
  • one date bucket
  • one tenant

Use partitioning based on actual access and processing patterns.


58. Streaming schemas evolve too

Events may add/change fields over time.

Design:

  • schema compatibility
  • defaults
  • nullable additions
  • versioning
  • quarantine for invalid events

Do not assume event contracts stay static because they are real-time.


59. Batch schema drift is easier to quarantine

A failed file or batch can be isolated without stopping all future processing if orchestration is designed correctly.

Streaming schema errors can stop or poison continuous processing if not handled.


60. Use dead-letter/quarantine patterns carefully

For invalid records:

valid
→ normal path

invalid
→ quarantine with reason/context

Do not silently drop bad records if they matter to business correctness.


61. Do not let one bad record stop the whole stream unnecessarily

Where platform semantics allow, isolate malformed events.

But do not convert serious schema/contract failures into silent data loss.


62. Streaming costs are not only compute

Consider:

event broker
state/checkpoint storage
continuous compute
network
monitoring
operational support
replay storage

63. Batch costs are not automatically lower

A poorly designed full daily scan of massive data can be more expensive than incremental streaming.

Measure actual workload.


64. Cost follows work performed

Useful question:

How much data are we reading, shuffling, storing, and writing?

not simply:

Is it batch or streaming?

65. Batch and streaming can share transformations

A strong design separates:

business transformation logic
from
execution mode

Where practical, the same logic can be reused for:

  • live incremental processing
  • historical replay
  • testing

66. Build a replay path for streaming logic

Example:

Bronze Delta table
→ batch read historical range
→ same Silver transformation

This helps:

  • recover from defects
  • rebuild state
  • validate logic changes

67. Lambda architecture adds duplicate logic

Classic lambda architecture uses:

batch path
+
speed/streaming path

This can produce operational duplication and inconsistent logic.

Use only when justified.

Modern designs often prefer one durable event/raw layer plus reusable incremental/batch processing.


68. Kappa-style thinking emphasizes replayable streams

Conceptually:

durable event log
→ stream processing
→ replay when needed

The practical requirement is not the architecture label.

The important capability is replayable source history and deterministic processing.


69. Avoid architecture labels as design goals

Do not choose:

Lambda
Kappa
real-time
event-driven

because the label sounds modern.

Define:

latency
correctness
replay
cost
failure behavior
operational ownership

first.


70. Use hybrid architecture when requirements differ

Example:

streaming
→ operational alerts

batch
→ authoritative daily finance totals

The same source can support multiple processing paths when business needs differ.


71. Use streaming for early signal, batch for reconciliation

Example:

stream
→ near-real-time dashboard

nightly batch
→ authoritative recomputation/reconciliation

This can be useful when streaming freshness matters but final correctness requires late-arriving data or authoritative source reconciliation.


72. Reconciliation should be deliberate

Compare:

streaming aggregate
vs
authoritative batch aggregate

Investigate divergence.

Do not treat batch correction as permission to ignore streaming quality.


73. Define source-of-truth semantics

Ask:

Is raw event log authoritative?
Is source database authoritative?
Is daily extract authoritative?

Recovery and reconciliation depend on this answer.


74. Data contracts matter more in streaming

Because streaming consumers react quickly, bad producer changes propagate quickly.

Use:

  • schema contracts
  • versioning
  • compatibility testing
  • staged rollout

75. Streaming requires stronger operational monitoring

At minimum watch:

job alive?
input rate
processing rate
lag
checkpoint progress
state growth
error rate
sink failures

A green process status alone is not enough.


76. Batch monitoring should track completeness

Watch:

expected batch arrived?
batch succeeded?
row count plausible?
watermark advanced?
duplicate batch?
late file?

A completed scheduler run does not guarantee complete data.


77. Define recovery before production

For batch:

How do we rerun one interval?

For streaming:

How do we replay from an earlier point?
What happens to state?
How do we avoid duplicate side effects?

If recovery is undefined, the architecture is incomplete.


78. Test failure scenarios

Useful tests:

worker crashes mid-processing
sink unavailable
duplicate event
late event
out-of-order event
schema change
checkpoint unavailable
backfill overlaps live data

Production resilience should be demonstrated, not assumed.


79. Streaming is not always “real-time”

Real end-to-end latency includes:

source delay
broker delay
processing delay
sink delay
consumer/cache refresh delay

A streaming engine alone does not guarantee real-time user experience.


80. Choose the simplest architecture that meets the requirement

Decision order:

1. Define required freshness
2. Define correctness/replay needs
3. Define expected volume and burst behavior
4. Define downstream capabilities
5. Define failure/recovery model
6. Estimate operational complexity/cost
7. Choose batch, streaming, or hybrid

What the result means

Dimension Batch Streaming
Input Bounded Unbounded/continuous
Typical latency Minutes to hours, but can be frequent Seconds to minutes depending on platform/design
State Watermarks/manifests/batch metadata Offsets/checkpoints/state stores/watermarks
Replay Usually straightforward Requires durable source + checkpoint/state strategy
Failure scope Usually bounded interval Can create backlog/lag and state concerns
Late data Next/reconciliation batch Explicit event-time/lateness policy often needed
Operational complexity Usually lower Usually higher
Random backfill Natural fit Often handled through separate replay/backfill path
Live alerts Poor fit if latency is high Strong fit
Large historical recomputation Strong fit Usually done through batch/replay

The core tradeoff is:

lower latency
often requires
more state + more operational complexity

What to look for

Streaming job is running but data is stale

Check:

  • source lag
  • input rate vs processing rate
  • checkpoint progress
  • sink latency
  • downstream cache/serving refresh

Batch finishes but data is incomplete

Check:

  • source arrival completeness
  • watermark
  • skipped files
  • duplicate filtering
  • late-arriving records

Streaming state keeps growing

Inspect:

  • unbounded joins
  • deduplication retention
  • window definitions
  • watermark/lateness policy
  • high-cardinality keys

Backfill creates duplicates

Check:

  • idempotent write logic
  • overlapping date ranges
  • live/backfill coordination
  • merge keys

Pipeline repeatedly falls behind during peaks

Check sustained throughput, partition skew, downstream sink capacity, and catch-up headroom.

Frequent batches are meeting the SLA comfortably

Streaming may add complexity without enough business value.

Production notes

Latency is a business requirement, not a technology feature

Do not optimize for lower latency than consumers need.

Replayability is critical

Whether batch or streaming, preserve enough durable source/history to recover from defects and rebuild outputs.

Streaming increases operational state

Checkpoints, offsets, watermarks, and state stores become part of correctness.

Treat them as production data.

Downstream systems may constrain the choice

A continuous event source does not mean the target database/API can safely absorb one write per event.

Cost and complexity should be measured together

Streaming can reduce freshness latency while increasing:

  • platform services
  • monitoring
  • incident complexity
  • state management

Hybrid is normal

Many mature platforms use both:

streaming for freshness
batch for recomputation/reconciliation

when the requirements justify it.

Common mistakes

1. Choosing streaming because the source is Kafka/Event Hub

Source transport does not define processing requirements.

2. Calling hourly batch “not real-time enough” without an SLA

Define the actual freshness requirement first.

3. Assuming streaming eliminates duplicates

Retries and at-least-once delivery still require idempotent processing.

4. Assuming streaming gives snapshot consistency

Live data can change while it is processed.

5. Advancing batch watermarks before successful writes

This can skip failed intervals.

6. Using timestamps without tie-breakers

Rows sharing the same timestamp can be skipped during incremental processing.

7. Ignoring late/out-of-order events

Event time and processing time are different.

8. Letting streaming state grow without bounds

Windows, joins, and dedupe need retention semantics.

9. Treating streaming checkpoints as disposable temp files

They are part of recovery/correctness.

10. Building streaming without a replay/backfill plan

Eventually you will need to repair history.

Next step

Use the requirement to choose the implementation path.

  • Minutes/hours of latency are acceptable → start with batch.
  • Sub-minute/continuous decisions are required → evaluate streaming.
  • Batch is too slow only during peak data windows → optimize incremental scope before changing architecture.
  • Streaming falls behind → inspect lag, throughput, partitioning, skew, and sink capacity.
  • Late data matters → define event-time and lateness policy.
  • Duplicates appear → add stable event/business keys and idempotent writes.
  • Need historical repair → build a replay/backfill path from durable raw data.
  • Need both fresh and authoritative results → consider streaming for freshness plus batch reconciliation.
  • Unsure which model to choose → quantify latency, correctness, replay, volume, operational ownership, and downstream constraints first.

← All cheat sheets