Browse cheat sheets · Data Engineering
Batch vs Streaming
Choose between batch and streaming data processing by comparing latency, correctness, operational complexity, cost, replay, state, and failure recovery for real production workloads.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Batch processing pattern
- 2. Streaming processing pattern
- 3. Micro-batch is still streaming
- 4. Batch is not necessarily “once per day”
- 5. Define the latency requirement before choosing architecture
- 6. Bounded batch window
- 7. Add a tie-breaker when timestamps are not unique
- 8. Batch watermark example
- 9. Never advance a batch watermark before success
- 10. Streaming checkpoint example
- 11. Keep source events durable when possible
- 12. Separate ingestion from business transformation
- 13. Streaming does not mean “no storage”
- 14. Event-time processing
- 15. Late-arriving data
- 16. Watermarks in streaming are not the same as batch checkpoints
- 17. Do not choose an arbitrary lateness threshold
- 18. Windowed streaming aggregation
- 19. Streaming joins require state
- 20. Static-to-stream joins are simpler
- 21. Deduplicate streaming events by durable event ID
- 22. Idempotent sinks reduce retry risk
- 23. Exactly-once claims need careful scope
- 24. At-least-once delivery is common
- 25. Batch reruns also need idempotency
- 26. Full reload batch
- 27. Incremental batch
- 28. Change Data Capture can feed either model
- 29. Event source does not require streaming consumer
- 30. Files do not require batch-only architecture
- 31. Use trigger-based batch when latency allows it
- 32. Streaming is useful for operational decisions
- 33. Batch is often enough for analytics
- 34. Do not use streaming just for dashboards without a latency requirement
- 35. Batch is easier to replay
- 36. Backfill must coexist with streaming
- 37. Use separate backfill controls
- 38. Batch failures are naturally bounded
- 39. Streaming failures can accumulate lag
- 40. Lag is a core streaming health metric
- 41. Throughput must exceed sustained arrival rate
- 42. Burst handling matters
- 43. Batch has peak-window pressure too
- 44. Streaming can move load downstream continuously
- 45. Micro-batching can protect downstream systems
- 46. Avoid one external API call per event at high volume
- 47. Streaming state can become large
- 48. Batch also has state, but it is often explicit
- 49. Track batch execution metadata
- 50. Track streaming execution metadata
- 51. Prefer deterministic transformations
- 52. Record processing time separately from event time
- 53. Separate business freshness from pipeline latency
- 54. Event ordering may not be guaranteed globally
- 55. Partition key affects streaming order and scale
- 56. Avoid one hot key
- 57. Batch partitioning also affects performance
- 58. Streaming schemas evolve too
- 59. Batch schema drift is easier to quarantine
- 60. Use dead-letter/quarantine patterns carefully
- 61. Do not let one bad record stop the whole stream unnecessarily
- 62. Streaming costs are not only compute
- 63. Batch costs are not automatically lower
- 64. Cost follows work performed
- 65. Batch and streaming can share transformations
- 66. Build a replay path for streaming logic
- 67. Lambda architecture adds duplicate logic
- 68. Kappa-style thinking emphasizes replayable streams
- 69. Avoid architecture labels as design goals
- 70. Use hybrid architecture when requirements differ
- 71. Use streaming for early signal, batch for reconciliation
- 72. Reconciliation should be deliberate
- 73. Define source-of-truth semantics
- 74. Data contracts matter more in streaming
- 75. Streaming requires stronger operational monitoring
- 76. Batch monitoring should track completeness
- 77. Define recovery before production
- 78. Test failure scenarios
- 79. Streaming is not always “real-time”
- 80. Choose the simplest architecture that meets the requirement
- What the result means
- What to look for
- Streaming job is running but data is stale
- Batch finishes but data is incomplete
- Streaming state keeps growing
- Backfill creates duplicates
- Pipeline repeatedly falls behind during peaks
- Frequent batches are meeting the SLA comfortably
- Production notes
- Latency is a business requirement, not a technology feature
- Replayability is critical
- Streaming increases operational state
- Downstream systems may constrain the choice
- Cost and complexity should be measured together
- Hybrid is normal
- Common mistakes
- 1. Choosing streaming because the source is Kafka/Event Hub
- 2. Calling hourly batch “not real-time enough” without an SLA
- 3. Assuming streaming eliminates duplicates
- 4. Assuming streaming gives snapshot consistency
- 5. Advancing batch watermarks before successful writes
- 6. Using timestamps without tie-breakers
- 7. Ignoring late/out-of-order events
- 8. Letting streaming state grow without bounds
- 9. Treating streaming checkpoints as disposable temp files
- 10. Building streaming without a replay/backfill plan
- Next step
- Related deep dives
Problem
You need to move or transform data, but the architecture decision is unclear:
Should this run every few minutes or hours as a batch?
Or should events be processed continuously as they arrive?
The wrong choice can create unnecessary:
- latency
- operational complexity
- cost
- duplicate processing
- state-management problems
- recovery difficulty
- pressure on downstream systems
The goal is to choose the simplest processing model that meets the real business latency and correctness requirements.
Quick answer
Use this decision model:
Minutes or hours of latency are acceptable
→ batch is usually simpler
Data must be processed continuously with low latency
→ streaming may be appropriate
Need easy replay and deterministic recomputation
→ batch is usually easier
Need immediate alerts, fraud detection, telemetry, or live operational state
→ streaming may be justified
Need both historical correctness and low latency
→ combine streaming ingestion with durable storage and replayable batch/backfill paths
A useful rule:
Do not choose streaming because the source emits events.
Choose streaming because the business needs continuous processing.
Code / commands
1. Batch processing pattern
Typical flow:
source
→ land raw data
→ process a bounded dataset
→ write curated output
→ checkpoint completion
Example pseudo-workflow:
start = get_last_successful_watermark()
end = get_current_cutoff()
df = read_source(
start_time=start,
end_time=end,
)
clean = transform(df)
validate(clean)
write_output(clean)
save_watermark(end)
The important property is that the input range is bounded.
2. Streaming processing pattern
Typical flow:
event source
→ continuously ingest
→ process records/micro-batches
→ maintain state/checkpoints
→ write output continuously
Conceptual Spark Structured Streaming example:
events = (
spark.readStream
.format("delta")
.table("bronze_events")
)
clean = (
events
.filter("event_type IS NOT NULL")
.select(
"event_id",
"event_time",
"event_type",
"customer_id",
)
)
query = (
clean.writeStream
.format("delta")
.option("checkpointLocation", "/checkpoints/silver_events")
.outputMode("append")
.toTable("silver_events")
)
Streaming requires durable checkpoint/state design, not only a continuously running loop.
3. Micro-batch is still streaming
Many production streaming engines process events as repeated small batches.
Conceptually:
events arrive continuously
→ every short interval, engine processes new data
→ checkpoint advances
This is still a streaming architecture because the pipeline is continuously incremental.
4. Batch is not necessarily “once per day”
Batch can run:
every 5 minutes
every 15 minutes
hourly
nightly
on file arrival
on demand
If five-minute latency satisfies the requirement, a small frequent batch may be simpler than a full streaming platform.
5. Define the latency requirement before choosing architecture
Write the requirement explicitly:
New transaction visible in analytics within:
15 minutes
or:
Fraud alert generated within:
5 seconds
Do not start with:
We should use Kafka/Spark Streaming because it is modern.
Architecture should follow the business SLA.
6. Bounded batch window
A robust batch usually defines:
[start, end)
Example:
WHERE event_time >= @start_time
AND event_time < @end_time
Half-open windows avoid overlap between adjacent batches when timestamps are the incremental boundary.
7. Add a tie-breaker when timestamps are not unique
Bad incremental state:
last_processed_timestamp
if thousands of rows can share the same timestamp.
Better state:
last_processed_timestamp
+
last_processed_id
or use a source-provided sequence/version.
8. Batch watermark example
Checkpoint:
{
"lastSuccessfulEnd": "2026-10-04T14:00:00Z"
}
Next batch:
start = 2026-10-04T14:00:00Z
end = 2026-10-04T15:00:00Z
Advance the watermark only after durable processing succeeds.
9. Never advance a batch watermark before success
Bad:
save watermark
→ process data
→ pipeline fails
The failed interval may be skipped on retry.
Safer:
read bounded input
→ transform
→ validate
→ write durably
→ persist watermark
10. Streaming checkpoint example
A streaming engine checkpoint typically tracks:
source offsets
progress
state-store metadata
sink commit information
Checkpoint storage should be durable and isolated per logical stream.
Do not point unrelated streaming jobs at the same checkpoint location.
11. Keep source events durable when possible
Useful pattern:
event bus / source
→ durable Bronze/raw layer
→ streaming transformation
Durable raw data makes it possible to:
- replay
- backfill
- debug
- rebuild downstream state
Streaming without replayability makes production recovery much harder.
12. Separate ingestion from business transformation
A strong architecture often uses:
Source events
→ durable ingestion
→ transformation
→ serving
Do not put every business rule into the ingestion layer if it prevents replaying raw source data later.
13. Streaming does not mean “no storage”
Streaming systems still need durable state for:
- offsets
- checkpoints
- deduplication
- windows
- joins
- aggregation state
- replay/backfill
Continuous computation increases the importance of state management.
14. Event-time processing
Streaming logic should often use:
event_time
rather than only:
processing_time
Example:
event_time
→ when the business event actually happened
processing_time
→ when the engine received/processed it
They can differ significantly.
15. Late-arriving data
Example:
event happened at 10:00
event arrived at 10:12
A streaming system must decide:
accept late event?
update prior result?
drop it?
route to quarantine?
This is a business correctness decision.
16. Watermarks in streaming are not the same as batch checkpoints
In event-time streaming, a watermark often expresses how far behind event time the engine expects late data may arrive.
Conceptually:
max event time seen
-
allowed lateness
The engine can use that boundary to clean old state.
Exact semantics depend on the streaming engine.
17. Do not choose an arbitrary lateness threshold
Use source and business evidence.
Example questions:
How late can events arrive?
How late are they in real production?
Can old windows be updated?
What downstream consumers expect corrections?
Do not invent a universal five-minute or one-hour watermark.
18. Windowed streaming aggregation
Conceptual PySpark example:
from pyspark.sql import functions as F
agg = (
events
.withWatermark("event_time", "30 minutes")
.groupBy(
F.window("event_time", "5 minutes"),
"merchant_id",
)
.agg(
F.count("*").alias("event_count")
)
)
Use only when a 30-minute lateness policy is actually appropriate to the source and business.
The number above is illustrative, not a universal recommendation.
19. Streaming joins require state
Joining two unbounded streams can require maintaining historical state.
Example:
orders stream
JOIN
payments stream
Without bounded time/state rules, state can grow indefinitely.
Use:
- event-time constraints
- watermarks
- bounded matching windows
where the engine requires/supports them.
20. Static-to-stream joins are simpler
Common pattern:
event stream
JOIN
small reference/dimension table
This is often simpler than stream-to-stream joins.
But decide how reference changes become visible:
- reload
- refresh
- broadcast update
- versioned table
21. Deduplicate streaming events by durable event ID
Conceptually:
event_id = globally stable source event identifier
Then deduplication can reason about replay/retry.
Avoid generating a new random event ID every time an event is reprocessed.
22. Idempotent sinks reduce retry risk
A pipeline is easier to recover when writing the same logical record twice does not corrupt state.
Examples:
MERGE by business/event key
upsert by stable ID
transactional append with dedupe downstream
Do not depend on “the event will only arrive once.”
23. Exactly-once claims need careful scope
“Exactly once” may refer to:
source offset processing
state update
sink commit
business side effect
These are not automatically identical.
For example:
stream writes row exactly once
does not mean:
email sent exactly once
Design external side effects separately.
24. At-least-once delivery is common
A retry may deliver/process the same event again.
Downstream logic should tolerate this where possible.
Use stable identifiers and idempotency.
25. Batch reruns also need idempotency
Batch is not automatically safe.
Bad rerun:
INSERT entire interval again
can duplicate output.
Safer options:
overwrite bounded partition
MERGE by key
delete/rebuild controlled interval
append with deterministic batch/event IDs
Choose according to table semantics.
26. Full reload batch
Pattern:
read complete source
→ rebuild complete target
Useful when:
- dataset is small enough
- logic changes often
- simple correctness matters more than incremental optimization
Do not implement complex CDC if a safe full rebuild is inexpensive.
27. Incremental batch
Pattern:
watermark
→ read only changed/new data
→ merge into target
Useful when full reload becomes too expensive.
Incremental design adds state and recovery complexity.
28. Change Data Capture can feed either model
CDC does not automatically mean streaming.
CDC records can be processed:
continuously
or:
in periodic batches
Choose consumption mode based on latency requirement.
29. Event source does not require streaming consumer
Example:
Kafka / Event Hub / Kinesis source
can still be consumed into durable storage and processed periodically.
Do not let source technology dictate downstream architecture automatically.
30. Files do not require batch-only architecture
File-arrival events can trigger near-real-time processing.
The architecture can be:
file arrives
→ event trigger
→ bounded processing
This is event-driven batch, not necessarily continuous streaming.
31. Use trigger-based batch when latency allows it
Example:
new file lands
→ orchestrator triggers pipeline
→ process only that file/batch
This can deliver low latency without continuously running a streaming engine.
32. Streaming is useful for operational decisions
Good candidates:
fraud detection
device telemetry alerts
security events
live personalization
real-time operational dashboards
inventory signals
These use cases lose value when processed too late.
33. Batch is often enough for analytics
Good candidates:
daily finance reports
hourly business aggregates
warehouse transformations
regulatory extracts
historical model features
backfills
If the consumer does not need immediate data, batch usually reduces operational burden.
34. Do not use streaming just for dashboards without a latency requirement
Ask:
Does this dashboard truly need 2-second freshness?
If 5–15 minute freshness is acceptable, frequent incremental batch may be simpler and cheaper.
35. Batch is easier to replay
Replay pattern:
choose historical interval
→ rerun transformation
→ compare output
This is valuable when:
- fixing defects
- changing business logic
- rebuilding history
Streaming systems should preserve equivalent replay capability if historical correction matters.
36. Backfill must coexist with streaming
Common production requirement:
live stream processes today
while
backfill repairs last month
Design for:
- write conflicts
- duplicate keys
- partition overlap
- downstream updates
- resource contention
Do not assume backfills will never happen.
37. Use separate backfill controls
Example:
live checkpoint
!=
backfill checkpoint
A historical backfill should not move the live stream’s checkpoint backward or overwrite its state accidentally.
38. Batch failures are naturally bounded
If hourly batch 14:00–15:00 fails:
rerun 14:00–15:00
The failure scope is easy to reason about.
Streaming recovery may involve:
- offsets
- state
- checkpoint compatibility
- event replay
- late data
39. Streaming failures can accumulate lag
If processing stops:
events continue arriving
→ backlog grows
Recovery requires enough capacity to:
process new arrivals
+
catch up backlog
Monitor consumer lag/backlog explicitly.
40. Lag is a core streaming health metric
Useful concept:
latest source position
-
consumer processed position
or time-based equivalent.
A running streaming job can still be unhealthy if lag keeps increasing.
41. Throughput must exceed sustained arrival rate
If source produces:
10,000 events/sec
and the pipeline sustainably processes:
8,000 events/sec
lag grows indefinitely.
Do not size streaming only for average traffic if real workloads have sustained peaks.
42. Burst handling matters
A stream may normally receive little traffic but occasionally spike.
Options include:
- buffering in durable broker/storage
- autoscaling
- partition scaling
- bounded backpressure
- catch-up capacity
43. Batch has peak-window pressure too
A nightly batch may compete with:
- warehouse users
- backups
- maintenance
- downstream reports
Batch simplifies processing state, but scheduling/resource contention still matters.
44. Streaming can move load downstream continuously
A downstream database that handled:
one bulk hourly merge
may behave differently under:
continuous small updates
Evaluate downstream write patterns.
Streaming upstream does not guarantee downstream systems are suited for constant writes.
45. Micro-batching can protect downstream systems
Instead of writing every event individually:
buffer small interval
→ write batch
This can reduce per-request/transaction overhead.
Choose interval based on latency and sink capacity.
46. Avoid one external API call per event at high volume
Pattern:
event
→ HTTP request
→ event
→ HTTP request
can overload downstream services.
Consider:
- batching
- queueing
- rate limiting
- async workers
- deduplicated side-effect processing
47. Streaming state can become large
Stateful operations include:
- windows
- deduplication
- stream-stream joins
- sessionization
If state is not bounded, memory/storage pressure can grow continuously.
Use appropriate retention/watermark semantics.
48. Batch also has state, but it is often explicit
Examples:
watermark table
processed file manifest
batch ID
high-water mark
Persist state transactionally where possible.
49. Track batch execution metadata
Useful fields:
batch_id
start_watermark
end_watermark
start_time
end_time
row_count
status
source_version
code_version
This helps replay and incident analysis.
50. Track streaming execution metadata
Useful signals:
checkpoint
source offset
processed rows/sec
input rows/sec
lag
state size
batch duration
failed batch
sink commit
Exact metrics depend on the platform.
51. Prefer deterministic transformations
Given the same input:
same code
+
same reference data/version
→ same output
This makes replay safer.
Avoid hidden dependencies on current time or mutable lookup state unless intentional.
52. Record processing time separately from event time
Useful columns:
event_time
ingestion_time
processed_time
These help answer:
When did event happen?
When did platform receive it?
When did transformation complete?
53. Separate business freshness from pipeline latency
Example:
pipeline processed event in 5 seconds
but the source itself delivered it 20 minutes late.
Your pipeline latency is low, but business data freshness is not.
Monitor both.
54. Event ordering may not be guaranteed globally
Distributed sources may preserve order only:
- within a partition
- within a key
- within a producer session
Do not assume global ordering unless the source guarantees it.
55. Partition key affects streaming order and scale
A partition key controls:
distribution
parallelism
per-key ordering
hot partitions
A poor key can create skew.
56. Avoid one hot key
Example:
partition_key = "all_events"
forces all events through one partition/key.
Choose a key aligned with:
- required ordering
- workload distribution
- consumer semantics
57. Batch partitioning also affects performance
A batch can be inefficient if all work lands in:
- one file
- one partition
- one date bucket
- one tenant
Use partitioning based on actual access and processing patterns.
58. Streaming schemas evolve too
Events may add/change fields over time.
Design:
- schema compatibility
- defaults
- nullable additions
- versioning
- quarantine for invalid events
Do not assume event contracts stay static because they are real-time.
59. Batch schema drift is easier to quarantine
A failed file or batch can be isolated without stopping all future processing if orchestration is designed correctly.
Streaming schema errors can stop or poison continuous processing if not handled.
60. Use dead-letter/quarantine patterns carefully
For invalid records:
valid
→ normal path
invalid
→ quarantine with reason/context
Do not silently drop bad records if they matter to business correctness.
61. Do not let one bad record stop the whole stream unnecessarily
Where platform semantics allow, isolate malformed events.
But do not convert serious schema/contract failures into silent data loss.
62. Streaming costs are not only compute
Consider:
event broker
state/checkpoint storage
continuous compute
network
monitoring
operational support
replay storage
63. Batch costs are not automatically lower
A poorly designed full daily scan of massive data can be more expensive than incremental streaming.
Measure actual workload.
64. Cost follows work performed
Useful question:
How much data are we reading, shuffling, storing, and writing?
not simply:
Is it batch or streaming?
65. Batch and streaming can share transformations
A strong design separates:
business transformation logic
from
execution mode
Where practical, the same logic can be reused for:
- live incremental processing
- historical replay
- testing
66. Build a replay path for streaming logic
Example:
Bronze Delta table
→ batch read historical range
→ same Silver transformation
This helps:
- recover from defects
- rebuild state
- validate logic changes
67. Lambda architecture adds duplicate logic
Classic lambda architecture uses:
batch path
+
speed/streaming path
This can produce operational duplication and inconsistent logic.
Use only when justified.
Modern designs often prefer one durable event/raw layer plus reusable incremental/batch processing.
68. Kappa-style thinking emphasizes replayable streams
Conceptually:
durable event log
→ stream processing
→ replay when needed
The practical requirement is not the architecture label.
The important capability is replayable source history and deterministic processing.
69. Avoid architecture labels as design goals
Do not choose:
Lambda
Kappa
real-time
event-driven
because the label sounds modern.
Define:
latency
correctness
replay
cost
failure behavior
operational ownership
first.
70. Use hybrid architecture when requirements differ
Example:
streaming
→ operational alerts
batch
→ authoritative daily finance totals
The same source can support multiple processing paths when business needs differ.
71. Use streaming for early signal, batch for reconciliation
Example:
stream
→ near-real-time dashboard
nightly batch
→ authoritative recomputation/reconciliation
This can be useful when streaming freshness matters but final correctness requires late-arriving data or authoritative source reconciliation.
72. Reconciliation should be deliberate
Compare:
streaming aggregate
vs
authoritative batch aggregate
Investigate divergence.
Do not treat batch correction as permission to ignore streaming quality.
73. Define source-of-truth semantics
Ask:
Is raw event log authoritative?
Is source database authoritative?
Is daily extract authoritative?
Recovery and reconciliation depend on this answer.
74. Data contracts matter more in streaming
Because streaming consumers react quickly, bad producer changes propagate quickly.
Use:
- schema contracts
- versioning
- compatibility testing
- staged rollout
75. Streaming requires stronger operational monitoring
At minimum watch:
job alive?
input rate
processing rate
lag
checkpoint progress
state growth
error rate
sink failures
A green process status alone is not enough.
76. Batch monitoring should track completeness
Watch:
expected batch arrived?
batch succeeded?
row count plausible?
watermark advanced?
duplicate batch?
late file?
A completed scheduler run does not guarantee complete data.
77. Define recovery before production
For batch:
How do we rerun one interval?
For streaming:
How do we replay from an earlier point?
What happens to state?
How do we avoid duplicate side effects?
If recovery is undefined, the architecture is incomplete.
78. Test failure scenarios
Useful tests:
worker crashes mid-processing
sink unavailable
duplicate event
late event
out-of-order event
schema change
checkpoint unavailable
backfill overlaps live data
Production resilience should be demonstrated, not assumed.
79. Streaming is not always “real-time”
Real end-to-end latency includes:
source delay
broker delay
processing delay
sink delay
consumer/cache refresh delay
A streaming engine alone does not guarantee real-time user experience.
80. Choose the simplest architecture that meets the requirement
Decision order:
1. Define required freshness
2. Define correctness/replay needs
3. Define expected volume and burst behavior
4. Define downstream capabilities
5. Define failure/recovery model
6. Estimate operational complexity/cost
7. Choose batch, streaming, or hybrid
What the result means
| Dimension | Batch | Streaming |
|---|---|---|
| Input | Bounded | Unbounded/continuous |
| Typical latency | Minutes to hours, but can be frequent | Seconds to minutes depending on platform/design |
| State | Watermarks/manifests/batch metadata | Offsets/checkpoints/state stores/watermarks |
| Replay | Usually straightforward | Requires durable source + checkpoint/state strategy |
| Failure scope | Usually bounded interval | Can create backlog/lag and state concerns |
| Late data | Next/reconciliation batch | Explicit event-time/lateness policy often needed |
| Operational complexity | Usually lower | Usually higher |
| Random backfill | Natural fit | Often handled through separate replay/backfill path |
| Live alerts | Poor fit if latency is high | Strong fit |
| Large historical recomputation | Strong fit | Usually done through batch/replay |
The core tradeoff is:
lower latency
often requires
more state + more operational complexity
What to look for
Streaming job is running but data is stale
Check:
- source lag
- input rate vs processing rate
- checkpoint progress
- sink latency
- downstream cache/serving refresh
Batch finishes but data is incomplete
Check:
- source arrival completeness
- watermark
- skipped files
- duplicate filtering
- late-arriving records
Streaming state keeps growing
Inspect:
- unbounded joins
- deduplication retention
- window definitions
- watermark/lateness policy
- high-cardinality keys
Backfill creates duplicates
Check:
- idempotent write logic
- overlapping date ranges
- live/backfill coordination
- merge keys
Pipeline repeatedly falls behind during peaks
Check sustained throughput, partition skew, downstream sink capacity, and catch-up headroom.
Frequent batches are meeting the SLA comfortably
Streaming may add complexity without enough business value.
Production notes
Latency is a business requirement, not a technology feature
Do not optimize for lower latency than consumers need.
Replayability is critical
Whether batch or streaming, preserve enough durable source/history to recover from defects and rebuild outputs.
Streaming increases operational state
Checkpoints, offsets, watermarks, and state stores become part of correctness.
Treat them as production data.
Downstream systems may constrain the choice
A continuous event source does not mean the target database/API can safely absorb one write per event.
Cost and complexity should be measured together
Streaming can reduce freshness latency while increasing:
- platform services
- monitoring
- incident complexity
- state management
Hybrid is normal
Many mature platforms use both:
streaming for freshness
batch for recomputation/reconciliation
when the requirements justify it.
Common mistakes
1. Choosing streaming because the source is Kafka/Event Hub
Source transport does not define processing requirements.
2. Calling hourly batch “not real-time enough” without an SLA
Define the actual freshness requirement first.
3. Assuming streaming eliminates duplicates
Retries and at-least-once delivery still require idempotent processing.
4. Assuming streaming gives snapshot consistency
Live data can change while it is processed.
5. Advancing batch watermarks before successful writes
This can skip failed intervals.
6. Using timestamps without tie-breakers
Rows sharing the same timestamp can be skipped during incremental processing.
7. Ignoring late/out-of-order events
Event time and processing time are different.
8. Letting streaming state grow without bounds
Windows, joins, and dedupe need retention semantics.
9. Treating streaming checkpoints as disposable temp files
They are part of recovery/correctness.
10. Building streaming without a replay/backfill plan
Eventually you will need to repair history.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.