Browse cheat sheets · Microsoft Fabric
Fabric Session Tags
Use Microsoft Fabric session tags to reuse compatible Spark sessions across notebook activities, reduce startup overhead, and avoid unnecessary session fragmentation.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Set a session tag on a Fabric pipeline Notebook activity
- 2. Enable High Concurrency before relying on session tags
- 3. Reuse one tag for related notebook activities
- 4. Use separate tags for unrelated workloads
- 5. Keep compute configuration compatible
- 6. Keep Environment and Spark settings aligned
- 7. Do not make session tags unique per pipeline run
- 8. Do not use one global tag for every notebook
- 9. Tune session density only after measuring concurrency
- 10. Use session reuse for short notebook chains
- 11. Measure startup and execution separately
- 12. Confirm reuse through Fabric monitoring
- 13. Understand REPL isolation in High Concurrency
- 14. Pass state explicitly between notebooks
- 15. Be careful with temporary views
- 16. Be deliberate with caching
- 17. Unpersist reusable data when finished
- 18. Do not use session tags to fix a slow Spark query
- 19. Avoid excessive concurrent notebooks in one shared session
- 20. Use different tags when workload isolation is more important
- 21. Understand session tag behavior as a reuse hint
- 22. Do not treat the tag as idempotency
- 23. Do not use session tags as a locking mechanism
- 24. Avoid depending on relative paths in High Concurrency scenarios
- 25. Use meaningful session-tag naming conventions
- 26. Document expected sharing boundaries
- What the result means
- What to look for
- Same tag but a new Spark session starts
- Pipeline startup improves but notebook runtime does not
- Shared-session notebooks slow each other down
- Results become inconsistent because notebooks depend on in-memory state
- A workload is short but pipeline duration is still dominated by startup
- Production notes
- Session tags require High Concurrency for pipeline notebook reuse
- Session sharing stays within compatibility boundaries
- A tag is not a guarantee
- Shared sessions use shared resources
- The default High Concurrency sharing limit is configurable
- Session reuse and billing are related but not identical to performance
- Pipeline code should remain independently correct
- Common mistakes
- 1. Setting a session tag without enabling High Concurrency
- 2. Assuming same tag guarantees same session
- 3. Giving every run a unique tag
- 4. Giving every notebook in the workspace the same tag
- 5. Sharing Python variables between notebooks implicitly
- 6. Using session tags to fix slow Spark transformations
- 7. Increasing spark.highConcurrency.max to the maximum immediately
- 8. Assuming shared session means serialized execution
- 9. Ignoring Lakehouse or Spark configuration differences
- Next step
- Related deep dives
Problem
A Fabric pipeline runs several notebook activities, and each notebook spends significant time acquiring or starting Spark compute before the real work begins.
For short notebooks, session startup can become a large part of total runtime.
Fabric session tags can help compatible notebook activities reuse an existing Spark session instead of starting a new one each time.
The important questions are:
- When does a session tag actually enable reuse?
- What must match between notebook activities?
- When should the same tag be reused?
- When should workloads use separate tags?
- How do you tell whether session reuse is helping?
- What problems can session tags not solve?
Quick answer
Use the same session tag for notebook activities that are intentionally part of the same compatible workload.
Example logical design:
Pipeline: Daily Sales
Notebook 1: ingest_sales
Session tag: daily-sales
Notebook 2: transform_sales
Session tag: daily-sales
Notebook 3: publish_sales
Session tag: daily-sales
Fabric can reuse an existing compatible Spark session for activities with the same tag instead of provisioning a new session.
For pipeline Notebook activities, session tags require:
Workspace settings
→ Spark settings
→ High concurrency mode
→ Enable high concurrency for pipelines running multiple notebooks
A session tag is primarily a Spark session startup and reuse optimization.
It does not make inefficient Spark transformations faster by itself.
Code / commands
1. Set a session tag on a Fabric pipeline Notebook activity
Session tags are configured on the Notebook activity in the Fabric pipeline.
Conceptually:
Notebook activity
→ Settings
→ Session tag
→ daily-sales
The tag can be any meaningful string.
Prefer stable names based on workload purpose:
daily-sales
commission-processing
customer-refresh
reporting-batch
Avoid random per-run values if your goal is session reuse.
2. Enable High Concurrency before relying on session tags
Session tags for pipeline notebook reuse depend on High Concurrency mode.
Enable the workspace option for pipeline notebooks:
Workspace settings
→ Data Engineering / Spark settings
→ High concurrency mode
→ Pipelines running multiple notebooks
If High Concurrency is disabled, setting a tag does not provide the intended shared-session behavior.
3. Reuse one tag for related notebook activities
Example:
Notebook Session tag
----------------------- ----------------
01_ingest_sales daily-sales
02_transform_sales daily-sales
03_validate_sales daily-sales
04_publish_sales daily-sales
Use this when the notebooks:
- belong to one workload
- are launched under compatible Spark settings
- can safely share underlying Spark compute
- benefit from reduced session startup
4. Use separate tags for unrelated workloads
Example:
Sales pipeline -> daily-sales
Commission pipeline -> commission
Reporting pipeline -> reporting
Separate tags help avoid packing unrelated workloads into the same session when you want clearer workload boundaries.
A session tag is a reuse hint, not a security boundary.
5. Keep compute configuration compatible
Session reuse requires compatible Spark context.
For notebook High Concurrency, important compatibility conditions include:
- same user boundary
- same default Lakehouse configuration
- same Spark compute configuration
If these differ, Fabric can create a separate Spark session even when the tag is the same.
Do not assume:
same tag = guaranteed same Spark session
6. Keep Environment and Spark settings aligned
If related notebooks are expected to share a session, avoid unnecessary differences in:
- Spark pool configuration
- Environment
- Spark properties
- default Lakehouse
A configuration difference can prevent session reuse.
Treat common session configuration as part of the workload contract.
7. Do not make session tags unique per pipeline run
This defeats reuse:
daily-sales-20261004-170001
daily-sales-20261004-171501
daily-sales-20261004-173002
Prefer:
daily-sales
when repeated activities should reuse compatible Spark compute.
Use run-specific tags only when deliberate isolation is more important than reuse.
8. Do not use one global tag for every notebook
Bad design:
session tag = shared
for every Data Engineering pipeline in the workspace.
This can create unnecessary contention and makes workload behavior harder to reason about.
Use tags that reflect real workload grouping.
9. Tune session density only after measuring concurrency
Fabric exposes the High Concurrency session-sharing limit through the Environment Spark property:
spark.highConcurrency.max
Example:
spark.highConcurrency.max = 10
This controls how many compatible notebook workloads can share one High Concurrency Spark session.
The platform default is lower than the supported maximum.
Do not increase it automatically.
Higher density means more workloads share the same finite Spark resources.
10. Use session reuse for short notebook chains
Session tags are especially useful when individual notebooks are short.
Example:
Notebook A Spark work: 25 sec
Notebook B Spark work: 35 sec
Notebook C Spark work: 20 sec
If each notebook also requires significant Spark session startup, session reuse can materially reduce end-to-end pipeline latency.
For a notebook that runs for hours, startup may be a much smaller part of total runtime.
11. Measure startup and execution separately
Track:
activity start
Spark session available
first Spark action starts
Spark action completes
notebook completes
This lets you distinguish:
session overhead
from:
actual Spark execution
Do not attribute all notebook duration to Spark query performance.
12. Confirm reuse through Fabric monitoring
When troubleshooting session reuse, inspect Fabric monitoring and Spark session/job information.
Look for whether related notebook activities:
- attach to the same underlying Spark application/session
- avoid repeated full startup delays
- still execute in isolated notebook contexts
If multiple new Spark sessions appear despite the same tag, inspect compatibility differences.
13. Understand REPL isolation in High Concurrency
High Concurrency can host multiple notebook workloads inside one Spark application.
Each workload runs in its own REPL context.
This means you should not design pipelines assuming Python variables from one notebook automatically exist in another.
For example, do not rely on:
# Notebook A
my_value = 123
being available in:
# Notebook B
print(my_value)
Use explicit persistence or parameters instead.
14. Pass state explicitly between notebooks
Use supported mechanisms such as:
- pipeline parameters
- notebook parameters
- Delta tables
- Lakehouse files
- control tables
- persisted metadata
Example:
# Notebook A
result.write.format("delta").mode("overwrite").saveAsTable(
"control.daily_sales_stage"
)
Then:
# Notebook B
result = spark.table(
"control.daily_sales_stage"
)
Session reuse should improve startup, not create hidden cross-notebook state dependencies.
15. Be careful with temporary views
A temporary view is session-scoped:
df.createOrReplaceTempView(
"sales_stage"
)
Even if workloads share an underlying Spark application, isolated REPL contexts mean temporary notebook state should not be treated as a reliable inter-notebook contract.
Persist data when another notebook depends on it.
16. Be deliberate with caching
A shared Spark session does not mean every notebook should cache aggressively.
Example:
df.cache()
df.count()
Caching consumes shared executor memory.
In a High Concurrency session, excessive caching from several workloads can create contention.
Cache only data that is expensive and reused enough to justify the memory cost.
17. Unpersist reusable data when finished
df.unpersist()
Do this when cached data is no longer needed.
Shared sessions make memory hygiene more important because several notebook workloads can compete for the same underlying Spark resources.
18. Do not use session tags to fix a slow Spark query
If a notebook spends:
20 sec starting session
15 min performing a shuffle-heavy join
session reuse can reduce the 20-second startup portion.
It will not fix the 15-minute join.
Investigate:
- scan size
- join strategy
- shuffle
- skew
- partitions
- driver-side operations
separately.
19. Avoid excessive concurrent notebooks in one shared session
If many notebooks share one session, they compete for:
- executors
- CPU
- memory
- shuffle resources
More session reuse is not automatically better.
Measure:
pipeline duration
session acquisition time
individual notebook duration
queueing
Spark stage duration
capacity consumption
Tune for throughput rather than maximum density.
20. Use different tags when workload isolation is more important
Separate tags can be appropriate when notebooks:
- have very different compute profiles
- run large competing shuffles
- require different Spark configuration
- should not contend heavily
- belong to different operational domains
Example:
heavy-monthly-report
realtime-enrichment
daily-ingestion
Session reuse is an optimization, not a requirement.
21. Understand session tag behavior as a reuse hint
A session tag tells Fabric to look for an existing compatible session.
If one exists and can accept the workload, Fabric can reuse it.
If not, Fabric can create a new session.
Therefore:
same session tag
does not mean:
one guaranteed physical Spark session forever
Capacity, compatibility, timing, and available session slots still matter.
22. Do not treat the tag as idempotency
A session tag does not mean:
run this notebook only once
or:
return the same execution
It is about Spark session placement/reuse.
Pipeline idempotency must be designed separately through:
- load keys
- checkpoints
- merge logic
- execution logs
- control tables
23. Do not use session tags as a locking mechanism
A session tag does not serialize workloads.
Several notebook activities may run concurrently in the same High Concurrency Spark application.
If notebooks modify the same Delta table, design write concurrency explicitly.
Use appropriate:
- partitioning
- merge logic
- transaction patterns
- retry/error handling
Session sharing does not remove Delta concurrency concerns.
24. Avoid depending on relative paths in High Concurrency scenarios
Fabric documents limitations around relative paths in High Concurrency notebook sessions.
Prefer explicit Lakehouse/OneLake-aware paths or table references where appropriate.
Example:
df = spark.table(
"sales_curated"
)
rather than relying on fragile relative filesystem assumptions.
25. Use meaningful session-tag naming conventions
Good:
sales-daily
commission-daily
reporting-refresh
customer-dim-build
Avoid:
test
spark
session1
abc
A useful tag should tell an operator what workload the shared session belongs to.
26. Document expected sharing boundaries
For each pipeline, document:
Session tag:
daily-sales
Expected notebooks:
- ingest_sales
- transform_sales
- publish_sales
Default Lakehouse:
lh_sales
Environment:
env_sales_prod
High Concurrency:
enabled
This makes it much easier to troubleshoot why one activity unexpectedly starts a new session.
What the result means
| Concept | Meaning |
|---|---|
| Session tag | A hint used to reuse/pack compatible Spark notebook workloads into an existing session. |
| High Concurrency | Fabric mode that allows compatible notebook workloads to share one Spark application. |
| REPL | Isolated notebook execution context inside a shared Spark application. |
| Session reuse | Avoiding a new Spark session startup when a compatible session already exists. |
spark.highConcurrency.max |
Configurable limit controlling notebook density in a High Concurrency session. |
| Same tag | Makes reuse possible; does not guarantee it. |
| Different tag | Encourages separate workload grouping/session placement. |
What to look for
Same tag but a new Spark session starts
Check:
- High Concurrency enabled?
- same user boundary?
- same default Lakehouse?
- same Spark compute configuration?
- compatible Environment/config?
- session still active?
- available High Concurrency slots?
The tag alone is not sufficient.
Pipeline startup improves but notebook runtime does not
This is expected if the bottleneck is the Spark workload itself.
Session tags reduce startup overhead, not scan/shuffle/compute cost.
Shared-session notebooks slow each other down
You may have too much concurrency for the available resources.
Consider:
- lowering simultaneous notebook count
- reducing
spark.highConcurrency.max - using separate tags
- changing orchestration
- increasing appropriate compute only after code-level review
Results become inconsistent because notebooks depend on in-memory state
This is an architecture problem.
Persist or pass state explicitly.
Do not use shared sessions as a substitute for a data contract.
A workload is short but pipeline duration is still dominated by startup
This is the strongest use case for session tags and High Concurrency.
Production notes
Session tags require High Concurrency for pipeline notebook reuse
For Fabric pipeline Notebook activities, the relevant High Concurrency workspace setting must be enabled.
Without it, the session-tag design will not deliver the intended session-sharing behavior.
Session sharing stays within compatibility boundaries
Fabric notebook High Concurrency requires compatible session context.
Important boundaries include:
- user
- default Lakehouse
- Spark compute settings
Configuration drift can fragment sessions.
A tag is not a guarantee
Treat the tag as a placement/reuse hint.
Fabric may still create a new Spark session if no compatible reusable session is available.
Shared sessions use shared resources
Notebook contexts are isolated, but executors and Spark application resources are shared.
Heavy notebooks can affect each other’s runtime.
The default High Concurrency sharing limit is configurable
Fabric supports changing notebook density through:
spark.highConcurrency.max
The default is five, and Fabric supports increasing the value up to the documented platform maximum.
Higher is not automatically better.
Session reuse and billing are related but not identical to performance
Fabric High Concurrency can improve resource utilization and reduce repeated session startup.
Do not optimize exclusively for session density if it makes execution slower through resource contention.
Pipeline code should remain independently correct
A notebook should still be correct if Fabric gives it a fresh Spark session.
Session reuse should be an optimization layer, not a correctness dependency.
Common mistakes
1. Setting a session tag without enabling High Concurrency
The expected pipeline session reuse will not occur.
2. Assuming same tag guarantees same session
Compatibility and available capacity still determine whether reuse happens.
3. Giving every run a unique tag
This prevents intentional reuse.
4. Giving every notebook in the workspace the same tag
This creates poor workload boundaries and can increase contention.
5. Sharing Python variables between notebooks implicitly
REPL contexts are isolated.
Persist or pass state explicitly.
6. Using session tags to fix slow Spark transformations
Tags help startup/session reuse, not inefficient Spark execution.
7. Increasing spark.highConcurrency.max to the maximum immediately
Higher concurrency can increase CPU/memory/shuffle contention.
Measure first.
8. Assuming shared session means serialized execution
High Concurrency is specifically designed for concurrent workloads.
Design data writes accordingly.
9. Ignoring Lakehouse or Spark configuration differences
Those differences can cause separate sessions even when the tag matches.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.