Browse cheat sheets · Microsoft Fabric

Technology: Microsoft FabricCategory: PerformanceDifficulty: IntermediateUpdated

Fabric Session Tags

Use Microsoft Fabric session tags to reuse compatible Spark sessions across notebook activities, reduce startup overhead, and avoid unnecessary session fragmentation.

On this page
  1. Problem
  2. Quick answer
  3. Code / commands
  4. 1. Set a session tag on a Fabric pipeline Notebook activity
  5. 2. Enable High Concurrency before relying on session tags
  6. 3. Reuse one tag for related notebook activities
  7. 4. Use separate tags for unrelated workloads
  8. 5. Keep compute configuration compatible
  9. 6. Keep Environment and Spark settings aligned
  10. 7. Do not make session tags unique per pipeline run
  11. 8. Do not use one global tag for every notebook
  12. 9. Tune session density only after measuring concurrency
  13. 10. Use session reuse for short notebook chains
  14. 11. Measure startup and execution separately
  15. 12. Confirm reuse through Fabric monitoring
  16. 13. Understand REPL isolation in High Concurrency
  17. 14. Pass state explicitly between notebooks
  18. 15. Be careful with temporary views
  19. 16. Be deliberate with caching
  20. 17. Unpersist reusable data when finished
  21. 18. Do not use session tags to fix a slow Spark query
  22. 19. Avoid excessive concurrent notebooks in one shared session
  23. 20. Use different tags when workload isolation is more important
  24. 21. Understand session tag behavior as a reuse hint
  25. 22. Do not treat the tag as idempotency
  26. 23. Do not use session tags as a locking mechanism
  27. 24. Avoid depending on relative paths in High Concurrency scenarios
  28. 25. Use meaningful session-tag naming conventions
  29. 26. Document expected sharing boundaries
  30. What the result means
  31. What to look for
  32. Same tag but a new Spark session starts
  33. Pipeline startup improves but notebook runtime does not
  34. Shared-session notebooks slow each other down
  35. Results become inconsistent because notebooks depend on in-memory state
  36. A workload is short but pipeline duration is still dominated by startup
  37. Production notes
  38. Session tags require High Concurrency for pipeline notebook reuse
  39. Session sharing stays within compatibility boundaries
  40. A tag is not a guarantee
  41. Shared sessions use shared resources
  42. The default High Concurrency sharing limit is configurable
  43. Session reuse and billing are related but not identical to performance
  44. Pipeline code should remain independently correct
  45. Common mistakes
  46. 1. Setting a session tag without enabling High Concurrency
  47. 2. Assuming same tag guarantees same session
  48. 3. Giving every run a unique tag
  49. 4. Giving every notebook in the workspace the same tag
  50. 5. Sharing Python variables between notebooks implicitly
  51. 6. Using session tags to fix slow Spark transformations
  52. 7. Increasing spark.highConcurrency.max to the maximum immediately
  53. 8. Assuming shared session means serialized execution
  54. 9. Ignoring Lakehouse or Spark configuration differences
  55. Next step
  56. Related deep dives

Problem

A Fabric pipeline runs several notebook activities, and each notebook spends significant time acquiring or starting Spark compute before the real work begins.

For short notebooks, session startup can become a large part of total runtime.

Fabric session tags can help compatible notebook activities reuse an existing Spark session instead of starting a new one each time.

The important questions are:

  • When does a session tag actually enable reuse?
  • What must match between notebook activities?
  • When should the same tag be reused?
  • When should workloads use separate tags?
  • How do you tell whether session reuse is helping?
  • What problems can session tags not solve?

Quick answer

Use the same session tag for notebook activities that are intentionally part of the same compatible workload.

Example logical design:

Pipeline: Daily Sales

Notebook 1: ingest_sales
Session tag: daily-sales

Notebook 2: transform_sales
Session tag: daily-sales

Notebook 3: publish_sales
Session tag: daily-sales

Fabric can reuse an existing compatible Spark session for activities with the same tag instead of provisioning a new session.

For pipeline Notebook activities, session tags require:

Workspace settings
→ Spark settings
→ High concurrency mode
→ Enable high concurrency for pipelines running multiple notebooks

A session tag is primarily a Spark session startup and reuse optimization.

It does not make inefficient Spark transformations faster by itself.

Code / commands

1. Set a session tag on a Fabric pipeline Notebook activity

Session tags are configured on the Notebook activity in the Fabric pipeline.

Conceptually:

Notebook activity
→ Settings
→ Session tag
→ daily-sales

The tag can be any meaningful string.

Prefer stable names based on workload purpose:

daily-sales
commission-processing
customer-refresh
reporting-batch

Avoid random per-run values if your goal is session reuse.


2. Enable High Concurrency before relying on session tags

Session tags for pipeline notebook reuse depend on High Concurrency mode.

Enable the workspace option for pipeline notebooks:

Workspace settings
→ Data Engineering / Spark settings
→ High concurrency mode
→ Pipelines running multiple notebooks

If High Concurrency is disabled, setting a tag does not provide the intended shared-session behavior.


Example:

Notebook                 Session tag
-----------------------  ----------------
01_ingest_sales          daily-sales
02_transform_sales       daily-sales
03_validate_sales        daily-sales
04_publish_sales         daily-sales

Use this when the notebooks:

  • belong to one workload
  • are launched under compatible Spark settings
  • can safely share underlying Spark compute
  • benefit from reduced session startup

4. Use separate tags for unrelated workloads

Example:

Sales pipeline       -> daily-sales
Commission pipeline  -> commission
Reporting pipeline   -> reporting

Separate tags help avoid packing unrelated workloads into the same session when you want clearer workload boundaries.

A session tag is a reuse hint, not a security boundary.


5. Keep compute configuration compatible

Session reuse requires compatible Spark context.

For notebook High Concurrency, important compatibility conditions include:

  • same user boundary
  • same default Lakehouse configuration
  • same Spark compute configuration

If these differ, Fabric can create a separate Spark session even when the tag is the same.

Do not assume:

same tag = guaranteed same Spark session

6. Keep Environment and Spark settings aligned

If related notebooks are expected to share a session, avoid unnecessary differences in:

  • Spark pool configuration
  • Environment
  • Spark properties
  • default Lakehouse

A configuration difference can prevent session reuse.

Treat common session configuration as part of the workload contract.


7. Do not make session tags unique per pipeline run

This defeats reuse:

daily-sales-20261004-170001
daily-sales-20261004-171501
daily-sales-20261004-173002

Prefer:

daily-sales

when repeated activities should reuse compatible Spark compute.

Use run-specific tags only when deliberate isolation is more important than reuse.


8. Do not use one global tag for every notebook

Bad design:

session tag = shared

for every Data Engineering pipeline in the workspace.

This can create unnecessary contention and makes workload behavior harder to reason about.

Use tags that reflect real workload grouping.


9. Tune session density only after measuring concurrency

Fabric exposes the High Concurrency session-sharing limit through the Environment Spark property:

spark.highConcurrency.max

Example:

spark.highConcurrency.max = 10

This controls how many compatible notebook workloads can share one High Concurrency Spark session.

The platform default is lower than the supported maximum.

Do not increase it automatically.

Higher density means more workloads share the same finite Spark resources.


10. Use session reuse for short notebook chains

Session tags are especially useful when individual notebooks are short.

Example:

Notebook A Spark work: 25 sec
Notebook B Spark work: 35 sec
Notebook C Spark work: 20 sec

If each notebook also requires significant Spark session startup, session reuse can materially reduce end-to-end pipeline latency.

For a notebook that runs for hours, startup may be a much smaller part of total runtime.


11. Measure startup and execution separately

Track:

activity start
Spark session available
first Spark action starts
Spark action completes
notebook completes

This lets you distinguish:

session overhead

from:

actual Spark execution

Do not attribute all notebook duration to Spark query performance.


12. Confirm reuse through Fabric monitoring

When troubleshooting session reuse, inspect Fabric monitoring and Spark session/job information.

Look for whether related notebook activities:

  • attach to the same underlying Spark application/session
  • avoid repeated full startup delays
  • still execute in isolated notebook contexts

If multiple new Spark sessions appear despite the same tag, inspect compatibility differences.


13. Understand REPL isolation in High Concurrency

High Concurrency can host multiple notebook workloads inside one Spark application.

Each workload runs in its own REPL context.

This means you should not design pipelines assuming Python variables from one notebook automatically exist in another.

For example, do not rely on:

# Notebook A
my_value = 123

being available in:

# Notebook B
print(my_value)

Use explicit persistence or parameters instead.


14. Pass state explicitly between notebooks

Use supported mechanisms such as:

  • pipeline parameters
  • notebook parameters
  • Delta tables
  • Lakehouse files
  • control tables
  • persisted metadata

Example:

# Notebook A
result.write.format("delta").mode("overwrite").saveAsTable(
    "control.daily_sales_stage"
)

Then:

# Notebook B
result = spark.table(
    "control.daily_sales_stage"
)

Session reuse should improve startup, not create hidden cross-notebook state dependencies.


15. Be careful with temporary views

A temporary view is session-scoped:

df.createOrReplaceTempView(
    "sales_stage"
)

Even if workloads share an underlying Spark application, isolated REPL contexts mean temporary notebook state should not be treated as a reliable inter-notebook contract.

Persist data when another notebook depends on it.


16. Be deliberate with caching

A shared Spark session does not mean every notebook should cache aggressively.

Example:

df.cache()
df.count()

Caching consumes shared executor memory.

In a High Concurrency session, excessive caching from several workloads can create contention.

Cache only data that is expensive and reused enough to justify the memory cost.


17. Unpersist reusable data when finished

df.unpersist()

Do this when cached data is no longer needed.

Shared sessions make memory hygiene more important because several notebook workloads can compete for the same underlying Spark resources.


18. Do not use session tags to fix a slow Spark query

If a notebook spends:

20 sec starting session
15 min performing a shuffle-heavy join

session reuse can reduce the 20-second startup portion.

It will not fix the 15-minute join.

Investigate:

  • scan size
  • join strategy
  • shuffle
  • skew
  • partitions
  • driver-side operations

separately.


19. Avoid excessive concurrent notebooks in one shared session

If many notebooks share one session, they compete for:

  • executors
  • CPU
  • memory
  • shuffle resources

More session reuse is not automatically better.

Measure:

pipeline duration
session acquisition time
individual notebook duration
queueing
Spark stage duration
capacity consumption

Tune for throughput rather than maximum density.


20. Use different tags when workload isolation is more important

Separate tags can be appropriate when notebooks:

  • have very different compute profiles
  • run large competing shuffles
  • require different Spark configuration
  • should not contend heavily
  • belong to different operational domains

Example:

heavy-monthly-report
realtime-enrichment
daily-ingestion

Session reuse is an optimization, not a requirement.


21. Understand session tag behavior as a reuse hint

A session tag tells Fabric to look for an existing compatible session.

If one exists and can accept the workload, Fabric can reuse it.

If not, Fabric can create a new session.

Therefore:

same session tag

does not mean:

one guaranteed physical Spark session forever

Capacity, compatibility, timing, and available session slots still matter.


22. Do not treat the tag as idempotency

A session tag does not mean:

run this notebook only once

or:

return the same execution

It is about Spark session placement/reuse.

Pipeline idempotency must be designed separately through:

  • load keys
  • checkpoints
  • merge logic
  • execution logs
  • control tables

23. Do not use session tags as a locking mechanism

A session tag does not serialize workloads.

Several notebook activities may run concurrently in the same High Concurrency Spark application.

If notebooks modify the same Delta table, design write concurrency explicitly.

Use appropriate:

  • partitioning
  • merge logic
  • transaction patterns
  • retry/error handling

Session sharing does not remove Delta concurrency concerns.


24. Avoid depending on relative paths in High Concurrency scenarios

Fabric documents limitations around relative paths in High Concurrency notebook sessions.

Prefer explicit Lakehouse/OneLake-aware paths or table references where appropriate.

Example:

df = spark.table(
    "sales_curated"
)

rather than relying on fragile relative filesystem assumptions.


25. Use meaningful session-tag naming conventions

Good:

sales-daily
commission-daily
reporting-refresh
customer-dim-build

Avoid:

test
spark
session1
abc

A useful tag should tell an operator what workload the shared session belongs to.


26. Document expected sharing boundaries

For each pipeline, document:

Session tag:
daily-sales

Expected notebooks:
- ingest_sales
- transform_sales
- publish_sales

Default Lakehouse:
lh_sales

Environment:
env_sales_prod

High Concurrency:
enabled

This makes it much easier to troubleshoot why one activity unexpectedly starts a new session.

What the result means

Concept Meaning
Session tag A hint used to reuse/pack compatible Spark notebook workloads into an existing session.
High Concurrency Fabric mode that allows compatible notebook workloads to share one Spark application.
REPL Isolated notebook execution context inside a shared Spark application.
Session reuse Avoiding a new Spark session startup when a compatible session already exists.
spark.highConcurrency.max Configurable limit controlling notebook density in a High Concurrency session.
Same tag Makes reuse possible; does not guarantee it.
Different tag Encourages separate workload grouping/session placement.

What to look for

Same tag but a new Spark session starts

Check:

  • High Concurrency enabled?
  • same user boundary?
  • same default Lakehouse?
  • same Spark compute configuration?
  • compatible Environment/config?
  • session still active?
  • available High Concurrency slots?

The tag alone is not sufficient.

Pipeline startup improves but notebook runtime does not

This is expected if the bottleneck is the Spark workload itself.

Session tags reduce startup overhead, not scan/shuffle/compute cost.

Shared-session notebooks slow each other down

You may have too much concurrency for the available resources.

Consider:

  • lowering simultaneous notebook count
  • reducing spark.highConcurrency.max
  • using separate tags
  • changing orchestration
  • increasing appropriate compute only after code-level review

Results become inconsistent because notebooks depend on in-memory state

This is an architecture problem.

Persist or pass state explicitly.

Do not use shared sessions as a substitute for a data contract.

A workload is short but pipeline duration is still dominated by startup

This is the strongest use case for session tags and High Concurrency.

Production notes

Session tags require High Concurrency for pipeline notebook reuse

For Fabric pipeline Notebook activities, the relevant High Concurrency workspace setting must be enabled.

Without it, the session-tag design will not deliver the intended session-sharing behavior.

Session sharing stays within compatibility boundaries

Fabric notebook High Concurrency requires compatible session context.

Important boundaries include:

  • user
  • default Lakehouse
  • Spark compute settings

Configuration drift can fragment sessions.

A tag is not a guarantee

Treat the tag as a placement/reuse hint.

Fabric may still create a new Spark session if no compatible reusable session is available.

Shared sessions use shared resources

Notebook contexts are isolated, but executors and Spark application resources are shared.

Heavy notebooks can affect each other’s runtime.

The default High Concurrency sharing limit is configurable

Fabric supports changing notebook density through:

spark.highConcurrency.max

The default is five, and Fabric supports increasing the value up to the documented platform maximum.

Higher is not automatically better.

Fabric High Concurrency can improve resource utilization and reduce repeated session startup.

Do not optimize exclusively for session density if it makes execution slower through resource contention.

Pipeline code should remain independently correct

A notebook should still be correct if Fabric gives it a fresh Spark session.

Session reuse should be an optimization layer, not a correctness dependency.

Common mistakes

1. Setting a session tag without enabling High Concurrency

The expected pipeline session reuse will not occur.

2. Assuming same tag guarantees same session

Compatibility and available capacity still determine whether reuse happens.

3. Giving every run a unique tag

This prevents intentional reuse.

4. Giving every notebook in the workspace the same tag

This creates poor workload boundaries and can increase contention.

5. Sharing Python variables between notebooks implicitly

REPL contexts are isolated.

Persist or pass state explicitly.

6. Using session tags to fix slow Spark transformations

Tags help startup/session reuse, not inefficient Spark execution.

7. Increasing spark.highConcurrency.max to the maximum immediately

Higher concurrency can increase CPU/memory/shuffle contention.

Measure first.

8. Assuming shared session means serialized execution

High Concurrency is specifically designed for concurrent workloads.

Design data writes accordingly.

9. Ignoring Lakehouse or Spark configuration differences

Those differences can cause separate sessions even when the tag matches.

Next step

Use the observed behavior to choose the next action.

  • Every notebook starts a new Spark session → verify High Concurrency and session-tag configuration.
  • Same tag does not reuse session → compare Lakehouse, user, Environment, and Spark compute settings.
  • Startup dominates pipeline duration → group compatible short notebooks under a stable session tag.
  • Shared notebooks compete heavily → reduce concurrency, lower session density, or separate workload tags.
  • Spark execution remains slow after reuse → move to Fabric Notebook Performance troubleshooting.
  • Notebooks depend on temporary in-memory state → replace that dependency with persisted data or parameters.
  • Concurrent writes conflict → review Delta write/concurrency patterns.
  • You need to prove reuse → inspect Fabric monitoring and Spark application/session behavior.

← All cheat sheets