Runbook Microsoft Fabric

Microsoft Fabric Capacity Saturation Runbook

A practical Microsoft Fabric runbook for investigating high capacity utilization, throttling, concurrency, carryforward and whether to optimize or scale.

Last reviewed
Platform
Microsoft Fabric
Who operates this
Fabric administrators · Platform engineers · Data engineers · Architects · SQL Server professionals operating Fabric

Open this runbook when

Fabric Capacity Metrics shows unusually high utilization, overage or throttling, or workloads begin slowing or being rejected during a known workload window.

Work through this page top to bottom. Throttling stages and product behaviour come from Microsoft Learn (references). The impact levels, the order of the steps and the observation windows are Javi Builds engineering judgement. Adapt them to your own change and incident process.

Trigger

Open this runbook when at least one of these is true:

  • Capacity Metrics shows utilization, overage or carryforward well above the usual level for this capacity.
  • Throttling appears in the Capacity Metrics app, or the capacity shows as At risk or Degraded in the Monitor hub.
  • Users or jobs get a capacity error, such as status CapacityLimitExceeded or “Your organization’s Fabric compute capacity has exceeded its limits”.
  • Pipelines, notebooks or reports slow down across several items during a known workload window.

Use a different procedure when:

  • One item is slow and nothing else is. Slow performance is often item design, not throttling. Start with Fabric performance troubleshooting.
  • Spark returns HTTP 430 (TooManyRequestsForCapacity). That is the Spark VCore concurrency limit, not CU throttling. Go to high-concurrency sessions, and come back here if the throttling charts are also raised.

Severity and impact

Establish impact before you act. Not every alert is an incident.

Question Where to confirm it
Is there actual throttling? Throttling charts and System events
Are operations delayed? Interactive delay trend
Are operations rejected? Interactive rejection and Background rejection trends, Impacted Items During Time Window
Are users affected? User reports, capacity errors, impacted interactive items
Is this background-only activity? Interactive and background split on the utilization chart
Is it a temporary peak above 100%? Throttling stays under 100% and carryforward clears

Read the impact from the capacity state, not from the utilization percentage:

    1. WatchOverage protectionUp to 10 minutes of future capacity used
    2. Fabric
      No throttling
    3. you
      Observe and correlate
    1. DegradedInteractive delay10 to 60 minutes used
    2. Fabric
      New interactive work waits 20 seconds
    3. you
      Reduce noncritical work
    1. MajorInteractive rejection60 minutes to 24 hours used
    2. Fabric
      New interactive work rejectedBackground work can still start
    3. you
      Mitigate now
    1. CriticalBackground rejectionMore than 24 hours used
    2. Fabric
      All new requests rejected
    3. you
      Mitigate and escalate
    1. Degraded or worseSurge protection activeAdmin threshold reached
    2. Fabric
      New background work rejected early
    3. you
      Treat as background impact
    1. Cost impactCapacity overage billedThrottling is being paid off
    2. Fabric
      No delay or rejection, 3× rate
    3. you
      Investigate as if throttled
Capacity state to operator response. Each row reads state, then what Fabric does, then your response. Future capacity is the CU already committed ahead of now. The tags are Javi Builds impact levels, not Microsoft terms.

Operations already running are allowed to finish. Throttling only affects requests made after it begins, so a healthy-looking running job does not prove the capacity is healthy.

Immediate checks

Do these in order. Capacity Metrics data usually arrives 10 to 15 minutes after the activity, so what you see is not the current minute.

  1. Read the current state. On the Health page of the preview report, open Latest, then Last 1 hour. The Monitor hub Capacities page (preview) also shows current CU %, throttling and carryforward.
  2. Check throttling, not utilization. Open the Throttling view and System events. Note the highest state reached and when it started and ended.
  3. Check for anything masking the picture. Capacity overage is on by default for new capacities, and a billed overage shows as “No interactive delay (capacity overage billed)”. Surge protection rejects background work below the 100% line, and the charts don’t show its thresholds, so read them in the Admin portal.
  4. Establish the time window. Record start, peak, end or recovery, and whether it recurs. Compare it with pipeline runs, notebooks, Warehouse activity, semantic model refreshes, Dataflows and other workloads.
  5. Read carryforward. Open Overage (Carryforward). Is cumulative overage falling, flat or growing? From a timepoint, Minutes to burndown estimates recovery if nothing else runs.
  6. Find the dominant workload. Use the item views below to rank by CU, and split interactive from background.
  7. Look for what changed. Deployments, new schedules, backfills, reruns, a SKU change, or workspaces moved onto this capacity. A halved SKU doubles CU % with no change in work, and a pause pushes all smoothed usage into one window, so compare CU(s) over time and System events before you trust a sudden jump.

Where to look in Capacity Metrics

The app has two reports. The default report is generally available and the redesigned preview report sits next to it in the navigation pane.

Question Preview report Default report
Did usage hurt capacity health? Smoothed: Last 24 Hours, CU % over time Compute page, Utilization tab
Is this a recurring pattern? Smoothed: Last 14 Days, capacity health heat map Compute page over 14 days
Is it actually throttling? Throttling view Throttling tab
Is overage building? Overage (Carryforward) Overages tab
Is paid overage hiding throttling? Capacity Overage Overage (Billed)
When did the state change? System Events System events table
What consumed it, and when did it run? Unsmoothed: Last 14 Days Timepoint page, from a selected timepoint
Which items hurt or were hurt? Approx CU Consumption in Window, Impacted Items During Time Window Matrix by item and operation
Which operations exactly? Item details page Interactive and background operation tables

Use smoothed views to judge capacity health and unsmoothed views to line consumption up with what ran. A background operation’s cost is smoothed over 24 hours, so a smoothed peak can include work that finished hours ago. Sovereign clouds don’t populate the throttling charts.

Classify the dominant workload

Decide which of these you have before you discuss resizing.

Shape What it looks like Usually means
One expensive workload One item or operation takes a disproportionate share of CU Tune that item
Many concurrent workloads Many runs overlap, sharp peak, quiet around it Stagger and limit parallelism
Normal sustained demand High across the day, no item stands out Evaluate capacity size
Unusual burst or backfill One-off spike tied to a known event Observe, then reschedule future runs

Evidence to collect

Copy this and fill it in as you go. Keep it with the incident record.

Incident window (with time zone):
Capacity SKU:
Metrics report and view used:
Typical utilization:
Peak utilization:
Highest throttling state reached:
Capacity overage billed (yes/no):
Surge protection active (yes/no):
Carryforward (falling/flat/growing):
Minutes to burndown:
Interactive or background impact:
Dominant workload and item kind:
Pipeline/notebook run IDs:
Concurrent runs at peak:
Execution durations vs normal:
Recent deployment or config change:
Backfill running (yes/no):
Mitigation attempted:
Result:

Do not include tenant IDs, credentials, connection strings, user emails or confidential data. Item names can be sensitive too, so check before you share this outside the team. Item and operation views can show user identities, so crop screenshots before you attach them.

Decision tree

Follow it from the symptom through the evidence to a pattern, and start with the action on that branch.

  1. SymptomHigh CU, throttling or slow work
  2. EvidenceThrottling, carryforward and top items
    • throttling under 100%, overage clears
      ObserveTemporary spike, no throttlingCorrelate it with the workload window. Do not scale automatically.
    • same window, day after day
      ReschedulePredictable concurrency spikeStagger work and reduce unnecessary parallelism.
    • one item leads CU
      OptimizeOne expensive operationTune that workload first.
    • high all day, throttling repeats
      ScalePersistent utilization and throttlingEvaluate a higher capacity, after checking the workload is expected.
    • cumulative overage flat or growing
      MitigateCarryforward does not burn downStrong capacity-pressure signal. Reduce load now, then evaluate capacity.
    • no workload or change explains it
      EscalateUnexpected platform-wide behaviorCheck Fabric service status, then escalate if evidence points there.
Capacity saturation decision tree. Each label above a box is the evidence that selects that branch. The tag is the action category. Patterns can overlap, so if more than one fits, handle the one with the most user impact first.

Common contributing factors

Check the ones that fit what you saw. They won’t all apply.

Factor Quick check
Too much concurrency Count overlapping runs at the peak in your execution log
Expensive notebooks A notebook leads CU in the item view, or its duration jumped
Repeated Spark session startup Many short notebook runs. Review startup and session sharing
Small-file orchestration Run counts scale with file counts, as in the production case study
Warehouse queries Warehouse leads the background table. Use Monitor in warehouse from the timepoint page
Semantic model refresh Refresh operations rank high, or several refreshes share a window
Backfills A one-off load or rerun lines up with the peak
Poorly scheduled background work Noncritical jobs share the critical window
Unexpected workload growth The 14-day trend climbs with no single event

Actions

Take the lowest-risk action that fits the evidence. Make one change at a time and record it, so you can tell what worked.

Low-risk actions

  1. Observe. With no throttling and carryforward clearing, keep watching and record the evidence.
  2. Mitigate: pause or reduce noncritical workloads. Hold reruns, ad hoc jobs and work with no deadline. Consumption already reported still counts, including canceled and failed operations.
  3. Mitigate: find runaway or repeated jobs. Look for retries and loops in the operation history, and stop what shouldn’t be running.
  4. Reschedule: stagger background work. Move noncritical stages out of the peak window.
  5. Reschedule: reduce unnecessary parallelism. Cap how many runs start together, so work queues in orchestration instead of landing at once.

Engineering changes

Do these after the incident unless a small, reviewed fix is what restores service.

  1. Optimize the expensive notebook or query identified in the item view.
  2. Reduce per-file orchestration. Batch files so one run handles many.
  3. Reuse sessions where appropriate. Share compatible Spark sessions and move inline installs into an Environment.
  4. Restructure workload scheduling around critical windows, using monitoring baselines.

Capacity actions

Sometimes capacity is the correct answer. Scaling after the evidence supports it is good engineering, not a failure.

  1. Isolate. Move critical or noisy workspaces to another capacity when one workload must not affect another. See scalable Fabric architecture.
  2. Scale: test the next F SKU when demand is sustained, or when throttling persists after the steps above.
  3. Validate after scaling. Compare CU(s) over time, throttling and carryforward with the earlier baseline.

Validation

After any action, confirm recovery with several signals. One metric alone should not determine recovery: utilization can fall while carryforward remains, and a quiet chart can simply mean rejected work never ran.

The capacity is healthy again when all of these are true:

  • No active throttling: every throttling trend is under 100% and System events shows NotOverloaded.
  • Carryforward decreasing or burned down: cumulative overage is falling or zero.
  • No billed overage standing in for recovery: capacity overage is not still paying off the excess.
  • Critical workloads completing normally: durations are near their usual range.
  • Interactive workloads responsive: reports and queries respond without delay.
  • CU returning toward the normal baseline for this time of day.
  • No unexpected rejection events in the operation tables or impacted items.

How long to observe. Allow for the 10 to 15 minute data lag, then watch at least through the next workload window. A background-heavy burst stays in the smoothed baseline for up to 24 hours, so expect that view to fall slowly.

The change did not work if:

  • Cumulative overage keeps growing after new work was reduced.
  • Throttling returns in the next workload window.
  • The capacity does not recover between windows.
  • Rejections continue, or critical work still misses its deadline.

If so, reverse the change where needed, go back to the decision tree with the new evidence, and consider escalating.

Rollback and safety

Know the cost and the way back before you act. These relief options end or shorten throttling, but none of them is a low-risk first move.

  • Temporarily scale up. Burns carryforward faster. Cost: the pay-as-you-go rate for the larger size. Reverse: scale back down after the checks below.
  • Pause and resume. Ends throttling immediately. Cost: remaining overage and smoothed usage are billed at once, and content can become unavailable. Reverse: resume, which also resumes billing.
  • Capacity overage. Keeps accepting work. Cost: three times the pay-as-you-go rate, with no extra performance, and the limit is not a hard cap. Reverse: lower the limit, which can bring throttling back.
  • Surge protection. Rejects background work early to protect interactive users. Cost: background jobs fail while it is active. Reverse: turn it off or adjust the thresholds.
  • Move workspaces. Spreads load across capacities. Cost: needs a target capacity with room, and adds risk mid-incident. Reverse: move the workspaces back.

Get approval first, through your own change process, for anything that affects cost, availability or other teams: resizing, pausing, overage limits, surge protection and moving workspaces. Resizing needs capacity admin rights and Azure permissions on the capacity.

Scaling down is unsafe until carryforward has burned down, the normal baseline clearly fits the smaller size, and the next workload window has passed cleanly. With capacity overage enabled, scaling down can create significant billed overage.

Do not do this

  • Do not resize solely because utilization briefly exceeded 100%.
  • Do not restart or re-run workloads blindly. Reruns add load, and failed runs still count.
  • Do not increase concurrency while troubleshooting saturation.
  • Do not make several tuning changes simultaneously.
  • Do not scale down immediately after the incident without checking carryforward and the normal baseline.
  • Do not assume one capacity graph identifies root cause.
  • Do not pause a capacity that is in use to clear throttling without approval.

Escalation

Escalate through your own incident process when any of these is true:

  • Background rejection occurs. All new requests are being rejected.
  • Interactive rejection keeps repeating after mitigation.
  • CU consumption cannot be explained by any workload or change you can find.
  • The capacity cannot recover between workload windows.
  • Critical workload impact persists after mitigation.
  • Evidence points toward a Fabric platform or service issue, such as behaviour across capacities or errors unrelated to load.

Throttling applies per capacity. If several unrelated capacities misbehave at once, suspect the service before the workload.

Check Microsoft service status

  1. Check the Microsoft Fabric support page for service status and known issues.
  2. If you are a Fabric administrator, check Service Health under Help + Support in the Admin portal, or in the Microsoft 365 admin center, where Fabric is listed under Power BI.
  3. If you need Microsoft, open a support ticket and attach your evidence checklist. Response times depend on your support plan.

After-action review

Do this while the evidence is still in the 14-day Metrics window.

  • What happened? Summarize the symptom and the impact.
  • When? Write the timeline: start, peak, mitigation, recovery.
  • What workloads contributed? Name the root cause or contributing factors.
  • Was throttling confirmed? State the highest stage reached, with evidence.
  • What mitigation was used? List each change and who approved it.
  • Did it work? Link the validation evidence.
  • What is the permanent action? Give it an owner and a date.
  • What telemetry was missing? Record the monitoring gap, and set capacity alerts if there were none.
  • Should capacity sizing change? Re-run the Capacity Planner with the incident numbers.
  • Should scheduling change? Review the critical windows.
  • Should this runbook be updated? Update the documentation, and the last reviewed date if you re-checked it.

Microsoft references

Checked on the last reviewed date above. Microsoft says the throttling policy may change.

Tools

Quick references

  • Monitoring & ObservabilityIntermediate

    Monitor Fabric Pipelines and Notebooks

    Monitor Microsoft Fabric pipelines and notebooks using Monitoring hub, Workspace Monitoring, Spark monitoring, KQL, alerts, and capacity telemetry so failures and performance regressions become observable.

  • PerformanceIntermediate

    Fabric Notebook Performance

    Troubleshoot and improve Microsoft Fabric notebook performance by separating Spark session startup, data scan, shuffle, skew, driver pressure, concurrency, and write-path bottlenecks.

  • TroubleshootingIntermediate

    Debug Fabric Pipelines

    Troubleshoot Microsoft Fabric pipeline failures by isolating the failed activity, separating transient from deterministic errors, validating parameters and identities, and using retries, logging, and controlled reruns safely.

Understand the why