Browse cheat sheets · AI Engineering for Data

Technology: AI Engineering for DataCategory: Cost ManagementDifficulty: IntermediateUpdated

LLM Cost & Token Budgeting

Estimate and control LLM workload cost with simple formulas for cost per request, per day and per month, plus token budgets, model tiering, caching, batch workloads and monitoring.

On this page
  1. Problem
  2. Quick answer
  3. Cost formulas
  4. Why pricing changes over time
  5. Code / commands
  6. 1. Estimate the cost of one request
  7. 2. Project daily and monthly cost
  8. 3. Use percentiles, not only averages
  9. 4. Include retries, escalations and reasoning tokens
  10. 5. Check throughput against rate limits
  11. 6. Set a token budget per request
  12. 7. Cap output with max output tokens
  13. 8. Add user and session budgets
  14. 9. Add workflow budgets
  15. 10. Tier models by task
  16. 11. Separate batch and interactive workloads
  17. 12. Estimate the effect of caching
  18. 13. Monitor cost continuously
  19. What the result means
  20. What to look for
  21. Common mistakes
  22. 1. Hardcoding prices
  23. 2. Estimating from the average request only
  24. 3. Ignoring output tokens
  25. 4. Forgetting retries and escalations
  26. 5. Budgeting monthly only
  27. 6. Assuming caching always saves money
  28. 7. Choosing the cheapest model without measuring quality
  29. Production notes
  30. Checklist
  31. Next step
  32. Related deep dives

Problem

You need to answer questions such as:

  • How much will this LLM feature cost per month?
  • What happens to cost if usage doubles?
  • Which feature, user or pipeline is driving spend?
  • How do we stop one workflow from consuming the whole budget?
  • Is a cheaper model, a cache or a batch workload worth it?

LLM cost is usage-based. It depends on tokens, request volume, model choice, retries and caching, and the prices themselves change over time.

Quick answer

request_cost = (input_tokens  × input_rate)
             + (output_tokens × output_rate)

monthly_cost = average_request_cost × requests_per_month

Rates are usually quoted per million tokens. Read them from configuration, not from code or documentation, because they change.

To control cost:

  1. Log tokens and estimated cost per request.
  2. Set a token budget per request and a max output token cap.
  3. Add budgets per user, session and workflow run.
  4. Route simple work to cheaper models.
  5. Use batch processing for work that is not interactive, where your provider offers it.
  6. Cache only where inputs repeat.
  7. Alert on budget burn, not just on the monthly invoice.

Cost formulas

Quantity Formula
Tokens per request input_tokens + output_tokens
Cost per request input_tokens × input_rate + output_tokens × output_rate
With cached input uncached_input × input_rate + cached_input × cached_input_rate + output_tokens × output_rate
Requests per day requests_per_minute × active_minutes_per_day (or from logs)
Effective requests requests × (1 + retry_rate)
Cost per day average_request_cost × effective_requests_per_day
Cost per month cost_per_day × days_in_month
Tokens per minute requests_per_minute × average_tokens_per_request

How cached tokens are reported and priced is provider-specific: some report them as part of input tokens, some separately, and some charge for writing to the cache. Check the usage fields of the API you call.

Why pricing changes over time

  • Providers release new models and reprice old ones.
  • Prices differ between model sizes and between input, output and cached input.
  • Batch, priority and regional offerings can have different rates.
  • Your own usage changes: longer prompts, new features, more users.

So:

  • keep rates in a versioned configuration (“rate card”), not in code
  • store the estimated cost and the rate card version with each logged request
  • re-estimate when you change model, prompt or traffic

Code / commands

1. Estimate the cost of one request

from dataclasses import dataclass


@dataclass(frozen=True)
class Rates:
    """Price per 1M tokens. Load from configuration: provider prices change."""
    input_per_m: float
    output_per_m: float
    cached_input_per_m: float | None = None


def request_cost(rates: Rates, input_tokens: int, output_tokens: int, cached_input_tokens: int = 0) -> float:
    """Cost of one request. `input_tokens` includes `cached_input_tokens`."""
    uncached = input_tokens - cached_input_tokens
    cached_rate = rates.cached_input_per_m if rates.cached_input_per_m is not None else rates.input_per_m
    return (
        uncached * rates.input_per_m
        + cached_input_tokens * cached_rate
        + output_tokens * rates.output_per_m
    ) / 1_000_000

2. Project daily and monthly cost

def monthly_cost(avg_request_cost: float, requests_per_day: float, days: int = 30, retry_rate: float = 0.0) -> float:
    return avg_request_cost * requests_per_day * (1 + retry_rate) * days


example = Rates(input_per_m=1.00, output_per_m=4.00)   # illustrative numbers, not real prices
cost = request_cost(example, input_tokens=1_500, output_tokens=400)
print(round(cost, 6))                                      # 0.0031
print(round(monthly_cost(cost, requests_per_day=50_000)))  # 4650

Worked example with those illustrative rates:

Step Value
Input 1,500 tokens × 1.00 / 1M = 0.0015
Output 400 tokens × 4.00 / 1M = 0.0016
Cost per request 0.0031
50,000 requests/day 155 per day
30 days 4,650 per month

Note how 400 output tokens cost more than 1,500 input tokens when the output rate is higher. Output length matters.


3. Use percentiles, not only averages

A few very long requests (large documents, long conversations, agent loops) can dominate cost.

SELECT DISTINCT
    feature,
    PERCENTILE_CONT(0.5)  WITHIN GROUP (ORDER BY input_tokens + output_tokens) OVER (PARTITION BY feature) AS p50_tokens,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY input_tokens + output_tokens) OVER (PARTITION BY feature) AS p95_tokens,
    MAX(input_tokens + output_tokens) OVER (PARTITION BY feature) AS max_tokens
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -7, SYSUTCDATETIME());

Budget for p95, not for the average request.


4. Include retries, escalations and reasoning tokens

Real cost per successful task is higher than cost per call when:

  • invalid output is retried
  • failed cases escalate to a larger model
  • the model bills reasoning tokens as output
  • an agent makes several calls per user request
cost_per_successful_task = total_cost_of_all_calls / tasks_completed_successfully

5. Check throughput against rate limits

tokens_per_minute   = requests_per_minute × average_tokens_per_request
requests_per_minute = peak traffic, not the daily average

Compare both with your provider’s request and token limits. A batch backfill can hit token-per-minute limits long before it hits a cost budget.


6. Set a token budget per request

Estimate the input before calling, and reserve room for the maximum output:

class BudgetExceeded(Exception):
    pass


def check_request_budget(estimated_input: int, max_output: int, request_budget: int) -> None:
    worst_case = estimated_input + max_output
    if worst_case > request_budget:
        raise BudgetExceeded(f"request needs up to {worst_case} tokens; budget is {request_budget}")

When the check fails, trim context (history, retrieval) before calling, or reject with a clear error.


7. Cap output with max output tokens

  • Set a max output token value for every call.
  • Size it from observed output lengths plus headroom, not from the model’s maximum.
  • Treat a truncated response as a failure (invalid JSON, cut-off text), and log it.

8. Add user and session budgets

def remaining_tokens(daily_limit: int, used_today: int) -> int:
    return max(daily_limit - used_today, 0)

Store usage per user and per session (for example in a table keyed by user and date), check it before each call, and return a clear message when the budget is spent. This also limits abuse of public endpoints.


9. Add workflow budgets

For pipelines and agents, budget the run, not only the call:

  • maximum calls per run
  • maximum total tokens per run
  • maximum cost per run
  • stop condition when the budget is reached, with partial results saved
class RunBudget:
    def __init__(self, max_tokens: int, max_calls: int):
        self.max_tokens, self.max_calls = max_tokens, max_calls
        self.tokens = self.calls = 0

    def charge(self, input_tokens: int, output_tokens: int) -> None:
        self.calls += 1
        self.tokens += input_tokens + output_tokens
        if self.tokens > self.max_tokens or self.calls > self.max_calls:
            raise BudgetExceeded(f"run stopped after {self.calls} calls and {self.tokens} tokens")

10. Tier models by task

Task Typical tier
Classification, tagging, short extraction Smaller, cheaper model
Summaries of known structure Smaller or mid-size model
Multi-step reasoning, ambiguous analysis Larger or reasoning model
Deterministic parsing, formatting, math No model: code, SQL or rules

Measure quality per tier; a cheaper model that fails often can cost more per successful task.


11. Separate batch and interactive workloads

Interactive Batch
Latency Seconds matter Minutes or hours acceptable
Volume Spiky Large, schedulable
Cost levers Shorter prompts, smaller models, caching Batch APIs where offered, off-peak scheduling, larger batches per call
Budget Per user / session Per run / per day

Run backfills and enrichment jobs as batch workloads with their own budget.


12. Estimate the effect of caching

expected_input_cost = (1 - hit_rate) × uncached_cost + hit_rate × cached_cost
  • Result caching saves the whole request on a hit, but only if identical inputs repeat.
  • Prompt/prefix caching, where supported, reduces the cost of repeated prefixes; rules and pricing vary.
  • Measure the real hit rate before counting the savings.

13. Monitor cost continuously

Store the estimated cost with each request:

SELECT
    CAST(requested_at AS date) AS usage_date,
    feature,
    SUM(input_tokens)   AS input_tokens,
    SUM(output_tokens)  AS output_tokens,
    SUM(estimated_cost) AS estimated_cost
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -30, SYSUTCDATETIME())
GROUP BY CAST(requested_at AS date), feature
ORDER BY usage_date, estimated_cost DESC;

Reconcile with the provider’s billing data regularly: your estimate is only as good as your rate card.

What the result means

Signal Likely meaning
Cost grows faster than requests Prompts, history or outputs are getting longer
Output cost dominates Answers are verbose or uncapped
One feature dominates Optimize or budget that feature first
p95 far above p50 A few long requests drive cost: cap them
Cost per successful task far above cost per call Retries or escalations
Estimate differs from the invoice Outdated rate card, unlogged calls or cached-token accounting

What to look for

  • Requests without a max output token cap.
  • Features without an owner or a budget.
  • Unlogged calls (scripts, notebooks, experiments).
  • Backfills running against interactive limits.
  • Retry storms after a deployment.
  • A rate card that is older than the current model version.

Common mistakes

1. Hardcoding prices

Prices change. Keep them in configuration with a version and an effective date.

2. Estimating from the average request only

Long-tail requests and agent loops dominate real spend.

3. Ignoring output tokens

Output is often priced higher per token than input.

4. Forgetting retries and escalations

Cost per call is not cost per successful task.

5. Budgeting monthly only

A runaway job can spend a month’s budget in hours. Use per-run, per-user and per-day limits.

6. Assuming caching always saves money

It saves money only when inputs or prefixes repeat, and some providers charge for cache writes.

7. Choosing the cheapest model without measuring quality

Wrong answers have a cost too: rework, escalations and lost trust.

Production notes

  • Keep a versioned rate card per model; log its version with every request.
  • Enforce budgets in code before calling the model, not after the invoice.
  • Alert on daily burn rate and on unusual per-feature spikes.
  • Give every LLM feature an owner and a budget.
  • Keep experiments and production on separate keys or projects so their costs are visible separately.
  • Review cost after every prompt, model or traffic change.
  • Log token counts and costs, not raw prompts, when prompts contain sensitive data.

Checklist

  • Rates come from versioned configuration.
  • Each request logs input, output and cached tokens, model and estimated cost.
  • Every call has a max output token cap.
  • Per-request, per-user/session and per-run budgets exist.
  • Peak tokens per minute fit within provider limits.
  • Model tier is chosen per task and measured.
  • Batch work is separated from interactive work.
  • Cache hit rate is measured before savings are claimed.
  • Estimates are reconciled with billing data.

Next step

  • Prompts are too large → LLM Token Optimization.
  • Unsure which model tier fits → Choosing the Right AI Model.
  • Context keeps growing per request → LLM Context Engineering.

← All cheat sheets