Browse cheat sheets · AI Engineering for Data
LLM Cost & Token Budgeting
Estimate and control LLM workload cost with simple formulas for cost per request, per day and per month, plus token budgets, model tiering, caching, batch workloads and monitoring.
On this page
- Problem
- Quick answer
- Cost formulas
- Why pricing changes over time
- Code / commands
- 1. Estimate the cost of one request
- 2. Project daily and monthly cost
- 3. Use percentiles, not only averages
- 4. Include retries, escalations and reasoning tokens
- 5. Check throughput against rate limits
- 6. Set a token budget per request
- 7. Cap output with max output tokens
- 8. Add user and session budgets
- 9. Add workflow budgets
- 10. Tier models by task
- 11. Separate batch and interactive workloads
- 12. Estimate the effect of caching
- 13. Monitor cost continuously
- What the result means
- What to look for
- Common mistakes
- 1. Hardcoding prices
- 2. Estimating from the average request only
- 3. Ignoring output tokens
- 4. Forgetting retries and escalations
- 5. Budgeting monthly only
- 6. Assuming caching always saves money
- 7. Choosing the cheapest model without measuring quality
- Production notes
- Checklist
- Next step
- Related deep dives
Problem
You need to answer questions such as:
- How much will this LLM feature cost per month?
- What happens to cost if usage doubles?
- Which feature, user or pipeline is driving spend?
- How do we stop one workflow from consuming the whole budget?
- Is a cheaper model, a cache or a batch workload worth it?
LLM cost is usage-based. It depends on tokens, request volume, model choice, retries and caching, and the prices themselves change over time.
Quick answer
request_cost = (input_tokens × input_rate)
+ (output_tokens × output_rate)
monthly_cost = average_request_cost × requests_per_month
Rates are usually quoted per million tokens. Read them from configuration, not from code or documentation, because they change.
To control cost:
- Log tokens and estimated cost per request.
- Set a token budget per request and a max output token cap.
- Add budgets per user, session and workflow run.
- Route simple work to cheaper models.
- Use batch processing for work that is not interactive, where your provider offers it.
- Cache only where inputs repeat.
- Alert on budget burn, not just on the monthly invoice.
Cost formulas
| Quantity | Formula |
|---|---|
| Tokens per request | input_tokens + output_tokens |
| Cost per request | input_tokens × input_rate + output_tokens × output_rate |
| With cached input | uncached_input × input_rate + cached_input × cached_input_rate + output_tokens × output_rate |
| Requests per day | requests_per_minute × active_minutes_per_day (or from logs) |
| Effective requests | requests × (1 + retry_rate) |
| Cost per day | average_request_cost × effective_requests_per_day |
| Cost per month | cost_per_day × days_in_month |
| Tokens per minute | requests_per_minute × average_tokens_per_request |
How cached tokens are reported and priced is provider-specific: some report them as part of input tokens, some separately, and some charge for writing to the cache. Check the usage fields of the API you call.
Why pricing changes over time
- Providers release new models and reprice old ones.
- Prices differ between model sizes and between input, output and cached input.
- Batch, priority and regional offerings can have different rates.
- Your own usage changes: longer prompts, new features, more users.
So:
- keep rates in a versioned configuration (“rate card”), not in code
- store the estimated cost and the rate card version with each logged request
- re-estimate when you change model, prompt or traffic
Code / commands
1. Estimate the cost of one request
from dataclasses import dataclass
@dataclass(frozen=True)
class Rates:
"""Price per 1M tokens. Load from configuration: provider prices change."""
input_per_m: float
output_per_m: float
cached_input_per_m: float | None = None
def request_cost(rates: Rates, input_tokens: int, output_tokens: int, cached_input_tokens: int = 0) -> float:
"""Cost of one request. `input_tokens` includes `cached_input_tokens`."""
uncached = input_tokens - cached_input_tokens
cached_rate = rates.cached_input_per_m if rates.cached_input_per_m is not None else rates.input_per_m
return (
uncached * rates.input_per_m
+ cached_input_tokens * cached_rate
+ output_tokens * rates.output_per_m
) / 1_000_000
2. Project daily and monthly cost
def monthly_cost(avg_request_cost: float, requests_per_day: float, days: int = 30, retry_rate: float = 0.0) -> float:
return avg_request_cost * requests_per_day * (1 + retry_rate) * days
example = Rates(input_per_m=1.00, output_per_m=4.00) # illustrative numbers, not real prices
cost = request_cost(example, input_tokens=1_500, output_tokens=400)
print(round(cost, 6)) # 0.0031
print(round(monthly_cost(cost, requests_per_day=50_000))) # 4650
Worked example with those illustrative rates:
| Step | Value |
|---|---|
| Input | 1,500 tokens × 1.00 / 1M = 0.0015 |
| Output | 400 tokens × 4.00 / 1M = 0.0016 |
| Cost per request | 0.0031 |
| 50,000 requests/day | 155 per day |
| 30 days | 4,650 per month |
Note how 400 output tokens cost more than 1,500 input tokens when the output rate is higher. Output length matters.
3. Use percentiles, not only averages
A few very long requests (large documents, long conversations, agent loops) can dominate cost.
SELECT DISTINCT
feature,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY input_tokens + output_tokens) OVER (PARTITION BY feature) AS p50_tokens,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY input_tokens + output_tokens) OVER (PARTITION BY feature) AS p95_tokens,
MAX(input_tokens + output_tokens) OVER (PARTITION BY feature) AS max_tokens
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -7, SYSUTCDATETIME());
Budget for p95, not for the average request.
4. Include retries, escalations and reasoning tokens
Real cost per successful task is higher than cost per call when:
- invalid output is retried
- failed cases escalate to a larger model
- the model bills reasoning tokens as output
- an agent makes several calls per user request
cost_per_successful_task = total_cost_of_all_calls / tasks_completed_successfully
5. Check throughput against rate limits
tokens_per_minute = requests_per_minute × average_tokens_per_request
requests_per_minute = peak traffic, not the daily average
Compare both with your provider’s request and token limits. A batch backfill can hit token-per-minute limits long before it hits a cost budget.
6. Set a token budget per request
Estimate the input before calling, and reserve room for the maximum output:
class BudgetExceeded(Exception):
pass
def check_request_budget(estimated_input: int, max_output: int, request_budget: int) -> None:
worst_case = estimated_input + max_output
if worst_case > request_budget:
raise BudgetExceeded(f"request needs up to {worst_case} tokens; budget is {request_budget}")
When the check fails, trim context (history, retrieval) before calling, or reject with a clear error.
7. Cap output with max output tokens
- Set a max output token value for every call.
- Size it from observed output lengths plus headroom, not from the model’s maximum.
- Treat a truncated response as a failure (invalid JSON, cut-off text), and log it.
8. Add user and session budgets
def remaining_tokens(daily_limit: int, used_today: int) -> int:
return max(daily_limit - used_today, 0)
Store usage per user and per session (for example in a table keyed by user and date), check it before each call, and return a clear message when the budget is spent. This also limits abuse of public endpoints.
9. Add workflow budgets
For pipelines and agents, budget the run, not only the call:
- maximum calls per run
- maximum total tokens per run
- maximum cost per run
- stop condition when the budget is reached, with partial results saved
class RunBudget:
def __init__(self, max_tokens: int, max_calls: int):
self.max_tokens, self.max_calls = max_tokens, max_calls
self.tokens = self.calls = 0
def charge(self, input_tokens: int, output_tokens: int) -> None:
self.calls += 1
self.tokens += input_tokens + output_tokens
if self.tokens > self.max_tokens or self.calls > self.max_calls:
raise BudgetExceeded(f"run stopped after {self.calls} calls and {self.tokens} tokens")
10. Tier models by task
| Task | Typical tier |
|---|---|
| Classification, tagging, short extraction | Smaller, cheaper model |
| Summaries of known structure | Smaller or mid-size model |
| Multi-step reasoning, ambiguous analysis | Larger or reasoning model |
| Deterministic parsing, formatting, math | No model: code, SQL or rules |
Measure quality per tier; a cheaper model that fails often can cost more per successful task.
11. Separate batch and interactive workloads
| Interactive | Batch | |
|---|---|---|
| Latency | Seconds matter | Minutes or hours acceptable |
| Volume | Spiky | Large, schedulable |
| Cost levers | Shorter prompts, smaller models, caching | Batch APIs where offered, off-peak scheduling, larger batches per call |
| Budget | Per user / session | Per run / per day |
Run backfills and enrichment jobs as batch workloads with their own budget.
12. Estimate the effect of caching
expected_input_cost = (1 - hit_rate) × uncached_cost + hit_rate × cached_cost
- Result caching saves the whole request on a hit, but only if identical inputs repeat.
- Prompt/prefix caching, where supported, reduces the cost of repeated prefixes; rules and pricing vary.
- Measure the real hit rate before counting the savings.
13. Monitor cost continuously
Store the estimated cost with each request:
SELECT
CAST(requested_at AS date) AS usage_date,
feature,
SUM(input_tokens) AS input_tokens,
SUM(output_tokens) AS output_tokens,
SUM(estimated_cost) AS estimated_cost
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -30, SYSUTCDATETIME())
GROUP BY CAST(requested_at AS date), feature
ORDER BY usage_date, estimated_cost DESC;
Reconcile with the provider’s billing data regularly: your estimate is only as good as your rate card.
What the result means
| Signal | Likely meaning |
|---|---|
| Cost grows faster than requests | Prompts, history or outputs are getting longer |
| Output cost dominates | Answers are verbose or uncapped |
| One feature dominates | Optimize or budget that feature first |
| p95 far above p50 | A few long requests drive cost: cap them |
| Cost per successful task far above cost per call | Retries or escalations |
| Estimate differs from the invoice | Outdated rate card, unlogged calls or cached-token accounting |
What to look for
- Requests without a max output token cap.
- Features without an owner or a budget.
- Unlogged calls (scripts, notebooks, experiments).
- Backfills running against interactive limits.
- Retry storms after a deployment.
- A rate card that is older than the current model version.
Common mistakes
1. Hardcoding prices
Prices change. Keep them in configuration with a version and an effective date.
2. Estimating from the average request only
Long-tail requests and agent loops dominate real spend.
3. Ignoring output tokens
Output is often priced higher per token than input.
4. Forgetting retries and escalations
Cost per call is not cost per successful task.
5. Budgeting monthly only
A runaway job can spend a month’s budget in hours. Use per-run, per-user and per-day limits.
6. Assuming caching always saves money
It saves money only when inputs or prefixes repeat, and some providers charge for cache writes.
7. Choosing the cheapest model without measuring quality
Wrong answers have a cost too: rework, escalations and lost trust.
Production notes
- Keep a versioned rate card per model; log its version with every request.
- Enforce budgets in code before calling the model, not after the invoice.
- Alert on daily burn rate and on unusual per-feature spikes.
- Give every LLM feature an owner and a budget.
- Keep experiments and production on separate keys or projects so their costs are visible separately.
- Review cost after every prompt, model or traffic change.
- Log token counts and costs, not raw prompts, when prompts contain sensitive data.
Checklist
- Rates come from versioned configuration.
- Each request logs input, output and cached tokens, model and estimated cost.
- Every call has a max output token cap.
- Per-request, per-user/session and per-run budgets exist.
- Peak tokens per minute fit within provider limits.
- Model tier is chosen per task and measured.
- Batch work is separated from interactive work.
- Cache hit rate is measured before savings are claimed.
- Estimates are reconciled with billing data.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.