Browse cheat sheets · AI Engineering for Data
LLM Token Optimization
Reduce LLM token usage in data applications without losing the context or answer quality you need: measure first, trim prompts and history, limit retrieval, compact outputs, cache and route.
On this page
- Problem
- Quick answer
- Token fundamentals
- Input vs output tokens
- Why repeated context is expensive
- Code / commands
- 1. Measure before optimizing
- 2. Shorten and deduplicate instructions
- 3. Keep stable instructions stable
- 4. Summarize or trim conversation history
- 5. Limit retrieved context
- 6. Chunk only when useful
- 7. Send the relevant schema, not the whole catalog
- 8. Ask for concise output formats
- 9. Structured output vs verbose prose
- 10. Cache stable context and results
- 11. Route tasks to the right model, or to no model
- 12. Batch small items when the task allows
- Technique decision table
- What to look for
- Common mistakes
- 1. Optimizing without a baseline
- 2. Using max output tokens to make answers short
- 3. Summarizing away identifiers
- 4. Retrieving top-20 “just in case”
- 5. Volatile data at the start of the prompt
- 6. Sending entire schemas or catalogs
- 7. Counting characters or words
- 8. Assuming fewer tokens is always cheaper
- 9. Ignoring retry and reasoning tokens
- Production notes
- Checklist
- Next step
- Related deep dives
Problem
An LLM feature works in testing. In production:
- the bill grows faster than usage
- latency increases as conversations get longer
- requests start hitting the context window limit
- the same instructions and schemas are resent on every call
- answers are long when a short structured result would do
Tokens drive three things at once: cost, latency and what fits in the context window.
The goal is not the smallest possible prompt. The goal is to send and generate only what the task needs, and to prove that quality did not drop.
Quick answer
- Measure token usage per feature, per model, input vs output.
- Remove what the model does not need: duplicated instructions, full schemas, stale history, low-relevance chunks.
- Constrain the output: compact structured format, explicit length, a max output token cap as a safety limit.
- Reuse what repeats: a stable prompt prefix (provider prompt caching, where supported) and cached results for identical requests.
- Route simple tasks to a smaller model, or to normal code.
- Re-check quality on an evaluation set after every change.
input tokens = system instructions + tool definitions + examples
+ conversation history + retrieved context + tool results + user input
output tokens = generated answer (+ reasoning tokens, on models that bill them)
Token fundamentals
- Models read and write tokens, not characters or words. A token is a piece of text produced by the model’s tokenizer.
- Token counts depend on the tokenizer: the same text can produce different counts on different model families.
- Code, JSON, long identifiers, numbers, whitespace and non-English text often tokenize less efficiently than plain English prose.
- Character-based estimates are fine for planning. For real numbers, use the model’s tokenizer or the usage fields returned by the API.
- Everything in the request counts: system prompt, tool/function definitions, examples, history, retrieved documents, tool results.
- The context window bounds what one request can hold. Depending on the model, input and output share it or output has a separate limit.
Input vs output tokens
| Input tokens | Output tokens | |
|---|---|---|
| What | Everything you send | Everything the model generates |
| You control it with | Prompt design, retrieval limits, history policy | Output format, instructions, max output tokens |
| Pricing | Usually priced per token | Usually priced per token, often at a higher rate than input |
| Latency | Adds processing time before the first token | Generation time grows with the number of output tokens |
Prices and the input/output ratio differ by model and change over time. Read them from the provider’s current price list, never from a cheat sheet.
Some models generate internal reasoning tokens that are billed as output even when they are not returned. Check the usage fields.
Why repeated context is expensive
Typical LLM APIs are stateless: a “conversation” resends the history on every turn. Fixed context (instructions, schema, examples) is paid again on every call, and history grows each turn.
def cumulative_input_tokens(base: int, per_turn: int, turns: int) -> int:
"""Input tokens sent over a whole conversation when the full history is resent each turn."""
return sum(base + per_turn * (turn - 1) for turn in range(1, turns + 1))
print(cumulative_input_tokens(base=2_000, per_turn=300, turns=20)) # 97000
With a 2,000-token fixed context and 300 tokens added per turn, 20 turns send 97,000 input tokens, of which 40,000 are the same fixed context repeated. The history part grows roughly with the square of the number of turns.
The same arithmetic applies to batch pipelines: an LLM call per row with a 1,500-token instruction block pays for those instructions on every row.
Code / commands
1. Measure before optimizing
Log usage for every call. Map the provider’s usage fields once, in your wrapper.
import json
import logging
import time
log = logging.getLogger("llm.usage")
def record_usage(feature: str, model: str, prompt_version: str, usage: dict, started: float) -> None:
"""Log one call's token usage. `usage` is the usage block returned by your provider's API."""
log.info(json.dumps({
"feature": feature,
"model": model,
"prompt_version": prompt_version,
"input_tokens": usage.get("input_tokens"),
"output_tokens": usage.get("output_tokens"),
"cached_input_tokens": usage.get("cached_input_tokens", 0),
"latency_ms": round((time.monotonic() - started) * 1000),
}))
Log counts, not prompt text: prompts can contain sensitive data.
Find the biggest consumers (T-SQL over a usage table):
SELECT
feature,
model,
COUNT(*) AS requests,
AVG(input_tokens) AS avg_input_tokens,
AVG(output_tokens) AS avg_output_tokens,
SUM(input_tokens + output_tokens) AS total_tokens
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -7, SYSUTCDATETIME())
GROUP BY feature, model
ORDER BY total_tokens DESC;
Then break one representative prompt down by component:
def prompt_breakdown(parts: dict[str, str], count_tokens) -> dict[str, int]:
"""Token count per prompt component; `count_tokens` is your model's tokenizer function."""
return {name: count_tokens(text) for name, text in parts.items()}
# {"system": 1850, "schema": 4200, "history": 6100, "retrieved": 3900, "question": 40}
Optimize the largest component first.
2. Shorten and deduplicate instructions
Long system prompts accumulate repeated rules, politeness and examples nobody re-tested.
Before:
You are a very helpful, careful and precise assistant. Please always make sure that you
answer accurately. It is very important that you are accurate. Always answer in JSON.
Make sure the answer is JSON. Do not include anything other than JSON in your answer.
After:
Classify the support ticket. Return only JSON matching the schema.
Keep instructions that change behavior. Remove the ones that repeat, and re-run the evaluation set to confirm.
3. Keep stable instructions stable
Put fixed content first and variable content last:
[stable] role, rules, output schema, examples
[variable] retrieved context, user input
This makes prompts easier to version and lets provider-side prompt/prefix caching work where it is offered (rules such as minimum length and expiry are provider-specific).
Do not inject timestamps, request IDs or user names into the stable part: any change at the start breaks prefix reuse.
4. Summarize or trim conversation history
Options, from simplest:
- keep the last N turns
- keep the last N turns plus a running summary of older ones
- keep extracted facts and decisions instead of raw turns
- drop tool outputs once they have been used
def build_history(turns: list[dict], summary: str | None, keep_last: int = 4) -> list[dict]:
"""Keep the last turns verbatim and replace older ones with a running summary."""
recent = turns[-keep_last:]
if summary and len(turns) > keep_last:
return [{"role": "user", "content": f"Summary of the earlier conversation:\n{summary}"}] + recent
return recent
Summaries lose detail. Instruct the summarizer to keep identifiers, numbers, table names and decisions verbatim.
5. Limit retrieved context
More chunks is not more accuracy. Low-relevance chunks cost tokens and can distract the model.
def select_chunks(chunks: list[dict], budget_tokens: int, min_score: float) -> list[dict]:
"""Highest-scoring, non-duplicate chunks that fit in a token budget."""
selected, seen, used = [], set(), 0
for chunk in sorted(chunks, key=lambda c: c["score"], reverse=True):
if chunk["score"] < min_score or chunk["text_hash"] in seen:
continue
if used + chunk["tokens"] > budget_tokens:
continue
selected.append(chunk)
seen.add(chunk["text_hash"])
used += chunk["tokens"]
return selected
Tune budget_tokens and min_score against an evaluation set; there is no universal top-k.
6. Chunk only when useful
Chunking serves retrieval over large corpora. It is not always needed:
- a short document that is entirely relevant can be sent once
- structured data is usually better queried (SQL, aggregates) than chunked into text
- chunks that are too small lose context; chunks that are too large waste tokens
7. Send the relevant schema, not the whole catalog
For SQL generation, send only the tables and columns the question needs, in a compact form:
orders(order_id int pk, customer_id int fk->customers, order_date date,
status varchar -- 'open' | 'shipped' | 'cancelled', amount decimal(12,2))
customers(customer_id int pk, country char(2), segment varchar)
Pre-select tables with metadata search or a table-level retrieval step. Add descriptions only where a name is ambiguous.
8. Ask for concise output formats
State the format and length you need:
Return only JSON matching the schema. No explanation. "reason" must be at most 15 words.
Also set a max output tokens cap, but as a safety limit, not as the way to make answers short: a cap that is too low truncates the output mid-JSON.
9. Structured output vs verbose prose
Prose:
After reviewing the ticket carefully, it appears that the customer is most likely
experiencing a billing issue, since they mention being charged twice...
Structured:
{"label": "billing_issue", "confidence": 0.82, "evidence": "charged twice"}
Structured output is usually shorter, easier to validate and easier to store. Keys cost tokens too: keep them short but readable. Use the provider’s structured output / JSON schema feature when available, and validate anyway.
10. Cache stable context and results
Two different mechanisms:
- Prompt/prefix caching (provider feature, where supported): repeated identical prefixes can be billed and processed at a reduced cost. Eligibility rules vary.
- Application result cache: identical inputs return a stored answer without calling the model.
import hashlib
import json
def cache_key(model: str, prompt_version: str, payload: dict) -> str:
"""Stable key for a result cache: same model, prompt version and normalized input."""
raw = json.dumps({"m": model, "v": prompt_version, "p": payload}, sort_keys=True, ensure_ascii=False)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
A result cache only helps when inputs repeat. Include the prompt version and model in the key, and set an expiry when the underlying data changes.
11. Route tasks to the right model, or to no model
def route(task: str, text: str) -> str:
if task in {"parse_date", "extract_email"}:
return "code" # deterministic parser: no LLM call
if task in {"classify_ticket", "extract_fields"} and len(text) < 4_000:
return "small-model"
return "large-model"
Routing does not reduce tokens; it makes each token cheaper and often faster. Escalate to a larger model when validation fails instead of sending everything to the largest one.
12. Batch small items when the task allows
Several records per call share one copy of the instructions:
{"records": [
{"id": "t-101", "text": "Refund not received"},
{"id": "t-102", "text": "Cannot log in after reset"}
]}
Require the IDs in the output and check that every input ID comes back exactly once. Large batches can lower quality and make one failure affect many records.
Technique decision table
| Technique | Saves input tokens | Saves output tokens | Main risk |
|---|---|---|---|
| Shorten / deduplicate the system prompt | Yes | — | Removing an instruction the model relied on |
| Summarize history | Yes | — | Losing facts, IDs or decisions |
| Reduce retrieved chunks | Yes | — | Dropping the chunk that held the answer |
| Send only the relevant schema | Yes | — | Wrong table pre-selection, wrong SQL |
| Request compact JSON | — | Yes | Less explanation for human readers |
| Set max output tokens | — | Caps the worst case | Truncated or invalid output |
| Prompt/prefix caching | Reduces cost of repeated input, where supported | — | Prefix changes defeat it; provider-specific rules |
| Cache repeated results | Whole request avoided | Whole request avoided | Stale answers, wrong cache key |
| Route simple tasks to a smaller model | Same tokens, cheaper | Same tokens, cheaper | Lower quality on hard cases |
| Batch records per call | Yes, shared instructions | Slightly | Partial failures, ID mismatches |
Savings depend on the workload. Measure them; do not assume a percentage.
What to look for
- Which component dominates input: instructions, schema, history, retrieval or tool results?
- The output/input ratio per feature.
- p50 and p95 tokens per request, not only averages.
- Requests close to the context window limit.
- Retries: a failed validation pays for the request twice.
- Cache hit rate, if you cache.
- In agent or tool loops, tool results reinjected at every step.
Common mistakes
1. Optimizing without a baseline
Without an evaluation set you cannot tell whether a shorter prompt is cheaper and still correct.
2. Using max output tokens to make answers short
The model does not plan for the cap; the output is cut off. Ask for brevity in the instructions and keep the cap as a safety limit.
3. Summarizing away identifiers
“The user asked about a table” is useless. Keep sales.orders, the date range and the decision made.
4. Retrieving top-20 “just in case”
Extra chunks cost tokens and can push the model toward the wrong source.
5. Volatile data at the start of the prompt
Timestamps or user IDs in the first lines defeat prefix caching and make prompt diffs noisy.
6. Sending entire schemas or catalogs
Thousands of columns for a question about two tables.
7. Counting characters or words
Estimates drift, especially for code, JSON and non-English text. Use the tokenizer or the API usage fields.
8. Assuming fewer tokens is always cheaper
A shorter prompt that causes retries, escalations or wrong answers can cost more per successful task.
9. Ignoring retry and reasoning tokens
They appear in the bill, not in your prompt.
Production notes
- Version prompts and log the prompt version with usage, so cost changes can be traced to a change.
- Set token budgets per feature, with alerts.
- Re-run the quality evaluation after every optimization.
- Tokenizers and prices are model-specific: re-measure when switching models.
- Leave headroom in the context window for the output.
- Log token counts, not raw prompts, when prompts can contain customer or sensitive data.
Checklist
- Usage is logged per feature, model and prompt version.
- The largest prompt component is known.
- Instructions are deduplicated and tested.
- Stable content comes first; volatile content last.
- History has a retention or summarization policy.
- Retrieval has a token budget and a relevance threshold.
- Only relevant schema is sent.
- The output format is compact and validated.
- Max output tokens is set as a safety cap.
- Caching is used only where inputs or prefixes actually repeat.
- Simple tasks are routed to cheaper models or to code.
- Quality is re-checked after each change.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.