Browse cheat sheets · AI Engineering for Data

Technology: AI Engineering for DataCategory: Token OptimizationDifficulty: IntermediateUpdated

LLM Token Optimization

Reduce LLM token usage in data applications without losing the context or answer quality you need: measure first, trim prompts and history, limit retrieval, compact outputs, cache and route.

On this page
  1. Problem
  2. Quick answer
  3. Token fundamentals
  4. Input vs output tokens
  5. Why repeated context is expensive
  6. Code / commands
  7. 1. Measure before optimizing
  8. 2. Shorten and deduplicate instructions
  9. 3. Keep stable instructions stable
  10. 4. Summarize or trim conversation history
  11. 5. Limit retrieved context
  12. 6. Chunk only when useful
  13. 7. Send the relevant schema, not the whole catalog
  14. 8. Ask for concise output formats
  15. 9. Structured output vs verbose prose
  16. 10. Cache stable context and results
  17. 11. Route tasks to the right model, or to no model
  18. 12. Batch small items when the task allows
  19. Technique decision table
  20. What to look for
  21. Common mistakes
  22. 1. Optimizing without a baseline
  23. 2. Using max output tokens to make answers short
  24. 3. Summarizing away identifiers
  25. 4. Retrieving top-20 “just in case”
  26. 5. Volatile data at the start of the prompt
  27. 6. Sending entire schemas or catalogs
  28. 7. Counting characters or words
  29. 8. Assuming fewer tokens is always cheaper
  30. 9. Ignoring retry and reasoning tokens
  31. Production notes
  32. Checklist
  33. Next step
  34. Related deep dives

Problem

An LLM feature works in testing. In production:

  • the bill grows faster than usage
  • latency increases as conversations get longer
  • requests start hitting the context window limit
  • the same instructions and schemas are resent on every call
  • answers are long when a short structured result would do

Tokens drive three things at once: cost, latency and what fits in the context window.

The goal is not the smallest possible prompt. The goal is to send and generate only what the task needs, and to prove that quality did not drop.

Quick answer

  1. Measure token usage per feature, per model, input vs output.
  2. Remove what the model does not need: duplicated instructions, full schemas, stale history, low-relevance chunks.
  3. Constrain the output: compact structured format, explicit length, a max output token cap as a safety limit.
  4. Reuse what repeats: a stable prompt prefix (provider prompt caching, where supported) and cached results for identical requests.
  5. Route simple tasks to a smaller model, or to normal code.
  6. Re-check quality on an evaluation set after every change.
input tokens  = system instructions + tool definitions + examples
              + conversation history + retrieved context + tool results + user input
output tokens = generated answer (+ reasoning tokens, on models that bill them)

Token fundamentals

  • Models read and write tokens, not characters or words. A token is a piece of text produced by the model’s tokenizer.
  • Token counts depend on the tokenizer: the same text can produce different counts on different model families.
  • Code, JSON, long identifiers, numbers, whitespace and non-English text often tokenize less efficiently than plain English prose.
  • Character-based estimates are fine for planning. For real numbers, use the model’s tokenizer or the usage fields returned by the API.
  • Everything in the request counts: system prompt, tool/function definitions, examples, history, retrieved documents, tool results.
  • The context window bounds what one request can hold. Depending on the model, input and output share it or output has a separate limit.

Input vs output tokens

Input tokens Output tokens
What Everything you send Everything the model generates
You control it with Prompt design, retrieval limits, history policy Output format, instructions, max output tokens
Pricing Usually priced per token Usually priced per token, often at a higher rate than input
Latency Adds processing time before the first token Generation time grows with the number of output tokens

Prices and the input/output ratio differ by model and change over time. Read them from the provider’s current price list, never from a cheat sheet.

Some models generate internal reasoning tokens that are billed as output even when they are not returned. Check the usage fields.

Why repeated context is expensive

Typical LLM APIs are stateless: a “conversation” resends the history on every turn. Fixed context (instructions, schema, examples) is paid again on every call, and history grows each turn.

def cumulative_input_tokens(base: int, per_turn: int, turns: int) -> int:
    """Input tokens sent over a whole conversation when the full history is resent each turn."""
    return sum(base + per_turn * (turn - 1) for turn in range(1, turns + 1))


print(cumulative_input_tokens(base=2_000, per_turn=300, turns=20))  # 97000

With a 2,000-token fixed context and 300 tokens added per turn, 20 turns send 97,000 input tokens, of which 40,000 are the same fixed context repeated. The history part grows roughly with the square of the number of turns.

The same arithmetic applies to batch pipelines: an LLM call per row with a 1,500-token instruction block pays for those instructions on every row.

Code / commands

1. Measure before optimizing

Log usage for every call. Map the provider’s usage fields once, in your wrapper.

import json
import logging
import time

log = logging.getLogger("llm.usage")


def record_usage(feature: str, model: str, prompt_version: str, usage: dict, started: float) -> None:
    """Log one call's token usage. `usage` is the usage block returned by your provider's API."""
    log.info(json.dumps({
        "feature": feature,
        "model": model,
        "prompt_version": prompt_version,
        "input_tokens": usage.get("input_tokens"),
        "output_tokens": usage.get("output_tokens"),
        "cached_input_tokens": usage.get("cached_input_tokens", 0),
        "latency_ms": round((time.monotonic() - started) * 1000),
    }))

Log counts, not prompt text: prompts can contain sensitive data.

Find the biggest consumers (T-SQL over a usage table):

SELECT
    feature,
    model,
    COUNT(*)                          AS requests,
    AVG(input_tokens)                 AS avg_input_tokens,
    AVG(output_tokens)                AS avg_output_tokens,
    SUM(input_tokens + output_tokens) AS total_tokens
FROM llm_usage_log
WHERE requested_at >= DATEADD(day, -7, SYSUTCDATETIME())
GROUP BY feature, model
ORDER BY total_tokens DESC;

Then break one representative prompt down by component:

def prompt_breakdown(parts: dict[str, str], count_tokens) -> dict[str, int]:
    """Token count per prompt component; `count_tokens` is your model's tokenizer function."""
    return {name: count_tokens(text) for name, text in parts.items()}

# {"system": 1850, "schema": 4200, "history": 6100, "retrieved": 3900, "question": 40}

Optimize the largest component first.


2. Shorten and deduplicate instructions

Long system prompts accumulate repeated rules, politeness and examples nobody re-tested.

Before:

You are a very helpful, careful and precise assistant. Please always make sure that you
answer accurately. It is very important that you are accurate. Always answer in JSON.
Make sure the answer is JSON. Do not include anything other than JSON in your answer.

After:

Classify the support ticket. Return only JSON matching the schema.

Keep instructions that change behavior. Remove the ones that repeat, and re-run the evaluation set to confirm.


3. Keep stable instructions stable

Put fixed content first and variable content last:

[stable]   role, rules, output schema, examples
[variable] retrieved context, user input

This makes prompts easier to version and lets provider-side prompt/prefix caching work where it is offered (rules such as minimum length and expiry are provider-specific).

Do not inject timestamps, request IDs or user names into the stable part: any change at the start breaks prefix reuse.


4. Summarize or trim conversation history

Options, from simplest:

  • keep the last N turns
  • keep the last N turns plus a running summary of older ones
  • keep extracted facts and decisions instead of raw turns
  • drop tool outputs once they have been used
def build_history(turns: list[dict], summary: str | None, keep_last: int = 4) -> list[dict]:
    """Keep the last turns verbatim and replace older ones with a running summary."""
    recent = turns[-keep_last:]
    if summary and len(turns) > keep_last:
        return [{"role": "user", "content": f"Summary of the earlier conversation:\n{summary}"}] + recent
    return recent

Summaries lose detail. Instruct the summarizer to keep identifiers, numbers, table names and decisions verbatim.


5. Limit retrieved context

More chunks is not more accuracy. Low-relevance chunks cost tokens and can distract the model.

def select_chunks(chunks: list[dict], budget_tokens: int, min_score: float) -> list[dict]:
    """Highest-scoring, non-duplicate chunks that fit in a token budget."""
    selected, seen, used = [], set(), 0
    for chunk in sorted(chunks, key=lambda c: c["score"], reverse=True):
        if chunk["score"] < min_score or chunk["text_hash"] in seen:
            continue
        if used + chunk["tokens"] > budget_tokens:
            continue
        selected.append(chunk)
        seen.add(chunk["text_hash"])
        used += chunk["tokens"]
    return selected

Tune budget_tokens and min_score against an evaluation set; there is no universal top-k.


6. Chunk only when useful

Chunking serves retrieval over large corpora. It is not always needed:

  • a short document that is entirely relevant can be sent once
  • structured data is usually better queried (SQL, aggregates) than chunked into text
  • chunks that are too small lose context; chunks that are too large waste tokens

7. Send the relevant schema, not the whole catalog

For SQL generation, send only the tables and columns the question needs, in a compact form:

orders(order_id int pk, customer_id int fk->customers, order_date date,
       status varchar -- 'open' | 'shipped' | 'cancelled', amount decimal(12,2))
customers(customer_id int pk, country char(2), segment varchar)

Pre-select tables with metadata search or a table-level retrieval step. Add descriptions only where a name is ambiguous.


8. Ask for concise output formats

State the format and length you need:

Return only JSON matching the schema. No explanation. "reason" must be at most 15 words.

Also set a max output tokens cap, but as a safety limit, not as the way to make answers short: a cap that is too low truncates the output mid-JSON.


9. Structured output vs verbose prose

Prose:

After reviewing the ticket carefully, it appears that the customer is most likely
experiencing a billing issue, since they mention being charged twice...

Structured:

{"label": "billing_issue", "confidence": 0.82, "evidence": "charged twice"}

Structured output is usually shorter, easier to validate and easier to store. Keys cost tokens too: keep them short but readable. Use the provider’s structured output / JSON schema feature when available, and validate anyway.


10. Cache stable context and results

Two different mechanisms:

  • Prompt/prefix caching (provider feature, where supported): repeated identical prefixes can be billed and processed at a reduced cost. Eligibility rules vary.
  • Application result cache: identical inputs return a stored answer without calling the model.
import hashlib
import json


def cache_key(model: str, prompt_version: str, payload: dict) -> str:
    """Stable key for a result cache: same model, prompt version and normalized input."""
    raw = json.dumps({"m": model, "v": prompt_version, "p": payload}, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

A result cache only helps when inputs repeat. Include the prompt version and model in the key, and set an expiry when the underlying data changes.


11. Route tasks to the right model, or to no model

def route(task: str, text: str) -> str:
    if task in {"parse_date", "extract_email"}:
        return "code"            # deterministic parser: no LLM call
    if task in {"classify_ticket", "extract_fields"} and len(text) < 4_000:
        return "small-model"
    return "large-model"

Routing does not reduce tokens; it makes each token cheaper and often faster. Escalate to a larger model when validation fails instead of sending everything to the largest one.


12. Batch small items when the task allows

Several records per call share one copy of the instructions:

{"records": [
  {"id": "t-101", "text": "Refund not received"},
  {"id": "t-102", "text": "Cannot log in after reset"}
]}

Require the IDs in the output and check that every input ID comes back exactly once. Large batches can lower quality and make one failure affect many records.

Technique decision table

Technique Saves input tokens Saves output tokens Main risk
Shorten / deduplicate the system prompt Yes — Removing an instruction the model relied on
Summarize history Yes — Losing facts, IDs or decisions
Reduce retrieved chunks Yes — Dropping the chunk that held the answer
Send only the relevant schema Yes — Wrong table pre-selection, wrong SQL
Request compact JSON — Yes Less explanation for human readers
Set max output tokens — Caps the worst case Truncated or invalid output
Prompt/prefix caching Reduces cost of repeated input, where supported — Prefix changes defeat it; provider-specific rules
Cache repeated results Whole request avoided Whole request avoided Stale answers, wrong cache key
Route simple tasks to a smaller model Same tokens, cheaper Same tokens, cheaper Lower quality on hard cases
Batch records per call Yes, shared instructions Slightly Partial failures, ID mismatches

Savings depend on the workload. Measure them; do not assume a percentage.

What to look for

  • Which component dominates input: instructions, schema, history, retrieval or tool results?
  • The output/input ratio per feature.
  • p50 and p95 tokens per request, not only averages.
  • Requests close to the context window limit.
  • Retries: a failed validation pays for the request twice.
  • Cache hit rate, if you cache.
  • In agent or tool loops, tool results reinjected at every step.

Common mistakes

1. Optimizing without a baseline

Without an evaluation set you cannot tell whether a shorter prompt is cheaper and still correct.

2. Using max output tokens to make answers short

The model does not plan for the cap; the output is cut off. Ask for brevity in the instructions and keep the cap as a safety limit.

3. Summarizing away identifiers

“The user asked about a table” is useless. Keep sales.orders, the date range and the decision made.

4. Retrieving top-20 “just in case”

Extra chunks cost tokens and can push the model toward the wrong source.

5. Volatile data at the start of the prompt

Timestamps or user IDs in the first lines defeat prefix caching and make prompt diffs noisy.

6. Sending entire schemas or catalogs

Thousands of columns for a question about two tables.

7. Counting characters or words

Estimates drift, especially for code, JSON and non-English text. Use the tokenizer or the API usage fields.

8. Assuming fewer tokens is always cheaper

A shorter prompt that causes retries, escalations or wrong answers can cost more per successful task.

9. Ignoring retry and reasoning tokens

They appear in the bill, not in your prompt.

Production notes

  • Version prompts and log the prompt version with usage, so cost changes can be traced to a change.
  • Set token budgets per feature, with alerts.
  • Re-run the quality evaluation after every optimization.
  • Tokenizers and prices are model-specific: re-measure when switching models.
  • Leave headroom in the context window for the output.
  • Log token counts, not raw prompts, when prompts can contain customer or sensitive data.

Checklist

  • Usage is logged per feature, model and prompt version.
  • The largest prompt component is known.
  • Instructions are deduplicated and tested.
  • Stable content comes first; volatile content last.
  • History has a retention or summarization policy.
  • Retrieval has a token budget and a relevance threshold.
  • Only relevant schema is sent.
  • The output format is compact and validated.
  • Max output tokens is set as a safety cap.
  • Caching is used only where inputs or prefixes actually repeat.
  • Simple tasks are routed to cheaper models or to code.
  • Quality is re-checked after each change.

Next step

  • Need to estimate monthly spend → LLM Cost & Token Budgeting.
  • Most tokens come from history or retrieval → LLM Context Engineering.
  • A smaller model might be enough → Choosing the Right AI Model.
  • Prompts are long because instructions are vague → Prompt Patterns for Data Engineering.

← All cheat sheets