Browse cheat sheets · AI Engineering for Data
Choosing the Right AI Model
A practical framework for choosing an AI model for data applications: when not to use an LLM, which dimensions matter, a decision tree, and how to evaluate candidates on quality, latency and cost per successful task.
On this page
- Problem
- Quick answer
- First: do you need a model at all?
- Comparison dimensions
- Decision tree
- Code / commands
- 1. Build an evaluation set first
- 2. Score candidates on the same cases
- 3. Compare on cost per successful task
- 4. Require structured output and validate it
- 5. Cascade: small model first, escalate on failure
- 6. Add retrieval or tools for private and current data
- 7. Multimodal input
- 8. Check throughput and limits
- 9. Context window size is not a quality guarantee
- 10. Pin versions and re-evaluate on changes
- What the result means
- What to look for
- Common mistakes
- 1. Using an LLM for deterministic work
- 2. Choosing from public rankings alone
- 3. Defaulting to the largest model
- 4. Defaulting to the cheapest model
- 5. Expecting the model to know private data
- 6. Trusting output without validation
- 7. Evaluating once
- Production notes
- Checklist
- Next step
- Related deep dives
Problem
You need a model for a data task: extract fields from documents, classify tickets, generate SQL, summarize incidents, answer questions over internal data.
There are many options: small and large models, reasoning models, multimodal models, embedding models, hosted and self-hosted. Each trades quality against latency, cost, throughput and operational effort.
No model is the best choice for every task. The right choice is the cheapest, fastest option that meets your quality bar on your own data, and sometimes that option is not an LLM at all.
Quick answer
Is the task deterministic (parse, format, compute, look up)?
→ normal code, SQL, rules or a parser. No LLM.
Simple extraction or classification, high volume?
→ smaller, cheaper model + structured output + validation.
Complex reasoning, ambiguous input, multi-step analysis?
→ larger or reasoning model; measure the latency and cost.
Needs current, private or domain data?
→ model + retrieval or tools. The model does not "know" your data.
Needs exact, machine-readable output?
→ structured output + schema validation + retry or escalation.
Input includes images, scans, charts?
→ multimodal model, or OCR/parsing first and a text model after.
Then confirm with an evaluation set: accuracy, p95 latency and cost per successful task.
First: do you need a model at all?
| Task | Usually better without an LLM |
|---|---|
| Parse dates, emails, IDs, known formats | Parser, regex, library |
| Aggregations, totals, filters | SQL |
| Lookups against reference data | Join / dictionary lookup |
| Fixed business rules | Rules in code |
| Exact arithmetic | Code |
| Format conversion (CSV → Parquet, JSON → table) | Code |
An LLM fits when the input is unstructured or ambiguous, the rules are hard to write down, or the output is natural language.
A common robust design: code for everything deterministic, the model only for the ambiguous part.
Comparison dimensions
| Dimension | What it means | How to evaluate it |
|---|---|---|
| Capability / size | General quality on complex tasks | Your evaluation set, not public rankings alone |
| Reasoning depth | Multi-step logic, planning, analysis | Hard cases from your domain |
| Latency | Time to first token and to full answer | p50 and p95 with your real prompt sizes |
| Cost | Price per input/output token, tokens used | Cost per successful task |
| Context window | How much input fits in one request | Your largest real request plus output |
| Structured output | Reliably returns valid JSON / schema | Validation pass rate |
| Tool use | Calls functions/APIs correctly | Correct tool and arguments rate |
| Multimodal support | Images, documents, audio | Your real files, including bad scans |
| Throughput | Requests and tokens per minute available | Provider limits vs your peak load |
| Reliability | Consistency, availability, version stability | Variance across runs; provider SLAs; version pinning |
| Data handling | Where data is processed and retained | Your security and compliance requirements |
Decision tree
1. Can code, SQL, rules or a parser solve it reliably?
yes → do that. Stop.
2. Is the input private, current or domain-specific?
yes → plan retrieval or tools first (any model will need them).
3. Is the task extraction / classification / short transformation?
yes → start with a smaller model + structured output.
4. Does it need multi-step reasoning or judgment on ambiguous input?
yes → test a larger or reasoning model; compare latency and cost.
5. Is it high volume?
yes → favor the cheapest model that passes; consider batch processing.
6. Is it interactive?
yes → check p95 latency; consider streaming and smaller models.
7. Must the output be exact?
yes → schema validation, deterministic post-checks, escalation on failure.
Code / commands
1. Build an evaluation set first
- 50–200 real, representative cases to start, including hard and edge cases
- the expected output for each case, or a scoring rule
- no secrets; mask or synthesize sensitive values
{"id": "case-017", "input": "Invoice INV-2291 paid twice on 2026-09-03", "expected": {"label": "billing_issue"}}
Without this, model choice is opinion.
2. Score candidates on the same cases
import statistics
import time
def evaluate(call_model, model: str, cases: list[dict], is_correct) -> dict:
"""Run one candidate model over a labelled evaluation set."""
latencies, correct, cost = [], 0, 0.0
for case in cases:
started = time.monotonic()
result = call_model(model, case["input"]) # your wrapper: returns {"output": ..., "cost": ...}
latencies.append(time.monotonic() - started)
cost += result["cost"]
correct += bool(is_correct(result["output"], case["expected"]))
return {
"model": model,
"accuracy": correct / len(cases),
"p50_s": statistics.median(latencies),
"p95_s": statistics.quantiles(latencies, n=20)[18],
"cost_per_correct": cost / correct if correct else float("inf"),
}
Run each candidate more than once: outputs can vary between runs.
3. Compare on cost per successful task
cost_per_successful_task = total_cost (including retries and escalations)
/ tasks that pass validation
A cheaper model with a low pass rate can cost more per successful task than a more expensive one.
| Model | Accuracy | p95 latency | Cost per correct answer |
|---|---|---|---|
| Small | measured | measured | measured |
| Medium | measured | measured | measured |
| Large / reasoning | measured | measured | measured |
Pick the cheapest row that meets the quality and latency targets.
4. Require structured output and validate it
import json
LABELS = {"billing_issue", "access_issue", "data_issue", "other"}
def parse_label(raw: str) -> str | None:
"""Return a valid label, or None so the caller can retry or escalate."""
try:
data = json.loads(raw)
except json.JSONDecodeError:
return None
label = data.get("label") if isinstance(data, dict) else None
return label if label in LABELS else None
Use the provider’s structured output or JSON schema feature when available. Validate anyway.
5. Cascade: small model first, escalate on failure
def classify(text: str, call_model) -> str:
for model in ("small-model", "large-model"):
label = parse_label(call_model(model, text)["output"])
if label is not None:
return label
return "needs_review" # human queue, not a guess
Track the escalation rate. If most cases escalate, start with the larger model.
6. Add retrieval or tools for private and current data
- The model’s training does not include your tables, yesterday’s pipeline run or your internal definitions.
- Retrieve documents, run an approved query, or call an API, then pass the result as context.
- Choose the model for how well it uses the provided context, not for what it might remember.
Retrieval can improve grounding, but it does not guarantee correctness: bad or missing documents still produce bad answers.
7. Multimodal input
- Scanned PDFs, screenshots, charts: test a multimodal model on your real files, including poor-quality ones.
- Alternative: OCR or a document parser first, then a text model. Often cheaper and easier to debug.
- Embedding models are for search and similarity, not for generating answers.
8. Check throughput and limits
peak_tokens_per_minute = peak_requests_per_minute × average_tokens_per_request
Compare with the provider’s request and token limits for each candidate. A model that fits the budget but not the throughput still fails in production.
9. Context window size is not a quality guarantee
- A larger window lets you send more, not necessarily better.
- Long inputs cost more and are slower.
- Models can use information in very long inputs unevenly. Test with your real document sizes.
10. Pin versions and re-evaluate on changes
- Use explicit model versions where the provider offers them.
- Re-run the evaluation set before switching versions or providers.
- Keep the evaluation set in source control next to the prompts.
What the result means
| Evaluation result | Decision |
|---|---|
| Small model passes quality and latency | Use it; add escalation for failures |
| Small model fails only on hard cases | Cascade: small first, escalate |
| All models fail on the same cases | Fix the input, context or task definition first |
| Large model passes but too slow | Smaller model + better context, streaming, or async processing |
| Quality varies between runs | Lower variance settings, stricter output schema, validation |
| Code or SQL solves most cases | Use code; send only the remainder to a model |
What to look for
- Accuracy on the hard cases, not only overall accuracy.
- p95 latency with real prompt sizes.
- Validation pass rate for structured output.
- Escalation and retry rates.
- Cost per successful task.
- Data residency and retention terms.
- Behavior after a model version update.
Common mistakes
1. Using an LLM for deterministic work
Parsing a date or summing a column with a model is slower, costlier and less reliable than code.
2. Choosing from public rankings alone
General benchmarks do not measure your schemas, documents or edge cases.
3. Defaulting to the largest model
It may be unnecessary for simple tasks, and it is usually slower and more expensive.
4. Defaulting to the cheapest model
Retries, escalations and wrong answers can make it the most expensive option.
5. Expecting the model to know private data
Use retrieval or tools; do not rely on what the model might remember.
6. Trusting output without validation
Even strong models occasionally return invalid or plausible-but-wrong output.
7. Evaluating once
Outputs vary between runs and between model versions.
Production notes
- Keep the model name and version in configuration, not scattered in code.
- Log the model version with every request.
- Wrap provider calls in one interface so models can be swapped and compared.
- Keep the evaluation set and the scoring code in source control.
- Have a fallback (another model, a queue for human review, or a clear error) for provider outages.
- Review data processing and retention terms before sending business data to any hosted model.
Checklist
- Deterministic parts are handled by code, SQL or rules.
- An evaluation set with hard cases exists.
- Candidates are compared on accuracy, p95 latency and cost per successful task.
- Structured output is validated.
- Failures escalate or go to review, not to a guess.
- Private data reaches the model through retrieval or tools.
- Throughput fits provider limits at peak.
- Model versions are pinned and re-evaluated on change.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.