Browse cheat sheets · AI Engineering for Data

Technology: AI Engineering for DataCategory: Model SelectionDifficulty: BeginnerUpdated

Choosing the Right AI Model

A practical framework for choosing an AI model for data applications: when not to use an LLM, which dimensions matter, a decision tree, and how to evaluate candidates on quality, latency and cost per successful task.

On this page
  1. Problem
  2. Quick answer
  3. First: do you need a model at all?
  4. Comparison dimensions
  5. Decision tree
  6. Code / commands
  7. 1. Build an evaluation set first
  8. 2. Score candidates on the same cases
  9. 3. Compare on cost per successful task
  10. 4. Require structured output and validate it
  11. 5. Cascade: small model first, escalate on failure
  12. 6. Add retrieval or tools for private and current data
  13. 7. Multimodal input
  14. 8. Check throughput and limits
  15. 9. Context window size is not a quality guarantee
  16. 10. Pin versions and re-evaluate on changes
  17. What the result means
  18. What to look for
  19. Common mistakes
  20. 1. Using an LLM for deterministic work
  21. 2. Choosing from public rankings alone
  22. 3. Defaulting to the largest model
  23. 4. Defaulting to the cheapest model
  24. 5. Expecting the model to know private data
  25. 6. Trusting output without validation
  26. 7. Evaluating once
  27. Production notes
  28. Checklist
  29. Next step
  30. Related deep dives

Problem

You need a model for a data task: extract fields from documents, classify tickets, generate SQL, summarize incidents, answer questions over internal data.

There are many options: small and large models, reasoning models, multimodal models, embedding models, hosted and self-hosted. Each trades quality against latency, cost, throughput and operational effort.

No model is the best choice for every task. The right choice is the cheapest, fastest option that meets your quality bar on your own data, and sometimes that option is not an LLM at all.

Quick answer

Is the task deterministic (parse, format, compute, look up)?
    → normal code, SQL, rules or a parser. No LLM.

Simple extraction or classification, high volume?
    → smaller, cheaper model + structured output + validation.

Complex reasoning, ambiguous input, multi-step analysis?
    → larger or reasoning model; measure the latency and cost.

Needs current, private or domain data?
    → model + retrieval or tools. The model does not "know" your data.

Needs exact, machine-readable output?
    → structured output + schema validation + retry or escalation.

Input includes images, scans, charts?
    → multimodal model, or OCR/parsing first and a text model after.

Then confirm with an evaluation set: accuracy, p95 latency and cost per successful task.

First: do you need a model at all?

Task Usually better without an LLM
Parse dates, emails, IDs, known formats Parser, regex, library
Aggregations, totals, filters SQL
Lookups against reference data Join / dictionary lookup
Fixed business rules Rules in code
Exact arithmetic Code
Format conversion (CSV → Parquet, JSON → table) Code

An LLM fits when the input is unstructured or ambiguous, the rules are hard to write down, or the output is natural language.

A common robust design: code for everything deterministic, the model only for the ambiguous part.

Comparison dimensions

Dimension What it means How to evaluate it
Capability / size General quality on complex tasks Your evaluation set, not public rankings alone
Reasoning depth Multi-step logic, planning, analysis Hard cases from your domain
Latency Time to first token and to full answer p50 and p95 with your real prompt sizes
Cost Price per input/output token, tokens used Cost per successful task
Context window How much input fits in one request Your largest real request plus output
Structured output Reliably returns valid JSON / schema Validation pass rate
Tool use Calls functions/APIs correctly Correct tool and arguments rate
Multimodal support Images, documents, audio Your real files, including bad scans
Throughput Requests and tokens per minute available Provider limits vs your peak load
Reliability Consistency, availability, version stability Variance across runs; provider SLAs; version pinning
Data handling Where data is processed and retained Your security and compliance requirements

Decision tree

1. Can code, SQL, rules or a parser solve it reliably?
   yes → do that. Stop.
2. Is the input private, current or domain-specific?
   yes → plan retrieval or tools first (any model will need them).
3. Is the task extraction / classification / short transformation?
   yes → start with a smaller model + structured output.
4. Does it need multi-step reasoning or judgment on ambiguous input?
   yes → test a larger or reasoning model; compare latency and cost.
5. Is it high volume?
   yes → favor the cheapest model that passes; consider batch processing.
6. Is it interactive?
   yes → check p95 latency; consider streaming and smaller models.
7. Must the output be exact?
   yes → schema validation, deterministic post-checks, escalation on failure.

Code / commands

1. Build an evaluation set first

  • 50–200 real, representative cases to start, including hard and edge cases
  • the expected output for each case, or a scoring rule
  • no secrets; mask or synthesize sensitive values
{"id": "case-017", "input": "Invoice INV-2291 paid twice on 2026-09-03", "expected": {"label": "billing_issue"}}

Without this, model choice is opinion.


2. Score candidates on the same cases

import statistics
import time


def evaluate(call_model, model: str, cases: list[dict], is_correct) -> dict:
    """Run one candidate model over a labelled evaluation set."""
    latencies, correct, cost = [], 0, 0.0
    for case in cases:
        started = time.monotonic()
        result = call_model(model, case["input"])  # your wrapper: returns {"output": ..., "cost": ...}
        latencies.append(time.monotonic() - started)
        cost += result["cost"]
        correct += bool(is_correct(result["output"], case["expected"]))
    return {
        "model": model,
        "accuracy": correct / len(cases),
        "p50_s": statistics.median(latencies),
        "p95_s": statistics.quantiles(latencies, n=20)[18],
        "cost_per_correct": cost / correct if correct else float("inf"),
    }

Run each candidate more than once: outputs can vary between runs.


3. Compare on cost per successful task

cost_per_successful_task = total_cost (including retries and escalations)
                         / tasks that pass validation

A cheaper model with a low pass rate can cost more per successful task than a more expensive one.

Model Accuracy p95 latency Cost per correct answer
Small measured measured measured
Medium measured measured measured
Large / reasoning measured measured measured

Pick the cheapest row that meets the quality and latency targets.


4. Require structured output and validate it

import json

LABELS = {"billing_issue", "access_issue", "data_issue", "other"}


def parse_label(raw: str) -> str | None:
    """Return a valid label, or None so the caller can retry or escalate."""
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return None
    label = data.get("label") if isinstance(data, dict) else None
    return label if label in LABELS else None

Use the provider’s structured output or JSON schema feature when available. Validate anyway.


5. Cascade: small model first, escalate on failure

def classify(text: str, call_model) -> str:
    for model in ("small-model", "large-model"):
        label = parse_label(call_model(model, text)["output"])
        if label is not None:
            return label
    return "needs_review"   # human queue, not a guess

Track the escalation rate. If most cases escalate, start with the larger model.


6. Add retrieval or tools for private and current data

  • The model’s training does not include your tables, yesterday’s pipeline run or your internal definitions.
  • Retrieve documents, run an approved query, or call an API, then pass the result as context.
  • Choose the model for how well it uses the provided context, not for what it might remember.

Retrieval can improve grounding, but it does not guarantee correctness: bad or missing documents still produce bad answers.


7. Multimodal input

  • Scanned PDFs, screenshots, charts: test a multimodal model on your real files, including poor-quality ones.
  • Alternative: OCR or a document parser first, then a text model. Often cheaper and easier to debug.
  • Embedding models are for search and similarity, not for generating answers.

8. Check throughput and limits

peak_tokens_per_minute = peak_requests_per_minute × average_tokens_per_request

Compare with the provider’s request and token limits for each candidate. A model that fits the budget but not the throughput still fails in production.


9. Context window size is not a quality guarantee

  • A larger window lets you send more, not necessarily better.
  • Long inputs cost more and are slower.
  • Models can use information in very long inputs unevenly. Test with your real document sizes.

10. Pin versions and re-evaluate on changes

  • Use explicit model versions where the provider offers them.
  • Re-run the evaluation set before switching versions or providers.
  • Keep the evaluation set in source control next to the prompts.

What the result means

Evaluation result Decision
Small model passes quality and latency Use it; add escalation for failures
Small model fails only on hard cases Cascade: small first, escalate
All models fail on the same cases Fix the input, context or task definition first
Large model passes but too slow Smaller model + better context, streaming, or async processing
Quality varies between runs Lower variance settings, stricter output schema, validation
Code or SQL solves most cases Use code; send only the remainder to a model

What to look for

  • Accuracy on the hard cases, not only overall accuracy.
  • p95 latency with real prompt sizes.
  • Validation pass rate for structured output.
  • Escalation and retry rates.
  • Cost per successful task.
  • Data residency and retention terms.
  • Behavior after a model version update.

Common mistakes

1. Using an LLM for deterministic work

Parsing a date or summing a column with a model is slower, costlier and less reliable than code.

2. Choosing from public rankings alone

General benchmarks do not measure your schemas, documents or edge cases.

3. Defaulting to the largest model

It may be unnecessary for simple tasks, and it is usually slower and more expensive.

4. Defaulting to the cheapest model

Retries, escalations and wrong answers can make it the most expensive option.

5. Expecting the model to know private data

Use retrieval or tools; do not rely on what the model might remember.

6. Trusting output without validation

Even strong models occasionally return invalid or plausible-but-wrong output.

7. Evaluating once

Outputs vary between runs and between model versions.

Production notes

  • Keep the model name and version in configuration, not scattered in code.
  • Log the model version with every request.
  • Wrap provider calls in one interface so models can be swapped and compared.
  • Keep the evaluation set and the scoring code in source control.
  • Have a fallback (another model, a queue for human review, or a clear error) for provider outages.
  • Review data processing and retention terms before sending business data to any hosted model.

Checklist

  • Deterministic parts are handled by code, SQL or rules.
  • An evaluation set with hard cases exists.
  • Candidates are compared on accuracy, p95 latency and cost per successful task.
  • Structured output is validated.
  • Failures escalate or go to review, not to a guess.
  • Private data reaches the model through retrieval or tools.
  • Throughput fits provider limits at peak.
  • Model versions are pinned and re-evaluated on change.

Next step

  • Need to estimate the cost of the chosen model → LLM Cost & Token Budgeting.
  • The model needs better instructions for data tasks → Prompt Patterns for Data Engineering.
  • The model needs better input, not a bigger model → LLM Context Engineering.

← All cheat sheets