Using LLMs with Your Data: Practical Patterns

An LLM can produce fluent language, but fluency is not evidence that it knows your organization’s current data, definitions, or access rules. A useful data application must retrieve or compute the relevant facts, authorize the user to see them, and constrain what the model can do with them.

That makes the LLM one component of an application architecture. The application still needs ordinary engineering: APIs, identity, permissions, schemas, validation, monitoring, tests, and deterministic code. The patterns below focus on those boundaries rather than on model hype.

The important distinction

A model’s training gives it general capabilities and public knowledge up to its training process. It does not automatically understand a private sales table, yesterday’s pipeline run, an internal policy, or the business meaning of “active customer.” Even if similar information appeared during training, it should not be treated as an authoritative copy of enterprise data.

Applications provide controlled context at request time. That context may come from retrieved documents, a SQL result, a semantic metric service, or an approved application function. The mechanism should return the smallest amount of relevant, authorized information needed for the task.

This separation is valuable. The data platform remains the system of record. Deterministic services perform calculations and enforce permissions. The LLM interprets a question, chooses among allowed capabilities, and expresses a result. It should not become an ungoverned alternate database.

Pattern 1: Retrieval-Augmented Generation

Retrieval-Augmented Generation, or RAG, retrieves relevant context before asking the model to answer:

User question
   ↓
Retrieve relevant context
   ↓
Construct bounded prompt and context
   ↓
LLM
   ↓
Response with supporting references

For unstructured data, retrieval often searches chunks of documents using keywords, embeddings, metadata filters, or a combination. Chunking should respect useful document boundaries: a complete policy section is generally better context than arbitrary fragments that omit definitions or exceptions. Store source identifiers and access metadata with each chunk so the application can authorize and cite it.

Structured data can also participate in retrieval. A catalog may return table descriptions, metric definitions, schema documentation, or precomputed summaries. Retrieving a schema is not the same as retrieving the answer, however. If the question asks for current revenue, a controlled query or metric function should calculate it from the source of record.

RAG quality depends on more than the model. Evaluate whether the right source was retrieved, whether the context contains the answer, whether the response is supported by that context, and whether citations point to accessible material. A polished answer based on the wrong document is still wrong.

Pattern 2: LLM plus SQL

Text-to-SQL can make data exploration more approachable, but directly executing arbitrary model output is dangerous. The query could access unauthorized data, perform expensive scans, expose sensitive columns, or modify the database.

A controlled flow separates interpretation, validation, execution, and explanation:

User question
   ↓
Classify intent and identify an approved data domain
   ↓
Generate a constrained query or choose a query template
   ↓
Validate tables, columns, operations, scope, and cost controls
   ↓
Execute through a read-only identity
   ↓
Return a bounded result
   ↓
LLM explains the result

Prefer an allow-list of schemas, views, and operations. Use a parser or database-aware validation layer rather than searching SQL text for unsafe words. Enforce read-only credentials, row limits, statement timeouts, and resource governance at the database boundary. Authorization must be applied before execution; asking the model to remember security rules is not an authorization control.

For well-known questions, query templates or stored procedures are safer and easier to test than open-ended generation. The model can map “show last month’s regional sales” to an approved report function and supply validated parameters. Reserve flexible SQL generation for environments where its value justifies the additional review and containment.

Pattern 3: A semantic layer

Database schemas encode storage, not always business meaning. A model may see net_amt without knowing whether it includes tax, returns, or currency conversion. A semantic layer gives the application a curated representation of metrics, fields, relationships, and definitions.

Useful semantic context includes:

• approved business metrics and their calculation ownership;
• human-readable field descriptions and units;
• allowed tables or views;
• valid relationships and join directions;
• synonyms used by the business;
• freshness, grain, and effective-date rules;
• fields that must not be exposed to the model.

This layer reduces ambiguity and creates a contract that can be tested independently of the prompt. It also allows the same metric definition to serve reports, APIs, and LLM-powered experiences. Do not duplicate a critical calculation in prompt prose if a governed metric service can compute it deterministically.

Pattern 4: Tools and function calling

Tools let the model request controlled application actions instead of receiving unrestricted system access. The application publishes a small capability with a typed input and output contract:

get_sales_summary(start_date, end_date, region)
get_customer_metrics(customer_id)
run_approved_report(report_name, parameters)

The model may propose a tool call, but the application validates its arguments, checks the user’s permissions, invokes the implementation, and returns a bounded result. A tool should represent a business capability, not a disguised unrestricted SQL console or shell.

Keep tools narrow enough to test. Document units and date semantics. Return structured errors that distinguish an invalid parameter, unauthorized request, missing data, and dependency failure. The model can then explain the outcome without inventing a result.

Tool calling also improves observability. The application knows which capability ran, with which approved parameters, against which data source. That is much easier to audit than trying to infer behavior from one large prompt.

Security is an application responsibility

Authenticate the user before accessing enterprise data. Authorize each retrieval, query, or tool call against that identity. Apply row-level and tenant-level controls in trusted application or data-platform layers, not only in natural-language instructions.

Use separate identities with least privilege. Store secrets in an appropriate secret manager or environment configuration and never place them in prompts. Treat retrieved documents and database values as untrusted input: they can contain instructions designed to manipulate the model. Prompt injection is not limited to the user’s question; it can be embedded in a document the system retrieves.

Define which sensitive fields may enter model context and which must be masked, aggregated, or excluded. Consider model-provider data handling, retention, regional requirements, and logging behavior. Tenant isolation must survive every path, including caches, traces, error messages, and evaluation datasets.

Finally, validate output before it drives an external action. A generated explanation may be shown with citations; a high-impact update, notification, or financial action needs deterministic checks and often explicit human approval.

A structured data example

The following example keeps data access behind an approved function. The model client is deliberately abstract: credentials and provider configuration belong in environment or managed configuration, not in source code.

from dataclasses import dataclass, asdict
import json
import os

@dataclass
class SalesSummary:
    period: str
    region: str
    order_count: int
    net_sales: float

def get_sales_summary(user, period: str, region: str) -> SalesSummary:
    if not user.can_read_region(region):
        raise PermissionError("Region is not authorized")

    # In production, call a parameterized query or governed metric service.
    return reporting_service.sales_summary(period=period, region=region)

def answer_sales_question(user, question: str, period: str, region: str):
    summary = get_sales_summary(user, period, region)
    context = json.dumps(asdict(summary))

    prompt = f"""Explain this approved sales summary.
Use only the supplied JSON. State when the data is insufficient.

Question: {question}
Data: {context}
"""

    model_name = os.environ["LLM_MODEL_NAME"]
    return model_client.generate(model=model_name, prompt=prompt)

The example is not complete production code, but its boundaries matter: permission is checked before data retrieval, the data function returns a defined structure, the prompt restricts its evidence, and no API key is embedded in the program. A production implementation would add input validation, timeouts, retry policy, output checks, and structured telemetry.

Large datasets stay in the data platform

Do not send billions of rows—or even a large raw extract—to an LLM. Models are not query engines, and context windows are not a substitute for storage and compute architecture.

Filter by the user’s authorized scope, calculate metrics in SQL or Spark, aggregate to the required grain, and retrieve only relevant documents or records. Then give the model the compact result needed to explain or compare. If the task requires exact arithmetic, perform it in code and let the model communicate the result.

This division improves correctness, latency, security, and cost. It also makes testing clearer: the data function can be tested for exact values, while the language layer can be evaluated for faithful explanation.

Cache carefully

Caching can reduce repeated data work and model calls. Useful candidates include repeated approved reports, retrieval results for stable documents, embeddings, and generated summaries whose source data has not changed.

Every cache key must include the dimensions that affect authorization and meaning: tenant, user scope or role, parameters, source version, model version, and prompt or tool version where relevant. A cache that ignores tenant identity can become a data leak.

Define freshness explicitly. A daily management summary and a live incident assistant have different tolerances. Invalidate cached explanations when their underlying metric or document version changes. Cache deterministic data results separately from generated wording when that gives you clearer control.

Observability without careless data collection

An LLM application needs telemetry across both conventional services and model interactions. A useful request record may contain:

• request or correlation ID;
• authenticated tenant and a non-sensitive user reference;
• model and configuration version;
• total latency and dependency latency;
• selected tools and their outcomes;
• data source and document identifiers;
• token usage;
• validation or safety decision;
• sanitized error category.

Do not blindly log full prompts, retrieved context, or model responses. They may contain personal information, secrets, or business data. Prefer structured metadata, redaction, controlled sampling, and access-restricted diagnostic storage. Make detailed content capture an explicit policy decision rather than a default SDK setting.

Trace the request across retrieval, SQL, tools, and model calls with one correlation ID. This lets an engineer distinguish slow retrieval from slow generation and identify whether an incorrect answer began with missing context, a faulty query, or unsupported model output.

Control cost through architecture

Cost is shaped by request volume, context size, output size, model choice, retries, and repeated work. Use the least complex model that meets the evaluated requirement. Retrieve fewer, better context items. Summarize stable documents once where appropriate. Put deterministic routing and validation outside the model.

Avoid sending the same schema catalog or policy collection with every request if a narrower tool contract will do. Set response limits. Detect retry loops. Measure cost per useful task rather than only cost per model call. Provider pricing changes, so architecture should expose usage and configuration rather than embed assumptions about a current price table.

A practical architecture

1. User

   • UI or API client

2. Application

   • Application / orchestrationOwns the request and the conversation

3. Auth

   • Authentication
   • Authorization

   Trust boundary: applied before any data leaves its governed layer.

4. Tools

   • Retrieval
   • Controlled SQL
   • Tool / function calls

5. Data platform

   • Governed data platformRow- and column-level security

6. LLM

   • LLMReceives bounded, selected context only

7. Response

   • Validated responseChecked before display or action

Cross-cutting concerns

• Audit logging
• Evaluation
• Cost limits
• Caching

The exact order can vary—a tool may call the data platform and then the model, for example—but the trust boundaries should remain visible. Identity and authorization apply before data leaves its governed layer. The model receives selected context. Output validation occurs before a response is presented or an action is proposed.

The planned roadmap includes related work on Python and Azure Functions, reporting APIs, and production data-platform observability.

When not to use an LLM

Use a normal SQL query when the user needs an exact table or metric and natural-language interpretation adds no value. Use deterministic business rules for eligibility, pricing, compliance, and other decisions that must be predictable and auditable. Use conventional search when keyword or faceted retrieval already solves the problem. Perform calculations in code rather than asking a model to simulate a calculator.

An LLM is valuable when language is central: interpreting varied questions, synthesizing a small set of authorized sources, explaining a structured result, or helping a user select among controlled capabilities. It is unnecessary risk when the requirement is already precise and deterministic.

Closing thoughts

Useful LLM data applications do not bypass data engineering; they depend on it. Governed definitions, reliable queries, authentication, tenant isolation, observability, and tested tools make the language layer trustworthy enough to use.

Treat the model as an interface and reasoning component, not the data platform or security boundary. Keep facts in systems of record, calculations in deterministic services, permissions in trusted controls, and the context supplied to the model as small and explicit as the task allows.