LLM observability: What to capture, what to trust, and how to act on it
Davis Christenhuis
LLM observability is the practice of capturing and analyzing how applications built with large language models execute, perform, and respond. It helps you understand what the model received, which steps ran, and what the application returned. Combined with evaluations and user feedback, it helps you assess whether the result met the user’s needs.
An LLM application can return an HTTP 200 response with acceptable latency and still produce an inaccurate, irrelevant, or unsafe answer. LLM observability closes that gap by connecting operational telemetry to the content and behavior of each request.
In this guide, you’ll learn which signals to capture, how to assess output quality in production, what changes when observing multi-step agents, and how to turn telemetry into a practical improvement loop.
📌 TL;DR
- LLM observability helps teams investigate whether an application is behaving as intended, not just whether it is running.
- Effective observability connects requests and execution steps with model and tool metadata, token usage, latency, cost estimates, and available quality or user-feedback signals, while limiting sensitive-content capture.
- Measuring output quality in production means combining automated checks, human review, and real user signals to understand whether responses are useful.
- Observing agents requires tracing sessions, tool calls, state changes, retries, and side effects across the full workflow.
- Production telemetry helps teams identify failures, connect them to specific changes, fix the right layer, and prevent regressions.
What is LLM observability?
LLM observability combines records of an application’s inputs, execution steps, performance, and outputs with available quality signals. It provides context for investigating how the application produced a specific outcome, during development, testing, and production.
Depending on the application, that record may include the input, prompt, model, retrieved documents, model response, tool calls, token consumption, latency, evaluation scores, and user feedback.
Traditional application performance monitoring (APM) remains useful for tracking availability, latency, request errors, and service dependencies. Those operational signals alone cannot establish whether an LLM response accurately answered the user’s question.
LLM observability adds context about prompts, retrieval, model calls, tool execution, and outputs, which teams can combine with evaluations and user feedback. Some APM platforms now provide these capabilities, so the distinction is about the evidence collected, not a strict divide between separate categories of tools.
LLM applications require this additional layer because their failures do not always generate conventional error signals. A model can confidently invent a detail, retrieve the wrong document, expose sensitive information, select an inappropriate tool, or consume far more tokens than expected without crashing.
Three characteristics make these failures particularly difficult to diagnose:
- Non-deterministic outputs: The same input can produce different wording, reasoning paths, and outcomes, so exact-output assertions have limited value outside tightly constrained tasks.
- Silent failures: The application can return a valid response even when its answer is unsupported, irrelevant, or inconsistent with the user’s intent.
- Behavioral drift: Changes to a prompt, model, tool definition, or data source can alter the application’s behavior without producing a conventional error. The underlying data can also change even when the application code does not.
Key components of LLM observability
An effective observability setup starts with a structured record of each request. Aggregate dashboards matter later, but they cannot explain a problem unless you can move from a metric to the specific request, trace, and span behind it.
Tracing and spans
A trace should represent the complete execution path triggered by a user request. Its spans represent individual operations, such as retrieving documents, assembling a prompt, calling a model, executing a tool, or processing the final response.
Consider a knowledge agent answering a question about an internal expense policy. The request may trigger a query rewrite, retrieve several documents, rerank the results, assemble a prompt, call a model, and add citations to the response. If the final answer is wrong, recording the model response alone does not tell you whether retrieval returned irrelevant documents, prompt assembly dropped important context, or the model ignored valid evidence.
The trace provides that causal structure. It lets you inspect the full request while preserving the timing, inputs, outputs, and relationships between its steps.
A collection of independent log lines rarely provides the same clarity. Engineers have to reconstruct the sequence manually, often without reliable identifiers connecting retrieval, generation, and tool activity. A structured trace preserves those relationships from the start.
What to capture, and when
The exact capture set depends on your application, data sensitivity, and debugging needs. Start with enough metadata to connect an outcome to its execution, then add content and quality signals where necessary and permitted.
- Request context: trace and span identifiers, operation type, timestamps, status, application or prompt version, and a policy-approved user or conversation identifier where needed.
- Model execution: available provider and model identifiers, supported request settings, latency, and time to first token where relevant.
- Retrieval and tools: document and tool identifiers, available relevance scores, execution times, errors, and retries. Capture arguments and results only where necessary and permitted.
- Consumption: provider-reported token usage where available, clearly labeled estimates otherwise, and the responsible feature or workflow. Account for applicable cache, reasoning-token, tool, and pricing differences when estimating cost.
- Outcome: available evaluation results, safety-check results, user feedback, and task-completion evidence, linked to the relevant execution. Retain response content only where necessary and permitted.
Raw inputs, conversation history, retrieved content, tool arguments, tool results, and responses can contain personal or confidential information. Do not log them by default. Omit or redact sensitive fields before they enter telemetry storage, restrict access to retained payloads, and set appropriate sampling and retention policies.
Quality and safety signals
Operational telemetry explains what the system did. Quality signals help determine whether it did the right thing.
You can attach automated evaluation scores, human labels, or user feedback to a trace. Quality and security checks may inspect permitted inputs, retrieved content, tool arguments and results, and outputs for signs of prompt injection, sensitive-data exposure, policy violations, or unsupported claims. These checks provide evidence for investigation, not a guarantee of safety. For agents that take actions, review authorization, approvals, and resulting state changes as well.
These scores become more useful when you can aggregate them and return to the underlying traces. Instead of alerting only when latency exceeds a threshold, you might alert when the proportion of failed policy checks rises or when feedback deteriorates after a release.
This mechanism does not make every quality judgment reliable. It creates a common structure in which quality evidence can be recorded, compared, and investigated. The more difficult question is which evidence your team should trust.
Measuring output quality in production
You usually need both offline evaluation and production observation. They cover different failure surfaces.
Offline evaluation runs controlled tests against a curated dataset rather than evaluating the live request stream. Teams use it before deployment and rerun it after changes to prompts, models, tools, or application logic.
A versioned dataset containing representative examples, edge cases, and previously observed failures can catch regressions within its coverage and support comparisons between configurations. Production findings should feed back into that dataset as user needs and application behavior evolve.
Production evaluation captures the requests, contexts, and user goals that your test set did not anticipate. Three mechanisms are commonly used:
- Automated evaluators apply code-based checks or model-based judgments to live or sampled traces.
- Human review asks people to inspect representative or high-risk conversations.
- User signals capture explicit feedback, retries, follow-up corrections, abandonment, and usage patterns.
These signals can be attached to individual traces, aggregated over time, and used to trigger alerts. A team might monitor the rate of failed policy checks over a rolling window, for example, and investigate when it crosses a defined threshold.
Automated scoring is useful when the criterion and evidence are well defined. Deterministic checks can validate a JSON schema, inspect tool names and arguments, or query an external system to verify that an intended record exists with the expected fields.
A recorded tool call or generated success message alone is not proof that the action succeeded. Model-based judges can help assess relevance, tone, or faithfulness when they have an appropriate rubric and access to the evidence needed for that judgment.
Human and user evidence remains important because not every useful outcome can be reduced to a stable rubric. A response can look strong in isolation while still failing to resolve the user’s actual task.
Where automated quality scoring breaks down
Automated scores are only useful when they measure the property the team actually cares about. An evaluator needs a clear task-specific rubric and the relevant evidence, which may include the user’s request, retrieved sources, tool results, and resulting system state. Without that context, it can reject acceptable behavior or approve an answer that failed the task.
Both kinds of error matter. False positives can waste review effort and weaken trust in alerts; false negatives can allow harmful or incorrect behavior to pass unnoticed. Model-based judges can also exhibit biases or be manipulated by adversarial content. Calibrate automated assessments against human-reviewed examples, test known failure cases, and check that improving a score corresponds to improving the intended outcome.
Dust made a similar trade-off when designing its agent observability. The company chose not to make dataset evaluation or LLM-as-a-judge the primary way to understand agents in production. That does not mean these methods are useless. Dataset evaluation can catch known regressions, validate model upgrades, and work well for stable use cases. Model-based judges can also assess specific criteria when the evaluation is reliable enough.
Dust emphasizes production behavior as a way to understand agent usefulness. Its built-in Insights show usage, feedback, tool behavior, retrieval patterns, and changes across agent versions. Retries and abandonment can provide additional clues, but these behavioral signals remain imperfect proxies for successful task completion.
Use automated evaluation where the criteria are clear and its error rates are acceptable for the decision being made. Assess both false positives and false negatives, retain human review for consequential or ambiguous cases, and avoid treating a single score as a complete measure of production quality.
Observability for agents and multi-step systems
Agent observability extends beyond individual LLM calls because an agent can plan, retrieve information, select tools, change state, and take actions across multiple turns. A session groups related traces across those turns, making it possible to inspect whether the agent preserved context, selected the right tools, and completed the overall task.
This also exposes failures that a single-call trace misses, such as tool loops, malformed arguments, lost state, partial workflows, and actions that never occurred. When an agent creates a ticket or updates a record, the generated confirmation is not enough. The trace should show the tool result and, where possible, the resulting record or status.
From telemetry to action: an operating loop
Observability only creates value when it changes how your team diagnoses failures and improves the application. A practical operating loop looks like this:
- Instrument the execution path: Capture enough context to connect the user request to retrieval, model calls, tool calls, and the final outcome.
- Collect feedback from real users: Add simple ways for people to react to or correct an output. Also look at implicit signals such as retries, rephrased questions, abandoned sessions, and declining repeat usage.
- Connect signals to changes: Record the agent version, prompt version, model, data-source changes, and deployment dates. This makes it easier to understand whether a change in feedback, usage, or cost followed a prompt edit, a model switch, or a retrieval change. Dust’s Agent Builder Insights lets builders examine usage, feedback, tool behavior, and retrieval patterns alongside agent versions. Separately, Admin Analytics lets workspace admins and managers analyze credit consumption by model and other dimensions. These views can help teams investigate what changed after a model update, but a before-and-after comparison does not by itself establish that the model change caused the outcome.
- Inspect failing traces: Aggregate metrics show that something changed, but individual traces help explain why. Review examples across users, request types, and versions.
- Fix the layer that caused the problem: The right intervention may involve retrieval, instructions, prompt assembly, model selection, tool definitions, or application logic.
- Add the failure to a regression set: Once you understand the problem, add a sanitized version to your tests and verify that the fix does not break known-good behavior.
See how you can use Dust to understand AI in production. Request a demo →
Frequently asked questions (FAQs)
Is LLM observability the same as LLM monitoring?
The terms often overlap, but observability generally implies the ability to investigate why behavior occurred, not only track predefined metrics. Monitoring may show that latency or negative feedback increased. Observability connects that change to the relevant traces, versions, inputs, retrieval steps, and tool calls.
How is observability different from evals?
Evaluations assess an application or output against defined criteria. Observability captures the execution context needed to investigate its behavior during development, testing, and production. Evaluation results can be linked to that context, alongside traces, user feedback, usage patterns, version history, and operational metrics.
Can observability detect prompt injection or data leakage?
Observability can record or score signs of prompt injection and sensitive-data exposure, but capture alone does not prevent them. Teams still need input controls, permissions, output checks, tool approval rules, and incident processes. Sensitive payloads should also be redacted before they enter the telemetry system.
How much cost or latency does LLM observability add?
The overhead depends on payload size, sampling, evaluator usage, storage, and where processing occurs. Basic asynchronous trace export may have limited application impact, while running additional model-based evaluators on every request can add meaningful cost. Sample selectively and reserve expensive checks for high-risk workflows.