Observability for AI Agents: Tracing Why Your Agent Made That Call

The agent was given a task: summarize the five most recent support tickets for a customer account and suggest a response category for each. It returned summaries that were accurate and response categories that were reasonable. The team shipped it.

Three weeks later, a support engineer noticed that for accounts with more than 500 tickets, the agent was consistently returning summaries of the oldest five tickets instead of the newest five. The agent’s outputs looked correct. The logic was wrong. Nobody had traced what the retrieval step actually returned, so nobody caught that the sort order was reversed.

Observability for AI agents is not primarily about catching errors. Errors are visible: the agent fails, throws an exception, returns something clearly wrong. The hard problem is the silent failure, where the agent does something plausible but incorrect, and the only way to catch it is to understand why it made the decisions it made, which means tracing the inputs and outputs at each step of the process.

Traditional application observability is designed around deterministic systems: a request comes in, code runs, a response goes out, and you can log the request, the code path, and the response to reproduce any failure. Agentic systems are non-deterministic and multi-step. The agent makes decisions at runtime, and the trace you need is not just the inputs and outputs to the system but the intermediate reasoning steps, tool call inputs and outputs, context at each decision point, and the path through the agent’s decision tree.

Why Standard Logging Is Not Enough

Standard application logging records events: “function called,” “API request made,” “response received.” For a deterministic system, these events are enough to reconstruct what happened. For an agentic system, they are missing the most important information.

When an agent calls a tool, the tool input is not just a function argument, it is the result of a reasoning step. The agent decided, based on its current context, which tool to call and what to pass to it. That decision is the event that matters, and standard logging does not capture the reasoning that produced it, only the call itself.

When an agent returns a result, the result is not just a function return value, it is a synthesis over multiple intermediate steps, tool outputs, and context frames. If the result is wrong, you cannot debug it by looking only at the final output and the initial input. You need the intermediate steps.

Agentic observability requires tracing rather than logging: a hierarchical representation of the agent’s execution where each decision point, tool call, and context frame is captured as a span within a parent trace, with the inputs and outputs at each level recorded and queryable.

The distinction matters for debugging. A support ticket that says “the agent returned wrong summaries” is unsolvable without the trace. With the trace, you can see: the retrieval tool was called with sort=ASC when the intent was sort=DESC, the summaries were generated from the retrieved results without any validation, and the context window at the time of retrieval did not include the sort direction specification. The fix is clear, the trace made it findable.

The Anatomy of an Agent Trace

A well-structured agent trace has four levels of span.

The root span represents the top-level task: the user’s request, the agent’s configuration, and the overall outcome. It records the task description, the agent version, the model being used, the total latency, and whether the task completed successfully. The root span is what you look at first when a user reports a problem. It tells you whether the failure was at the task level or at a sub-step.

Tool call spans are children of the root span, one per tool invocation. Each tool call span records the tool name, the inputs to the tool, the outputs from the tool, and the latency. Tool call inputs and outputs are the most valuable observability data in an agentic system, because they show you what the agent decided to ask for and what it received. Discrepancies between what was asked and what was appropriate are the most common source of silent failures.

Reasoning spans capture the model’s chain of thought at key decision points, where chain-of-thought reasoning is enabled. For agents using chain-of-thought prompting, the intermediate reasoning is a first-class artifact that should be captured and stored, not discarded after the model produces the final output. The reasoning span is what tells you why the agent called a specific tool, which is not always obvious from the tool call input alone.

Context frames are snapshots of the agent’s context window at key points in the execution. The context frame at the start of a tool call shows you exactly what the agent knew when it made the decision. Context frames are expensive to store at full size, but storing the key variables in the context (the current task, the previous tool outputs, the agent’s current plan) at each decision point is far less expensive and nearly as useful for debugging.

Tooling for Agentic Observability

The agentic observability landscape has three categories of tools, each covering a different part of the problem.

LLM-native tracing tools are built specifically for tracing language model calls and agentic workflows. LangSmith (from LangChain) traces agents built with LangChain and LangGraph, capturing the chain of tool calls, the model inputs and outputs at each step, and the agent’s reasoning traces where available. Weights and Biases Weave provides similar capabilities with a stronger emphasis on production monitoring and eval integration. These tools are the right choice if you are building on a framework they support, because the integration is shallow and the trace quality is high.

OpenTelemetry-based instrumentation works for agents built outside of framework ecosystems. OpenTelemetry is the CNCF standard for distributed tracing, and it has Python and JavaScript SDKs that you can use to instrument any application. The challenge is that standard OTEL spans are not designed to capture the specific structure of agentic workflows (tool calls, context frames, reasoning spans), so you need to define a schema for how agentic concepts map to OTEL spans and attributes. This is more work than a native integration but produces traces that flow naturally into any OTEL-compatible backend (Jaeger, Honeycomb, Grafana Tempo, Datadog).

Semantic conventions for LLM observability are emerging from the OpenTelemetry community. The gen_ai.* semantic conventions define standard attribute names for LLM span metadata: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. Using these conventions means your traces are interoperable with tooling that understands the conventions, and your dashboards and alerts can be built on a stable attribute schema rather than custom strings.

For new projects, start with a native tracing tool if you are using a supported framework. For existing projects with established OTEL infrastructure, extend the existing instrumentation with the gen_ai.* conventions. Avoid creating a parallel observability stack for AI features; integrating into existing infrastructure makes the traces more useful because the team already knows how to query the existing stack.

Capturing the Right Data at Each Step

What you capture in each span matters as much as that you capture it. Over-capturing makes traces expensive to store and slow to search. Under-capturing leaves you without the context to debug failures.

For tool call spans: always capture the tool name, the input parameters, the output size, and the latency. For the output content: capture the full output for tools that return small structured data (IDs, metadata, status), and capture a summary or hash plus the first 500 characters for tools that return large text. Full output capture for large tools can make traces orders of magnitude larger and more expensive to store. The summary is usually enough to diagnose “did the retrieval return relevant results?” without storing the full retrieved text in the trace.

For reasoning spans: capture the chain-of-thought output at decision points where the agent’s reasoning affected a downstream action. Storing all chain-of-thought output is expensive; storing it at the moments when the agent made a tool selection or a branching decision is targeted and proportionate.

For context frames: capture the count of tokens in context at each decision point, the presence or absence of key context variables, and any truncation events (where the context was clipped because it exceeded the model’s context window). Context truncation is a significant failure mode that is invisible without this data: the agent may be making decisions with incomplete information because a key part of its context was truncated, and this will not appear in any of the tool call spans.

Token usage should be captured at every model call: input tokens, output tokens, and the model version. Token usage lets you track cost attribution by task type, identify prompts that are growing unexpectedly, and spot model version changes that affect output length.

Querying Traces for QA

Traces are only useful if you query them systematically rather than only during incident investigations.

For QA purposes, the most useful queries are: traces where a specific tool was called with inputs outside an expected range, traces where the total latency exceeded a threshold, traces where a specific error was raised in a specific span, and traces where the agent called a tool more times than expected for a given task type (which can indicate a loop or a planning failure).

Build these queries as dashboards and alerts before you need them for incident investigation. A dashboard that shows the distribution of tool call counts per task type over the past week will surface a spike in unexpected tool calls before any user reports a problem.

For regression testing after agent changes, run the agent against a fixed set of reference tasks and compare the traces to a recorded baseline. A trace comparison for a well-defined task should show the same sequence of tool calls with similar inputs. A trace that diverges from the baseline (different tool sequence, different sort order in a retrieval query, unexpected additional tool calls) is a regression signal even if the output looks correct.

From Observability to Understanding

The agent trace for the support ticket scenario that opened this post would have shown, within minutes of investigation: the retrieval tool was called, the inputs included a sort parameter, the sort parameter value was ASC, the expected value was DESC. That is a one-line fix that would have been a multi-week mystery without the trace.

This is what observability gives you for AI agents: the ability to go from “the agent did the wrong thing” to “here is specifically why it did the wrong thing” without guessing. The non-determinism of the agent means you cannot always reproduce a failure by running the same input again. The trace is the record of what actually happened, not a reconstruction from a deterministic replay.

Building this observability layer before you need it is the practice. The team that instruments their agent on day one and has three months of traces when the first production incident occurs can diagnose the incident in an hour. The team that starts instrumenting after the incident spends the time trying to reproduce a failure they can no longer observe. For agentic systems, observability is not a nice-to-have feature that you add when you have time. It is the prerequisite for understanding what your system is doing, which is the prerequisite for being able to make it better.