Standard observability tools hide the longest, costliest, and most consequential LLM agent failures. They were built around microservice calls, where each span has a fairly predictable role and latency. Agent chains introduce semantic decisions, variable generation time, and evaluation data that this model does not represent well.
The request-response assumption does not hold for LLM calls
Traditional tracing expects span latency to follow a knowable distribution. A downstream HTTP call may take 5ms to 500ms, making its 95th percentile predictable from historical data. An LLM call has no comparable stability. One llm:generate span may last 200ms to 45 seconds, depending on output length, model load, and whether streaming is enabled at the transport layer.
Observability backends also commonly build timeout expectations into span visualizations. A span lasting more than a few seconds may be flagged, truncated in the UI, or classified as an anomaly by default alerting rules. That makes the longest agent calls the least visible. The 30-second document processing span that began a hallucinated chain of reasoning may be the very span an on-call engineer cannot locate at 2am.
Span attribution becomes ambiguous in multi-step agents
In a microservice trace, parent-child links mirror structure: service A calls service B, which calls service C. Span context headers are enough to reconstruct that invocation graph.
Agent spans have a semantic relationship instead. A language model emits an intent, the agent chooses a tool, and the tool result returns to the model as an observation. The parent span is the decision step that initiated the action, not a conventional programming caller. Standard OpenTelemetry span context propagation does not record this distinction. Instrument a ReAct-style agent with a vanilla OTel SDK and the result is a flat span list, missing the agent-action-observation loop that explains the run.
This distinction is critical after an incident. For a RAG agent that produces a confidently wrong answer, debugging requires identifying which retrieval call returned which document chunks, then finding the generation step that failed to remain grounded in them. Without semantic parent-child attribution between retrieval and generation, timestamps show sequence but not causality. You know the tool call preceded generation, but not that generation was conditioned on the tool result.
Token counts and latency need separate treatment
Standard trace metrics provide duration and error status. LLM calls also need input token count and output token count, because those values reveal problems that duration cannot.
High input tokens paired with low output tokens may signal a near-miss at the context window boundary. The model received more context than it could effectively attend to, and its output compressed as a consequence. Unexpectedly high output tokens may indicate that the model ignored a length constraint or departed from the required format. A latency histogram exposes neither pattern.
General-purpose tools usually store token counts as arbitrary span attributes. You can add them manually, but they are not first-class dashboard or alerting dimensions. A P95 latency chart therefore cannot explain a pipeline's bimodal latency distribution when the relationship between tokens and latency remains buried in raw attribute JSON instead of an aggregation.
Evaluation scores have no native home in standard traces
Standard distributed tracing was not built to keep evaluation results beside performance data. A hallucination score or rubric result can be logged as a span attribute, but Datadog or Grafana has no built-in query for all traces in which the LLM generation span score fell below 0.70 this week, grouped by prompt template.
The usual outcome is a split between trace data in one system and evaluation data in another. Correlation then depends on a manual join using trace IDs. That is exactly the connection needed during a quality incident in which outputs deteriorate gradually over two days. Two systems, two queries, and spreadsheet-based correlation are unreliable under incident pressure.
What purpose-built agent tracing actually requires
LLM agent tracing needs specific primitives beyond general-purpose distributed tracing.
Structured attributes designed for LLM calls
Each LLM call span should index these fields as first-class attributes rather than freeform key-value pairs: model identifier, input token count, output token count, temperature, sampling parameters, and a stable prompt hash. Store the hash instead of the full prompt text, which is expensive to retain at scale. These are the first fields needed during debugging, so they must be indexed, filterable, and aggregatable.
Tool call spans should include the tool name, a hash of the tool input, output size, and whether the tool returned an error. Agent decision loop spans should record the iteration number, the stopping condition evaluated by the agent, and whether that condition was met.
Evaluation scores as first-class span attributes
A more useful design writes evaluation results directly onto the span that produced the evaluated output instead of storing them elsewhere and joining by trace IDs. The LLM generation span can then show its faithfulness score, instruction-following score, and rubric failures beside latency and token metrics. Trace filtering can use those scores just as it uses error status or latency percentile.
The architecture is straightforward: the evaluation system writes back to the trace store rather than a separate database. The cost is slightly larger span payloads. The benefit is one query surface for performance and quality data.
The boundary is important: OpenTelemetry is not wrong for this domain. Its span-and-trace primitive is appropriate. The problem is the span vocabulary, the default attribute schema, and the assumption in general-purpose backends that all spans have roughly equivalent diagnostic needs. Teams that debug production AI failures fastest instrument the semantic layer, attaching evaluation results and agent-action-observation structure to spans instead of instrumenting only the network layer.