What Is AI Observability?

AI observability is the practice of watching every step an AI system takes so you can debug it when it produces the wrong answer.

Traditional observability answers questions like: is the server up, how fast are requests processing, what percentage of API calls returned 500 errors. Those questions still matter for AI systems. They also miss the questions that matter most. Was the answer right. Did the retrieval surface the correct documents. Did the agent take a reasonable path through its tool calls. Did the model hallucinate.

An AI system can be technically healthy by every standard metric and still be producing wrong answers. AI observability is the layer that catches that.

The Difference From Traditional Monitoring

Traditional observability was built for deterministic systems. A function takes an input and returns an output. If the output is wrong, the code has a bug. Find the bug, fix it, move on.

AI systems are not deterministic. The same prompt to the same model on the same day can produce different outputs. The wrong answer is not evidence of a bug in the code. The code executed correctly. The model returned what it returned. The failure is in the reasoning, or the retrieval, or the tool selection, or the context assembly. Traditional monitoring cannot see any of those layers.

Three signals matter for traditional systems: metrics, logs, and traces. AI observability adds a fourth: the semantic quality of the output. Every trace carries not just what happened but whether what happened was correct.

What Gets Captured

An AI observability platform captures the full execution graph of every AI operation. Not just the final response, but every step that produced it.

The user input. The prompt or query as the user sent it. Structured and unstructured. Any metadata about the user session, the conversation history, the source application.

Every model call. Which model was invoked. Which system prompt. Which user prompt. Which temperature and other parameters. The full response. Token counts for input and output. Latency. Cost.

Every retrieval. If the system uses RAG, every retrieval operation gets captured. Which query embedding was generated. Which chunks were returned. What their similarity scores were. Which ones the generator actually used.

Every tool call. If the system uses tools or function calling, every call gets captured. Which tool was chosen. What arguments were passed. What the tool returned. Whether it succeeded or failed.

Every reasoning step. For agents, the full plan, the chosen path, any retries, any handoffs to sub-agents. The execution graph that turned a user request into a final answer.

Quality signals. Automated evaluations that ran on the output. Faithfulness scores. Groundedness scores. Task completion assessments. Any custom evaluators the team has built.

All of this gets stored as a trace. A trace is one complete story: user input, all the steps in between, final output, quality signals attached to each step. When something goes wrong, the trace is the diagnostic path. Without it, every failure is a mystery.

Traces and Spans

The two vocabulary terms that show up in every AI observability tool.

A trace is one complete user interaction. Everything from the moment the user sent the request to the moment the system returned the final response. A single trace can contain dozens of individual operations.

A span is one operation inside a trace. A model call is a span. A retrieval is a span. A tool call is a span. A reasoning step is a span. Spans are nested. A parent span (an agent execution) contains child spans (the individual model calls and tool calls that ran during the agent execution).

The span model came from distributed tracing in web infrastructure and got adapted for AI. The OpenTelemetry standard, which most AI observability platforms now support, was originally built for tracing microservices. AI operations became a natural fit because agents behave like distributed systems with their own microservices.

One agent run in 2026 emits 40 to 75 spans. That volume is why observability platforms bill on spans and why span reduction became a real optimization concern for teams running high-volume agent systems.

Why Standard APM Tools Are Not Enough

Some teams try to use their existing application performance monitoring stack for AI. Datadog. New Relic. Honeycomb. These tools are excellent at what they were built for. They struggle with AI observability for three specific reasons.

They cannot judge quality. APM tools measure latency, throughput, error rates, and resource utilization. They do not have opinions about whether the answer was correct. AI observability requires an evaluation layer that scores outputs against quality criteria. Standard APM does not ship this.

They do not understand the AI graph. An AI system's execution graph is not the same shape as a web application's request graph. Model calls, retrievals, tool calls, agent loops, sub-agent handoffs. Rendering these coherently requires an AI-aware trace viewer. Standard APM treats them all as generic spans and loses the specific context that makes debugging possible.

They lack the semantic layer. When a support agent produces a wrong answer, the debug path starts with "what did the retrieval return, what did the model reason from that, what tool did it call, what did the tool return." Traditional APM can show the calls happened. It cannot show what was in them at the semantic level the human debugger needs.

The pragmatic pattern most production teams have converged on: use AI observability for the AI-specific layer, keep APM for the surrounding infrastructure, and correlate them by trace ID when incidents span both.

The Tools the Field Has Settled On

By 2026 the AI observability market has consolidated around a small set of primary platforms. Every serious production team is using one of them.

LangSmith. Purpose-built for LangChain and LangGraph applications. The deepest framework integration in the field. Node-by-node state diffs. Full agent execution graphs. Replay against new model versions. If the AI stack is LangChain-native, LangSmith fits like a glove.

Arize Phoenix. The open-source observability tool that ships 50+ research-backed metrics covering faithfulness, relevance, safety, toxicity, and hallucination detection. Framework-agnostic. OpenTelemetry-based. The strong default for teams that want self-hosted tracing plus evaluation in one tool. Arize AX is the hosted commercial version.

Langfuse. Open-source, self-hostable, MIT-licensed. Strong trace store. Growing evaluation capabilities. The go-to when data residency or budget rules out sending traces to a vendor.

Traceloop and OpenLLMetry. Not a platform in the same shape as the others. OpenLLMetry is an Apache 2.0 OpenTelemetry instrumentation library for LLM applications. Traceloop is the hosted backend one place the traces can go. Most modern backends can ingest OpenLLMetry spans, which makes OpenLLMetry the safest instrumentation choice for teams that want portability across backends.

Braintrust. Evaluation-first platform with generous free tier. Best when the roadmap is driven by continuous quality measurement and non-engineers help define scorers.

Datadog LLM Observability. The natural choice for teams that already run Datadog for the rest of their stack and want AI observability inside the same platform. Correlates AI operations with the surrounding infrastructure automatically.

Confident AI, Helicone, Latitude, Maxim, Galileo. Specialized platforms that address particular corners of the observability workflow. Confident AI for enterprise standardization across teams. Helicone for cost tracking with almost no code change. Latitude for closing the loop from detected issue to opened PR. Maxim and Galileo for high-volume production evaluation.

The choice between them is a workflow decision, not a metric decision. All of the primary platforms cover the same underlying signal. What differs is how the signal integrates into the team's existing engineering workflow.

The Three Jobs Any Observability Platform Has to Do

Regardless of which platform a team picks, three jobs need to be covered.

Capture. Every AI operation gets traced. Every trace stores the full execution graph. Nothing important gets lost. The instrumentation layer is comprehensive without being expensive to run.

Score. Every trace gets automated quality signals attached. Faithfulness, groundedness, task completion, or whatever custom evaluators the application needs. The scoring layer runs continuously against production traffic, not just against static test sets.

Alert. When quality signals drift, someone gets notified. When faithfulness drops below the threshold. When latency spikes on a particular tool. When a specific prompt version starts producing worse outputs than the version it replaced. The alerting layer is what turns observability from a debugging tool into a monitoring tool.

Any platform that ships all three well is a viable choice. Platforms that only ship one or two require the team to build the missing layers themselves.

The Failure Mode That Observability Alone Cannot Solve

Observability answers the question "what happened." It does not answer the question "was the source material trustworthy in the first place."

An AI system can be fully observable, score high on all quality signals, and still be producing confident wrong answers because its underlying source material is stale, contradictory, or wrong. The trace shows every step of the pipeline executing correctly. The evaluators score the output favorably against the retrieved context. The user gets a wrong answer that everyone in the observability platform thinks is right.

This is the trust-layer problem observability does not address. Observability captures what happened inside the system. It cannot capture whether the information the system was working with should have been trusted at all.

Every mature AI infrastructure practice has an observability layer. The teams that ship the most reliable AI systems have observability plus a trust layer sitting underneath it, watching the source material that feeds every trace.

Where to Start

If a team has never done AI observability, three steps get to a real baseline.

Pick one platform based on the existing stack. LangChain team, pick LangSmith. Self-hosting requirement, pick Langfuse or Arize Phoenix. Already on Datadog, use their LLM Observability add-on.

Instrument every AI operation. Every model call. Every retrieval. Every tool call. Every agent step. Do not sample yet. Full traces on everything until the volume forces sampling decisions.

Add three quality evaluators. Faithfulness for RAG. Task completion for anything action-oriented. Latency for performance. Watch the trend lines. Alert on regressions.

That is the minimum viable AI observability. Everything beyond it is refinement. Sampling strategies as volume grows. Custom evaluators for domain-specific quality concerns. Cost dashboards. User-session correlation. Root-cause analysis workflows for common failure patterns.

The teams that debug AI reliably in production have this loop running. The teams that do not are the teams whose incident reports read "we do not know why it happened."

Structure before tools. Strategy before speed. Systems before scale.

Observability tells you what happened. Trust tells you whether what happened should be believed. Both layers are required for AI systems in production.

The AI Operator Framework is the operator playbook for shipping AI work that compounds. Read the framework here or take the course.