AI observability means capturing the full course of an AI request and making it queryable: which model was used, what the input was, how many tokens it consumed, which tool the agent chose, what the tool returned, and why the final answer came out as it did. The goal is that when someone says "it quoted the wrong price to a customer yesterday," you can pull up that exact trace and look.
Traditional service monitoring cares about availability: is the endpoint returning 200, is latency within budget, is the error rate climbing. The difficulty with AI applications is that they almost always return 200. The call succeeded, latency was fine, and the answer was wrong, the citation invented, the tool called with the wrong arguments. Traditional monitoring cannot see any of that.
Grab It in One Sentence First
Observability answers not "is the service down" but "what was it actually thinking at the time."
An analogy. Traditional monitoring is the footfall counter at a restaurant door: it tells you how many people came and whether there was a queue. AI observability is the kitchen camera: which ingredients went into a dish, how long it cooked, how many times it was redone. When a customer complains, the first can only tell you it was busy; the second shows you which step went wrong.
What It Records
The structure borrows from distributed tracing: one user request is a trace containing nested spans — each model call, tool execution, and retrieval is a span with its own inputs, outputs, and duration.
The AI-specific fields fall into a few groups: model and parameters (which model, what temperature), token usage (input, output, cache hits — which convert directly into money), finish reason (completed normally, truncated, or stopping to call a tool), and optionally content capture (the prompt and response text).
Content capture is the trade-off that needs the most thought: without it you can't debug, and with it you're retaining user input long-term inside your monitoring system, with privacy and compliance implications. Common practice is redaction by default, capture by sampling rate, and masking of sensitive fields.
Standardization: OpenTelemetry GenAI Semantic Conventions
Early on, every platform recorded its own field names, so the same thing was called different things in different tools and switching vendors meant re-instrumenting. OpenTelemetry's GenAI semantic conventions exist to fix that: they define which span names and attribute names model calls, tool executions, agent runs, and retrieval should emit — for example gen_ai.request.model for the model and gen_ai.usage.input_tokens / gen_ai.usage.output_tokens for usage.
The benefit is instrument once, swap backends later: an agent call inside LangChain and a raw API request end up with the same data shape.
Note that as of 2026 the GenAI semantic conventions remain in Development status, so attribute names and structures may still change, and conventions for multi-agent systems and MCP are still being drafted. The spec provides switches like OTEL_SEMCONV_STABILITY_OPT_IN to manage version transitions. Instrumenting against it today is reasonable, but leave room for renames.
Versus Neighboring Concepts
Versus evals. These two form a loop. Observability records what actually happened in production; evals judge whether it was good. Failures found in production should be distilled into eval cases, and the same eval suite verifies the fix. Observability without evals leaves you firefighting after the fact; evals without observability means you only ever test scenarios you imagined.
Versus traditional APM. The metric set differs. APM cares about QPS, p99 latency, and error rate. AI applications additionally care about token cost, cache hit rate, tool-call success rate, retrieval quality, and answer correctness — and that last one can't be judged automatically, requiring sampled human review or model-based scoring.
Versus guardrails. Guardrails intercept at runtime and block problems as they happen; observability records after the fact for analysis and improvement. They share many signals but act at different moments.
Versus context engineering. Observability supplies the facts that context engineering needs. Breaking a call's tokens down by source — how much is system prompt, tool definitions, history, retrieval results — is what tells you which part to optimize.
Where People Get It Wrong
"We have logs, so we have observability." Scattered logs can't answer which steps a request went through. What matters is structure and correlation: every span of one request must be viewable together, not pieced back from three services' log files.
"Latency looks fine, so we're fine." The signature AI failure is producing a wrong answer quickly. Quality metrics have to sit alongside performance metrics.
"Capturing everything is safest." Full capture means your monitoring system holds a large volume of raw user input, which turns a breach into a major disclosure and often counts as processing requiring separate justification under compliance regimes. Redaction by default plus sampling is usually the better starting point.
"Adopting a platform gives it to you automatically." Tools give you traces. Deciding which metrics to watch, which failures become test cases, and what standard counts as good remains work you have to do.
Where to Start
You don't need everything at once. The minimum useful set is three things: a traceable ID per request (so user feedback maps to a specific call), the model, token usage, and finish reason (cost and truncation problems become visible immediately), and tool call arguments and results (where agents most often go wrong).
Those three locate most production issues. Add the rest as needed: retrieval quality, cache hit rate, cost broken down by user or feature, and a regular pipeline moving production failures back into your eval set.