AI Observability
— seeing what the agent does
An AI agent that "just does not work" is a debugging nightmare — non-deterministic input, a chain of LLM and tool calls, and climbing costs. Observability turns the black box transparent: every trace, every prompt, every token and cost. In this guide: the four pillars, tracing and evals, and LangSmith vs Langfuse.
Why observability is critical for AI systems
In regular software, the same input always returns the same output. In an LLM-based system it does not: the same question can return different answers, an agent may pick a different tool path on each run, and the cost varies from request to request. Without a dedicated monitoring tool, you are "blind" — you do not know why the agent decided what it did, where it got stuck, and how much it cost.
AI Observability is the equivalent of monitoring in the AI world: it records the entire chain of calls to anagent and to an LLM, measures quality with evals, and tracks costs and latency — so you can debug, improve and trust the system in production.
The four pillars
Tracing — the heart of the system
A trace is a record of a whole request, broken intospans — each step in the chain. In a typical agent, one trace contains the initial prompt, the LLM decision, a call to a search tool, a retrieval from avector DB, and the final answer — each with its input, output, time and cost.
When something goes wrong, you open the trace and see exactly where: maybe the retrieval returned irrelevant documents, maybe the tool failed, maybe the prompt was confusing. That is the difference between minutes of debugging and hours of guessing.
In 2026 there is convergence around OpenTelemetry (OTel) as an open standard for LLM tracing. Most tools support it, so you are not locked to a single vendor — the same instrumentation can send data to Langfuse, LangSmith or any other backend.
Evals — how you know it actually works
"Looks good" is not a metric. Evals are systematic tests of output quality, and without them every "improvement" to a prompt is a gamble. Three main approaches:
- Ground truth — you have a known correct answer and compare against it (accurate for closed tasks: classification, data extraction).
- LLM-as-judge — another model rates the answer against criteria (relevance, faithfulness to the source, tone). Suited to open-ended tasks.
- Human feedback — 👍/👎 from real users, or expert rating. Expensive but reliable.
The real power: evals alongside CI/CD. You run a set of examples (a dataset) on every change to a prompt or model, and see whether quality rose or fell — before it reaches users. And yes — security testing is part of evals.
LangSmith vs Langfuse — what to choose
| Tool | Model | Best for |
|---|---|---|
| LangSmith | Managed (closed) | Those already on LangChain/LangGraph |
| Langfuse | Open source + cloud | Self-host, framework-agnostic |
| Arize / Phoenix | Open source + cloud | Advanced evals, ML too |
| Helicone | Lightweight proxy | Cost tracking & caching, fast |
- LangSmith — from the makers of LangChain. Smooth integration with LangGraph, tracing, datasets and evals in one place. Managed.
- Langfuse — open source, framework-agnostic, allows full self-host (important for privacy). The most popular in the community in 2026.
- Arize Phoenix — strong at evals and analysis, open, and suited to broader ML teams too.
- Helicone — a light proxy that adds cost tracking, caching and rate limiting in one line. Excellent for a fast start.
How to start — the first step
Do not wait until it is "tidy". Add tracing right now — it is a handful of lines. With Langfuse it looks like this:
from langfuse.openai import openai # transparent wrapper
resp = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role":"user","content":"Summarize the document"}],
)
# every call is logged automatically: prompt, response, tokens, cost, latency1. Add tracing on day one — to see what happens. 2. Set up cost tracking with an alert on overruns. 3. Build a dataset of examples and run evals on every change. 4. Connect feedback from real users into that same dataset. Observability is not a one-off project — it is a continuous improvement loop.