AI Agent Observability: Tracing, Metrics, and Cost in Production

Your agent gave a wrong answer, and you have no idea where it went off the rails. Was it the retrieved context, a bad tool call, a truncated prompt, or the model itself? You cannot answer that by re-reading the final output. You need to see the run.

What is AI agent observability?#

AI agent observability captures each step of a production agent run as connected data so you can replay a failure and find the exact point it broke. It records the LLM calls, the tool calls, and the prompts and outputs of a single request, correlated by one id. It answers "what happened", not "is the agent good".

This is a different job from testing. Evaluation runs before you ship and scores the agent against a fixed set of cases. Observability runs after you ship and watches real traffic you never anticipated. If you want the pre-release side, our guide to testing and evaluating AI agents in n8n covers that half. This post is about the 3am incident, when a real user got a wrong answer and you have to explain why.

The three pillars: traces, metrics, and logs#

Classic observability rests on 3 signals, and each maps cleanly onto an agent. Traces show the run as a tree of steps. Metrics count and time those steps in aggregate. Logs hold the raw prompt and output text. Together these 3 pillars move you from "something is wrong" down to the one span that caused it.

Here is how each pillar maps to a question you will actually ask during an incident.

A trace is the backbone. One user request becomes one trace, and every model call or tool call inside it becomes a span nested under that trace. Because all 3 signals share the trace id, you can jump from a spiking latency metric to the exact trace, then read the logged prompt on the slow span without guessing.

One request equals one trace. Each LLM call and tool call is a child span under it, correlated by one trace id.

How does a trace tree work for an agent?#

A trace is one request rendered as a tree of spans. The root span is the whole run. Under it sit child spans for each LLM call and each tool call, in order, each stamped with a start time and duration. Parent and child spans share one trace id, so the tree reconstructs the agent's real path step by step.

Say a user asks your support agent a question. The root span opens. The agent makes a first LLM call to plan, which becomes a child span. That plan decides to call a search tool, so a tool span opens, runs for 0.8 seconds, and returns 5 documents. A second LLM call reads those documents and writes the answer. If the answer is wrong, you open the trace and see immediately whether the search span returned junk or the final model call ignored good context. The OpenTelemetry project publishes GenAI semantic conventions that define standard names for these LLM spans, so a trace from one library reads the same in any compatible tool.

What should you capture per step?#

Capture enough per step to reproduce and price the run without re-running it. For every LLM call and tool call, record the input, the model or tool name, tokens in and out, cost, latency, and the result. Skip any one of these and you will hit a failure you cannot explain.

Log all of these per step. The field you skip is usually the one you need during an incident.

Two fields do the heavy lifting and get skipped most. Cost per step turns a vague "our bill went up" into "this one tool loops 7 times and burns tokens each pass". Tool arguments and results turn "the agent hallucinated" into "the API returned an error string and the model treated it as data". Capture the arguments a tool was called with, not just that it was called, or you lose the ability to tell a bad call from a bad response.

n8n execution log versus a dedicated tool#

If you build agents in n8n, you already have a basic trace. The execution log shows every node's input and output for a run, which is enough to see that data entered a node wrong or a branch went the wrong way. For LLM-level detail like per-call tokens and cost, push spans to a dedicated tool.

The n8n log is a free node-level trace; a dedicated tool adds per-call token, cost, and latency detail plus dashboards.

The split is about depth. The n8n log answers node-level questions and needs zero setup, and for many workflows that is genuinely enough. A dedicated LLM-observability tool adds the per-call token, cost, and latency breakdown, plus aggregate dashboards across thousands of runs and a searchable prompt history. You can bridge the two: send spans from n8n to a tool like Langfuse over HTTP, or start by reading the usage object the AI node returns and logging it. If you are wiring tools into an n8n agent in the first place, our guide to giving an n8n AI agent tools covers that side.

Which observability tool should you use?#

Pick by whether you need self-hosting and how much you want built for you. Langfuse is open-source and self-hostable, so it fits teams that must keep prompt data in-house. LangSmith, Helicone, and Arize Phoenix are the other common names. OpenTelemetry GenAI conventions are the emerging open standard, so favor tools that speak it.

The honest advice is to not adopt any of them on day one. A single agent with light traffic is served fine by structured logs and your framework's own run view. Reach for a dedicated tool when volume, cost, or a multi-agent setup makes raw logs impossible to search. The Langfuse documentation is a reasonable place to see what a full tracing layer offers before you commit. When you graduate to several agents handing work to each other, tracing stops being optional, which is why our multi-agent orchestration guide treats it as a core concern.

Start on structured logs. Reach for a dedicated tool when volume, cost, or a multi-agent setup outgrows them.

Where observability pays off#

Observability pays off the moment you stop trusting your agent blindly. It turns "it gave a wrong answer" into a specific span you can point at, turns a rising bill into a named loop, and turns a slow response into the one tool call that stalled. You capture the run once and reuse it for debugging, cost control, and evidence.

Start small and honest. Log every step's input, model, tokens, cost, latency, and result to structured logs, and give each request a trace id you can grep. That alone resolves most incidents. Add a dedicated tool when your volume, your bill, or a multi-agent topology outgrows plain logs. The goal never changes: when a run breaks, you should be able to open it and see exactly where.