Observability for multi-step agent systems

“The agent gave a bad answer” does not identify a failing component. A multi-step turn may include context assembly, routing, model generations, tool calls, validation, and persistence. Debugging requires a trace that shows those steps as one operation and records enough metadata to distinguish latency, cost, and behavior failures.

The foundation is a shared correlation model. Every event should carry the same trace identifier while preserving separate identifiers for the user turn, model call, and evaluation case.

Define the telemetry contract first

This section creates a stable event envelope. Product analytics, tracing, and evaluation reports can use different backends as long as they share these fields.

A shared telemetry envelope can make that contract concrete:

from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from typing import Any, Protocol


@dataclass(frozen=True)
class Event:
    name: str
    trace_id: str
    turn_id: str
    span_id: str
    parent_span_id: str | None
    timestamp: str
    attributes: dict[str, Any]


class Sink(Protocol):
    def emit(self, event: dict[str, Any]) -> None: ...


def record(
    sink: Sink,
    *,
    name: str,
    trace_id: str,
    turn_id: str,
    span_id: str,
    parent_span_id: str | None,
    attributes: dict[str, Any],
) -> None:
    event = Event(
        name=name,
        trace_id=trace_id,
        turn_id=turn_id,
        span_id=span_id,
        parent_span_id=parent_span_id,
        timestamp=datetime.now(timezone.utc).isoformat(),
        attributes=attributes,
    )
    sink.emit(asdict(event))

Use W3C trace context or the equivalent supported by the application’s tracing library when crossing HTTP and queue boundaries. A process-local context variable cannot correlate work after a message enters another service unless the identifiers are serialized with it.

Trace the call graph

This section records where time was spent. A turn should be the root span, with child spans for context assembly, routing, each generation, each tool call, and response validation.

Model spans should record provider, model family, input tokens, output tokens, finish reason, retry count, and duration. Tool spans should record a stable tool category, status, duration, and response size. Avoid raw arguments and results unless an approved debugging mode provides encryption, retention limits, and access controls.

Names must remain low-cardinality. Use tool.name="search" and keep a resource identifier out of the span name; otherwise the tracing backend receives a new series for every request.

Separate traces, metrics, and evaluation

This section assigns each question to the right signal. A trace explains one execution. Metrics describe behavior across many executions. Evaluation judges whether a known case produced an acceptable result.

Derive latency, error rate, token use, and retry rate from structured events. Calculate percentiles by operation and model rather than averaging all steps together. A healthy top-level latency can hide a tool whose tail latency is degrading if most turns never call it.

Evaluation results should carry eval_run_id, case_id, prompt version, tool-schema version, and model version. They may link to a trace identifier from the synthetic run, but they should not be mixed with production success metrics. Development fixtures and real user traffic answer different questions.

Redact before export

This section keeps observability from becoming a second ungoverned copy of user data. Redaction belongs in the application before events leave the trust boundary.

from collections.abc import Mapping
from typing import Any


ALLOWED_MODEL_FIELDS = {
    "model",
    "input_tokens",
    "output_tokens",
    "duration_ms",
    "status",
    "retry_count",
}


def safe_model_attributes(raw: Mapping[str, Any]) -> dict[str, Any]:
    return {
        key: raw[key]
        for key in ALLOWED_MODEL_FIELDS
        if key in raw
    }

An allowlist is safer than removing known sensitive keys because new fields otherwise begin exporting automatically. Hashing a user identifier may support aggregation, but it does not make the rest of a prompt safe. Treat prompts, tool arguments, retrieved documents, and generated answers as content with a separate capture policy.

Define retention by signal. Aggregate metrics may remain useful longer than raw traces. Debug payload capture should have the shortest lifetime and the narrowest access.

Connect a bad metric to one trace

This section shows the practical value of shared identifiers. Suppose a dashboard reports higher token use for turns that call search. The analyst can select one trace_id, open its span tree, and see whether the increase came from query rewriting, retrieved text, retries, or final synthesis.

That flow only works when analytics events preserve trace identifiers and the tracing system makes them searchable. It does not require one vendor. It requires the same field names, propagation rules, and sampling decision across the stack.

Head-based sampling can discard the failed trace before the failure occurs. Use tail-based sampling when possible so errors, slow turns, and high-cost turns are retained while ordinary successful traces are sampled more aggressively.

Test the instrumentation

This section prevents telemetry from silently disappearing during refactoring. Contract tests should execute a synthetic turn and assert one root span, a shared trace identifier, valid parent-child relationships, and the expected lifecycle events.

Add schema validation at the collector boundary and alert on a sudden drop in event volume. An observability system that stops observing should produce an operational failure, not a reassuring empty dashboard.

A usable agent telemetry stack answers three different questions with linked evidence: what path one turn took, how the population behaves, and whether a controlled case still meets its contract. Shared correlation and pre-export redaction make those answers available without turning observability into a leak of application content.