Keeping agents fast with fewer hops and smaller context

Agent latency is the sum of model calls, tool calls, storage operations, retries, and queueing. Model choice affects the total, but architecture decides how many times each cost appears on the critical path.

Optimization should begin with a trace of one complete turn. Count the hops, measure their durations, and record the bytes or tokens passed between them. The resulting call graph usually identifies a removable boundary before it identifies a model parameter worth tuning.

Measure the critical path

This section adds timing around the units that contribute to a turn. Percentiles should be computed from completed traces rather than from unrelated service averages.

A small timing wrapper is enough to expose the critical path:

from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from time import perf_counter
from typing import Protocol


class EventSink(Protocol):
    def emit(self, name: str, fields: dict) -> None: ...


@asynccontextmanager
async def timed_step(
    sink: EventSink,
    *,
    trace_id: str,
    step: str,
) -> AsyncIterator[None]:
    started = perf_counter()
    status = "ok"
    try:
        yield
    except Exception:
        status = "error"
        raise
    finally:
        sink.emit(
            "agent.step",
            {
                "trace_id": trace_id,
                "step": step,
                "status": status,
                "duration_ms": round((perf_counter() - started) * 1000, 1),
            },
        )

Wrap routing, each model generation, every tool call, context assembly, and response persistence. The trace should distinguish parallel work from sequential work; adding span durations overstates latency when calls overlap.

Use a direct tool for deterministic fetches

This section removes a model hop from capabilities that do not require reasoning. A sub-agent is justified when it owns a separate policy, multi-step plan, or conversation. Fetching a typed record is a function call.

A deterministic fetch can use a direct tool boundary like this:

from dataclasses import asdict, dataclass
from typing import Protocol


@dataclass(frozen=True)
class ServiceStatus:
    state: str
    updated_at: str


class StatusClient(Protocol):
    async def fetch(self, service_id: str) -> ServiceStatus: ...


async def get_service_status(
    service_id: str,
    client: StatusClient,
) -> dict[str, str]:
    """Return authoritative status for a service visible to the caller."""
    status = await client.fetch(service_id)
    return asdict(status)

Authorization remains inside the client or service boundary. The model selects the tool when the user’s request requires current status, but no second model is needed to decide how to call one deterministic endpoint.

Compare both designs with the same cases. A lower hop count is not an improvement if tool-selection accuracy falls or the main prompt becomes overloaded with irrelevant tool descriptions.

Give persistent state a size and lifetime contract

This section keeps sessions from becoming a cache for arbitrary tool output. Persistent state should contain the minimum data needed to resume a conversation.

Define a typed state contract with an explicit serialized-size limit and lifecycle for every field. Identifiers needed to resume work may persist; request-local routing hints and full tool outputs should not.

Large tool results belong in a result store with an identifier and expiry, not in the session record. Per-turn routing signals belong in request-local state and should disappear after the response. The size limit turns “keep state lean” into a testable contract.

Compile data into a compact, typed card

This section reduces prompt syntax without weakening data boundaries. Passing raw JSON to the model preserves structure, but it often includes unused fields and repeated keys.

Compile only approved values into a deterministic card with stable labels, escaping, and a total budget. Raw objects may preserve structure, but they often expose unused fields and repeat keys that consume tokens without adding evidence.

Escaping is necessary because tool data is untrusted input even when it comes from an internal API. Keep labels stable, omit null or irrelevant fields according to a documented policy, and snapshot-test the rendered card. Measure its token count against the previous representation with the tokenizer used by the target model.

Stop after a definitive tool result

This section removes a final model call when the interface already has everything it needs. The optimization applies only when tool output is safe and complete for direct rendering.

Return a discriminated result from the tool layer that says whether the payload is ready to render or still needs summarization. The decision comes from the tool contract rather than a model guessing from the payload shape.

Use render for a validated UI payload whose labels and values are already user-facing. Use summarize when the model must explain, compare, or adapt the result to the conversation. Do not expose raw backend objects merely to save a generation.

Verify latency changes against behavior

Run the same evaluation cases before and after each change. Compare end-to-end p50 and p95 latency, model-call count, tool-call count, prompt tokens, retry rate, and task success. Averages alone hide the slow paths that users notice.

The effective sequence is to remove unnecessary serial hops, bound persistent state, fetch live data only when needed, compile compact prompt inputs, and skip narration when the response is already complete. Each change should be visible in a trace and neutral or positive in the behavior suite.