CHAPTER 15 · Observability and Evaluation · 2 / 7
What to capture: the trace
The unit of agent observability is the trace: the complete record of one turn. A good trace includes:
- Inputs: the assembled context (system prompt version, the messages, the available tools), the user, the model and settings.
- Each model call: the request, the response, token counts (input/output/reasoning), latency, and finish reason.
- Each tool call: name, arguments, result (or error), and duration.
- The loop shape: how many iterations, in what order.
- Outputs: the final answer, emitted structured data (citations), and any artifacts produced.
- Outcome: success/failure, and any error details.
The event timeline you already build for streaming and persistence (Chapters 8, 9) is most of this for free: it's a structured, ordered record of the turn. Observability often means also routing that timeline (plus the model-call metadata) to a tracing system where you can search, aggregate, and inspect it.