The telemetry you need is not a single score

A pass rate can support a release review. It cannot diagnose a system. If the number falls, the person responsible still needs to know whether the agent misunderstood a user, inherited stale state, selected an unnecessary tool, or took the right action too late.

Aggregate metrics also hide customer asymmetry. The same system can work beautifully for direct users and fail for people who are uncertain, interrupted, or operating through an intermediary. An average makes those groups disappear.

Useful telemetry links an invocation to the observation, action, state transition, and later outcome. It carries the agent version, world version, tool result, latency, cost, and relevant policy checks. For physical systems it should also preserve the sensor conditions and the confidence behind an inference.

With that chain intact, a team can ask productive questions. Did the model ignore evidence or never receive it? Did the world change after the action? Did the system recover on its own? Did a seemingly successful outcome require too much time, cost, or human intervention?

A reliable system needs both. Outcome metrics show whether it created value. Path metrics show whether the result was safe, efficient, and repeatable. A booked meeting after six unwanted messages is not a clean success. A robot completing a task after repeated unsafe recovery attempts is not deployment ready.

Trace-level evidence keeps these distinctions visible. It gives engineering, product, and operations a shared language for improving behavior instead of arguing over a number stripped of context.