How to compare two AI SDR versions

A fair comparison gives the baseline and candidate the same reviewed market, personas, starting state, time horizon, tools, and guardrails. Persona records should remain byte-identical across both runs. Otherwise an audience change can be mistaken for an agent improvement.

Capture every configuration that can alter behavior, including the model, prompt, context policy, tool descriptions, retry rules, and orchestration. The comparison is between complete agent versions, not model names in isolation.

Start with evidence-backed outcomes such as meaningful replies, qualified interest, accepted meeting times, successful calendar creation, opt-outs, and handoffs. Then split results by the buyer attributes that matter to the release question. Averages can hide that one cohort improved while another absorbed a new failure.

Do not treat a simulated conversion rate as a revenue forecast. The reliable use is relative comparison inside the same controlled world, followed by production validation after launch.

For each meaningful delta, locate the earliest action where the versions behave differently. Review the observation, buyer state, agent response, tool result, and later outcome. This turns a metric change into an engineering target.

The cause may sit in the prompt, context, policy, model, tool contract, wait logic, or adapter. The evidence should support that attribution. A plausible explanation without the matching trajectory remains a hypothesis.

A candidate can improve meetings and still be unacceptable if it creates more unwanted follow-ups or false booking claims. Release criteria should combine outcome gains with non-negotiable guardrails and operational constraints.

Keep the reviewed comparison as part of the release artifact. After deployment, monitor the same version in production and return confirmed failures to the regression suite. That preserves continuity between pre-release evidence and live behavior.