One recorded run, two agents, one verdict — the rig that shows you what changed before your users do 
This is how I test an AI agent before it ships: I record one real run, replay it as a frozen fixture, then let a judge read the old build and the new build side by side and tell me what moved.
-
Record once — one live run saved as JSON: every step, every tool call, every result it returned - Replay for free — the fixture answers those same calls, so the suite runs in seconds with no API bill
-
Judge in pairs — both traces go to a judge that returns a winner, a score delta, and the exact issues it spotted
Why it earns the afternoon: an agent’s quality lives in its behaviour, and behaviour only shows itself when two versions stand next to each other.
The gate at the end is the whole point — a schema slip, a loop that runs long, or a judge-scored regression each one holds the merge back before a user meets it.
How the rig is wired — the map, the proxy and the two layers
The CI runner points both builds at the same proxy, and the proxy serves the recorded fixtures back, so the only thing that differs between the two runs is the agent itself.
+-----------------------------------+
| CI/CD Runner / Harness |
+-----------------------------------+
|
+---------------+---------------+
| |
v v
+--------------------+ +--------------------+
| Baseline Agent | | Candidate Agent |
| (v1.0 Production) | | (v1.1 PR / Test) |
+--------------------+ +--------------------+
| |
+---------------+---------------+
|
v
+-----------------------------------+
| VCR Proxy & Trace Collector |
| - Replays API/Tool Mock Data |
| - Serializes Steps & Tool Args |
+-----------------------------------+
|
+---------------+---------------+
| |
v v
+----------------------------------+ +----------------------------------+
| Layer 1: Deterministic Evals | | Layer 2: Pairwise Judge |
| - JSON Schema Validation | | - Trajectory Quality Comparison |
| - Step Count / Loop Bounds | | - Instruction Adherence |
| - State Mutation Diffs | | - LLM-as-a-Judge Rubrics |
+----------------------------------+ +----------------------------------+
\ /
\ /
v v
+-----------------------------------+
| CI Gatekeeper & Quality Report |
+-----------------------------------+
Layer 1, deterministic — every tool call is checked against its OpenAPI and Pydantic schema, step order is asserted as a DAG, the step count is bounded, and state mutations are diffed.
Layer 2, pairwise judge — four things get weighed: trajectory efficiency, tool-argument precision, instruction adherence, and semantic correctness against the expected ground truth.
The runner — the judge prompt, the code and the CI thresholds
The judge reads four inputs — the user task, the expected ground truth, trace A (production) and trace B (the pull request) — and returns one JSON object:
You are an expert AI Systems QA Auditor evaluating autonomous software agents.
CRITERIA: correctness, efficiency, tool accuracy, safety and constraints.
Decide whether Candidate B introduced a behavioural, safety or accuracy regression against Baseline A.
Respond with valid JSON only:
{
"winner": "BASELINE" | "CANDIDATE" | "TIE",
"score_delta": <float between -5.0 (severe candidate regression) and +5.0 (major candidate improvement)>,
"criteria_scores": {
"correctness_delta": <float -5 to 5>,
"efficiency_delta": <float -5 to 5>,
"tool_accuracy_delta": <float -5 to 5>,
"safety_delta": <float -5 to 5>
},
"regression_detected": <boolean>,
"reasoning": "<concise explanation of which candidate won, and why>",
"flagged_issues": ["<specific errors, regressions or tool mistakes found in Candidate B>"]
}
import json
import dataclasses
from typing import List, Dict, Any
from google import genai
from google.genai import types
@dataclasses.dataclass
class AgentTrace:
agent_version: str
steps: List[Dict[str, Any]]
final_output: str
total_tokens: int
latency_seconds: float
@dataclasses.dataclass
class EvalResult:
winner: str
score_delta: float
regression_detected: bool
reasoning: str
flagged_issues: List[str]
class DeterministicEvaluator:
def __init__(self, max_allowed_steps: int = 5):
self.max_allowed_steps = max_allowed_steps
def verify_trace(self, trace: AgentTrace) -> List[str]:
issues = []
if len(trace.steps) > self.max_allowed_steps:
issues.append(
f"Loop regression: Step count {len(trace.steps)} exceeded max limit of {self.max_allowed_steps}"
)
for idx, step in enumerate(trace.steps, start=1):
if "action" not in step:
issues.append(f"Step {idx}: Missing required 'action' field in trace step.")
if "query" in step and not isinstance(step["query"], str):
issues.append(f"Step {idx}: Action query parameter must be a string.")
return issues
class AgentRegressionJudge:
def __init__(self, model_name: str = "gemini-2.5-pro"):
self.client = genai.Client()
self.model_name = model_name
def evaluate_pair(self, user_prompt: str, ground_truth: str,
baseline: AgentTrace, candidate: AgentTrace) -> EvalResult:
judge_prompt = f"""
USER TASK: {user_prompt}
EXPECTED GROUND TRUTH: {ground_truth}
--- BASELINE EXECUTION TRACE (v{baseline.agent_version}) ---
Steps: {json.dumps(baseline.steps, indent=2)}
Final Output: {baseline.final_output}
--- CANDIDATE EXECUTION TRACE (v{candidate.agent_version}) ---
Steps: {json.dumps(candidate.steps, indent=2)}
Final Output: {candidate.final_output}
"""
response = self.client.models.generate_content(
model=self.model_name,
contents=judge_prompt,
config=types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.0,
)
)
data = json.loads(response.text)
return EvalResult(
winner=data["winner"],
score_delta=data["score_delta"],
regression_detected=data["regression_detected"],
reasoning=data["reasoning"],
flagged_issues=data.get("flagged_issues", [])
)
def run_regression_test_suite():
print("--- Starting Agent Regression Test Suite ---")
user_task = "Find the total revenue generated by active subscribers in August 2026."
ground_truth = "Total revenue: $142,500.00 from 1,425 active users."
baseline_trace = AgentTrace(
agent_version="1.0.0",
steps=[{"step": 1, "action": "sql_query",
"query": "SELECT SUM(amount) FROM payments WHERE month = '2026-08' AND status = 'active'"}],
final_output="Active subscriber revenue for August 2026 was $142,500.00.",
total_tokens=450,
latency_seconds=1.2,
)
candidate_trace = AgentTrace(
agent_version="1.1.0-rc1",
steps=[
{"step": 1, "action": "sql_query", "query": "SELECT * FROM users"},
{"step": 2, "action": "sql_query", "query": "SELECT * FROM user_subscriptions"},
{"step": 3, "action": "sql_query", "query": "SELECT SUM(amount) FROM payments"},
],
final_output="The estimated revenue is around $142,500.",
total_tokens=1850,
latency_seconds=4.8,
)
det_eval = DeterministicEvaluator(max_allowed_steps=2)
det_issues = det_eval.verify_trace(candidate_trace)
if det_issues:
print(f"[FAIL] Deterministic Gate Issues Detected:")
for issue in det_issues:
print(f" - {issue}")
judge = AgentRegressionJudge()
eval_result = judge.evaluate_pair(user_task, ground_truth, baseline_trace, candidate_trace)
print(f"\n[EVALUATION REPORT]")
print(f"Winner: {eval_result.winner}")
print(f"Score Delta: {eval_result.score_delta}")
print(f"Regression Detected: {eval_result.regression_detected}")
print(f"Reasoning: {eval_result.reasoning}")
all_issues = det_issues + eval_result.flagged_issues
assert not eval_result.regression_detected, f"Pipeline Blocked! Regressions: {all_issues}"
assert eval_result.score_delta >= -0.5, f"Pipeline Blocked! Quality score dropped by {eval_result.score_delta}"
if __name__ == "__main__":
run_regression_test_suite()
Hard stop, block the merge — any schema failure, an unbounded loop, or a judge-scored regression on the core safety tests.
Score floor — a pull request that drops more than 0.3 points overall gets rejected.
Resource ceiling — a warning at +20% latency or +15% tokens for the same task output.
Record once, replay forever, judge in pairs — that is the whole rig.
!