Project Plan A: Deterministic Agent Mock & Replay Harness (VCR for AI Agents)

Overview

A lightweight, high-performance recording and replay harness designed to eliminate non-determinism, third-party API costs, and network latency during AI agent CI/CD testing. Inspired by VCR-style HTTP mocking, this tool intercepts tool calls and external API interactions during an initial agent run, serializing them into deterministic test fixtures for rapid offline regression testing.

Key Technical Objectives

• Deterministic Fixture Generation: Capture outbound JSON-RPC/REST calls, arguments, and return states into versioned, human-readable .yaml or .json trace files.
• Zero-Cost CI Runs: Intercept outbound tool calls during automated test suites and mock responses directly from local fixtures, bypassing expensive LLM calls and third-party APIs.
• Fuzzy & Exact Payload Matching: Support exact argument matching as well as configurable schema-based fuzzy matching for dynamic parameters (e.g., timestamps, generated UUIDs).
• State Drift & Break Detection: Instantly catch breaking API schema changes, invalid argument mutations, and infinite tool-execution loops before code hits production.

System Architecture & Workflow

[ Agent Under Test ] 
        |
        v  (Outbound Tool Request)
+-----------------------------------------------------------+
|                   Mock & Replay Proxy                     |
|                                                           |
|  Mode: RECORD  ---> Save request/response to local trace  |
|  Mode: REPLAY  ---> Match against fixture & inject response|
+-----------------------------------------------------------+
        |                                    |
        v (RECORD Mode)                      v (REPLAY Mode)
[ Live External API ]              [ Saved Local Fixtures ]
      

Core Components

1. Interceptor Proxy Engine: Middleware interceptor written in Go or Python that acts as a local gateway for outbound agent tool execution.
2. Fixture Serialization Engine: Converts execution traces into structured, version-controlled mock files containing sequence indexes and payload hashes.
3. Sequence Matcher: Ensures tool invocations happen in the exact expected sequence, raising immediate regression flags if an agent skips steps or changes tool call order.

Deliverables & Milestones

• Phase 1: Core HTTP/RPC proxy interceptor and JSON fixture recorder.
• Phase 2: Sequence-matching engine with configurable tolerance for dynamic fields (UUIDs, timestamps).
• Phase 3: Integration runner for PyTest / Go testing framework.
• Phase 4: Benchmarking suite demonstrating 10x faster test execution times and 100% cost reduction in offline CI environments.