Project Plan B: Dual-Layer Automated Agent Regression & Evaluation Suite
Overview
An end-to-end evaluation pipeline built to detect performance, accuracy, and behavioral regressions in multi-step AI agents. By combining fast, low-cost deterministic assertions with an automated LLM-as-a-Judge semantic evaluator, this suite provides comprehensive pass/fail metrics across model fine-tunes, prompt revisions, and framework updates.
Key Technical Objectives
- Multi-Layer Regression Gating: Enforce a two-tier evaluation strategy—Layer 1 for zero-cost structural/schema validation, and Layer 2 for semantic quality scoring.
- Trajectory Divergence Analysis: Trace and compare candidate execution pathways against established baseline trajectories to detect extra steps, hallucinated tools, or plan degradation.
- Automated LLM-as-a-Judge Scoring: Standardize qualitative checks (e.g., safety, conciseness, tone, adherence to instructions) using structured evaluation rubrics and automated scoring models.
- CI/CD Pass/Fail Thresholds: Output structured JSON/JUnit test results with customizable threshold metrics to automatically block regressive PRs in automated build pipelines.
Evaluation Layer Architecture
| Evaluation Layer | Scope & Target | Execution Mechanism | Key Metrics & Assertions |
|---|---|---|---|
| Layer 1: Deterministic Gates | Structure, Schema & Safety | Code Assertions & Schema Matching | Step-count bounds, OpenAPI argument validation, Tool sequence DAG, DB state diffs |
| Layer 2: Semantic Gates | Accuracy, Trajectory & Quality | Embedding Models & LLM-as-a-Judge | Cosine text similarity, rubric score variance, tool hallucination rate, factual correctness |
Core Components
- Trajectory Collector: Captures full execution trees, including thoughts, tool inputs, outputs, execution latencies, and final system states.
- Deterministic Assert Engine: Evaluates tool call sequence accuracy, schema validity, and max-step limits in milliseconds.
- Semantic Rubric Judge: Sends execution traces and rubrics to an evaluation model, outputting normalized scores (0.0–1.0) with detailed reasoning logs.
- Diff & Reporting Generator: Produces side-by-side terminal and HTML diff reports comparing baseline vs. candidate trajectories.
Deliverables & Milestones
- Phase 1: Trajectory schema specification and log collector implementation.
- Phase 2: Deterministic assertion library (tool order validation, schema checks, max-step bounds).
- Phase 3: Structured LLM-as-a-Judge grading harness with rubric prompt templates and JSON output parsing.
- Phase 4: GitHub Actions integration showcasing automated pass/fail PR checks with visual diff artifacts.