Agent Evaluation
Objective, code-driven evaluation harnesses for AI agent behavior — testing side effects, enforcing guardrails, and replacing subjective quality checks.
1 / Beyond Vibe Checks
The standard approach to evaluating AI agents is subjective: run the agent, read its output, and decide if it 'looks right.' This approach does not scale, is not reproducible, and cannot detect subtle failures like incorrect side effects or security boundary violations. Agent evaluation, as I implemented it in Vigil, replaces subjective quality assessment with objective, code-driven pass/fail assertions.
2 / State-Based Assertions
The core evaluation strategy was testing agent final state and side effects rather than raw text output. After an agent executed a task inside an ephemeral Docker sandbox, the evaluation harness inspected the sandbox state: filesystem changes, database modifications, environment variable values, and created files. These state-based assertions provided reliable verification regardless of slight text variations in the agent's explanatory output.
3 / Security Policy Testing
Vigil's evaluation harness included automated prompt injection stress-tests against predefined security policy baselines. Test cases attempted to convince the agent to execute banned commands, access restricted file paths, or exceed resource quotas. The harness verified that the agent refused these requests and that the container sandbox enforced hard limits even if the agent attempted violations.
4 / Custom Pytest Architecture
The evaluation harness was built on Pytest with custom plugins that managed sandbox lifecycle — provisioning containers before tests, capturing execution logs, and cleaning up after assertions completed. Each test case defined expected pre-conditions and post-conditions, and the harness verified state transitions deterministically.
5 / Non-Determinism Challenge
The fundamental challenge in agent evaluation is non-determinism. The same agent with the same prompt can produce different outputs across runs. The mitigation strategies: fix temperature seeds where possible, write fuzzy assertion helpers for semantic equivalence, and focus assertions on observable side effects rather than exact text output. This does not eliminate non-determinism — it makes tests resilient to it.
6 / Honest Framing
Vigil does not make agents 'safe.' It provides resource-constrained runtime isolation and objective verification — measuring what an agent actually does rather than what it claims to do. The distinction matters: safety is a claim about all possible behaviors, while evaluation is a measurement of observed behaviors under specific conditions.