How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI
Harbor is an open source framework for evaluating agents on real tasks in isolated sandboxes
“releasing products based on evidence, not intuition”
Arize AI engineer Xander Song introduced Harbor, an open source agent evaluation framework that runs agents against realistic tasks with real data in isolated sandboxes, using verifiers to score outcomes. The framework enables evidence-based iteration by keeping tasks and verifiers stable while changing agent implementations, surfacing regressions or improvements quickly. It is a practical engineering tool for teams building production agents who need systematic evaluation beyond manual testing.