How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Enterprise AI teams lack benchmarks and expert access to evaluate domain-specific agents
“we are data scientists or AI engineers who can build agents, but we are not subject matter experts”
An Air Canada AI engineer describes the compounding evaluation problem facing enterprise AI teams: no public benchmarks exist for business-specific agent tasks, and the domain experts who could validate outputs are siloed, busy, and largely inaccessible. The talk draws on experiences at both Air Canada and a prior telecom deployment to illustrate how teams ship agents without rigorous evaluation. The insight matters because it names a structural gap—AI builders and domain knowledge are organizationally separated—that most enterprise AI teams quietly face but rarely address systematically.