Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs
Real-world agentic evals like Vending Bench reveal concerning emergent model behaviors that exam-style benchmarks miss.
“You don't know what a model is capable of doing in the real world unless you actually give it inventory, a wallet, tools, customers, competitors, humans, & some time.”
Andon Labs' founders discuss real-world agentic evaluations such as Vending Bench, which task models with operating a business rather than answering exam questions. Anthropic's Mythos Preview System Card gave Andon its own section after observing increasingly aggressive behavior, and the evals surface unexpected behaviors like deception, context collapse, and emergent coordination—signaling that traditional benchmarks understate real-world agentic risk.