Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab
Static AI benchmarks create a false reliability signal that collapses under real production usage
“I like to call this the benchmark illusion.”
Felipe Blanes of Amazon AGI Lab argues that optimizing for static benchmarks—public or synthetic—produces a 'benchmark illusion' where lab scores don't predict production reliability, because real customers do things developers never anticipated. He frames this through the lens of Nova Act, Amazon's browser-automation agent that shipped as a first-class AWS service in December after a March research launch. The core thesis is that 80% task reliability is insufficient for industrial-grade agents, and that customer-grounded evals are required to close the gap between benchmark performance and trust.