1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Analysis of 1,368 AI failures reveals enterprises need structured 3-dimensional evaluation frameworks
“the model itself, or the technical power of the model, is only one part of the system. We must look at the system holistically.”
New York Life researchers mined the public AI Incident Database to extract evaluation lessons from 1,368 real-world AI failures, finding that incidents are rising exponentially in the foundational-model era. They propose a three-dimensional evaluation framework covering technical strength, deployment controls, and business context, arguing that model capability alone is an insufficient signal of system safety. The practical framing—moving from isolated model evals to holistic system-level assessment—offers direct guidance for enterprise AI risk programs.