Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Fixing one agent issue often breaks other behaviors, requiring observability-driven regression testing
“the biggest problem is not identifying what's broken, but that after you've made the fixes, you have to make sure you haven't broken 10 other things”
Sofia Jakovcevic of Arize AI argues that agent failures occur at runtime rather than the code level, making traditional regression testing insufficient. She proposes a five-step observability-driven regression loop that can be automated into CI/CD pipelines to catch cascading failures after prompt or logic changes. The talk addresses a widespread pain point for teams shipping production agents where even a single-line prompt change can silently degrade other behaviors.