Tool Problem or Model Problem? How Clay Tells the Difference
Clay uses LangSmith to diagnose whether agent failures stem from tools or model inference
“If someone says there's a problem with the actual answers that our agents are outputting, being able to understand like, was that a tool problem? Was that an inference problem just kind of where in the stack that actually kind of fell down?”
Clay describes using LangSmith as a production profiling layer to diagnose customer-reported issues at scale, separating tool failures from model inference failures in their agent stack. The insight highlights a growing operational challenge for AI product teams: attributing failures in multi-step agentic pipelines. While practically useful, this is a vendor case study rather than a novel industry development.