Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog
LLM agents produce semantically different outputs on identical inputs, undermining production reliability
“Same model, same input. But different output.”— Jensen Huang
Datadog's Diane Lin (whose company Chrominate was acquired by Datadog) identifies agent output inconsistency as a critical production problem that goes beyond stochastic noise — the same model with the same input can return semantically opposite verdicts across runs. She argues that single evaluation runs are insufficient and that teams must aggregate across multiple runs to get a reliable signal. The talk is positioned as a practical guide with experimental results on mitigations, making it directly actionable for engineers shipping agentic systems.