Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health
ECRI named AI chatbot misuse the #1 health technology hazard of 2026, not a frontier problem but a production baseline issue.
“Most AI safety failures in health care are not model failures. They are architectural decisions that were made before even a single token was generated.”
Hinge Health's AI/ML lead Rashi Agrawal presents a practitioner framework for shipping safe member-facing health AI, grounding the talk in documented harms: a consumer health AI under-triaged life-threatening emergencies 50% of the time, and ECRI ranked AI chatbot misuse as the top health tech hazard of 2026. Her three non-negotiable principles — constraint is architecture, deterministic rules sit above the model, and safety is a continuous evaluation layer not a launch gate — offer concrete engineering guidance for teams building in high-stakes domains.