The Hallway Track
8 tracked signals on data-quality.
Drug discovery AI hits scaling walls from data information limits, not model size or compute
“test loss flatlines after 1.5B parameters while training loss continues to drop as you scale, that tells you that your model is limited by the amount of information in your data”
Benchling sends the same task to multiple model families and cross-compares; disagreement flags likely errors for human review.
“Because what we saw was if two models disagree, there's usually an error.”
Low-quality RL training environments and broken harnesses actively degrade models and ruin training runs.
“researchers don’t want your broken RL environments because they will make our models worse”
Biomedical imaging AI is bottlenecked by data quality and availability, not model architecture
Quality beats quantity in SFT: 1,000 curated examples can match models trained on far more data.
“A wrong demonstration gets imitated.”
AI quality gains come from better data access, not prompt or model optimization.
“when AI has access to the right data, it spends more time reasoning and coming up with the right outcomes and higher quality outcomes”
Snorkel argues model improvement should come from high-quality, expert-in-the-loop data rather than scaling model size.
“starting from the research-backed work that Snorkel's been doing since its inception.”
Snorkel argues task quality and data quality are the same, driving agentic training outcomes
“the quality of data is critical”