Cerebras Big Chip Club: Dropout, Test-Time Compute, and Wafer-Scale AI
Cerebras achieves 1.5x inference speedup via layer dropout enabling early exit in LLMs
“you can make like very high impact contributions. Working at a hardware company rather than like a model builder or an application level company. If the hardware becomes widely adopted then any impact that you had will be multiplied by every other layer of the stack.”
Cerebras researchers at ICML presented 'Don't Drop Dropout,' showing that layer dropout — applied more aggressively to deeper layers — trains models whose intermediate representations are ready to produce outputs without traversing all layers. This enables early exit inference with a demonstrated 1.5x speedup. The technique is hardware-adjacent, positioning Cerebras's wafer-scale chips as a natural accelerator for sparse, dropout-enabled training.