What If We Stopped Using GPUs? | YC Paper Club
GPU design shifted from compute efficiency to memory bandwidth after transformers emerged
“attention is equal to n-squared”
A YC Paper Club speaker traces how Nvidia's hardware priorities shifted after GPT-2/GPT-3 — from maximizing FLOP efficiency to scaling memory capacity and bandwidth, driven by the O(n²) memory demands of attention mechanisms. The talk frames this as a co-adaptation between hardware architecture and optimizer design, implicitly questioning whether GPU-centric compute remains optimal. The signal is relevant to AI infrastructure debates but the content is heavily degraded auto-generated captions, limiting extractable insight.