Scaling first and debugging later is an expensive habit. @​TheZachMueller does the reverse: he proves the job on the GPUs he already has, drives out every hidd

Scaling first and debugging later is an expensive habit. @​TheZachMueller does the reverse: he proves the job on the GPUs he already has, drives out every hidden wait, then promotes the exact same code to bigger GPUs. In this video, he trims a 500-million-parameter mixture-of-experts from 62 hours…