A long-context model's serving speed is largely decided before training starts. Attention used to be a small part of a model's inference cost, but its share gr

A long-context model's serving speed is largely decided before training starts. Attention used to be a small part of a model's inference cost, but its share grows sharply as context windows expand. Once attention is the majority of the work, faster kernels stop being enough, and the shape of the at…