The history of computing is a history of layers. Each generation of engineers builds a floor and the next generation treats that floor as the ground. Assembly programmers stood on microcode, compiler writers stood on assembly, and application developers stand on runtimes so deep that a single keystroke passes through more abstraction than the Apollo guidance computer contained in total. This is not decadence; it is the only way anything of scale has ever been built. No single mind holds a modern system, so the layers are load-bearing walls for cognition itself.

Machine learning inverted part of this story. For decades the boundary between hardware and software was negotiated through instruction sets, stable contracts that changed on the scale of decades. Then the economics of matrix multiplication took over. A model is mostly a very long sequence of multiply-accumulate operations, and the value of everything above and below that sequence is measured by how little it gets in the way. Frameworks rose and fell on the latency of a kernel launch. Whole compiler stacks were written to fuse three operations into one. The abstraction tower did not disappear, but for the first time in a generation, the floor mattered again to people working at the top.

Inference is where this pressure concentrates. Training happens once, in a data center, with a budget; inference happens everywhere, forever, on whatever hardware the user already owns. A laptop that generates thirty tokens per second is a different product from one that generates three, and the difference between them is rarely the model. It is the memory traffic: how many bytes move, how far they move, and how often the same bytes move twice. Quantization exists because moving four bits costs a quarter of moving sixteen. Key-value caches exist because recomputing attention over a growing prefix is quadratic waste. Every trick in the modern inference stack is a refusal to move a byte that does not need to move.

The history of computing is a history of layers, and the layers are load-bearing. Each generation builds a floor and the next treats it as ground. The question for any new inference library is which floor it chooses to stand on, and which walls it decides to rebuild from scratch.
