Speculative Decoding: Faster Inference Without Changing the Output
Decoding is bound by memory bandwidth, not arithmetic, which is the gap speculative decoding exploits. How draft-and-verify keeps the output distribution intact, why acceptance rate collapses as you speculate further ahead, and what recent measurement says about heavy load.
