Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built in Multi Token Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so ...