During early training loops of Nova-Mini V1, autoregressive generation without caching re-computed attention over the entire prompt at every single token step, causing latency to spike to 4,570ms.
By implementing an explicit past_key_values tensor cache with Multi-Query Attention (7Q / 1KV), per-step generation dropped to 50–80ms on standard CPU hardware.