The KV cache can speed up inference because after the first iteration, the KV of the post sentance will not change, therefore we can store it. However, every iteration will append the input sentance with the new token, and in my opinion, that means the position encoding should be changed, and thus …

Isn't KV cache influenced by position encoding in inference?

OrfeasFil May 15, 2024, 4:18pm 2

Hello there,

Did you find any answer? I have the same question

Topic		Replies	Views
Transformer KV-Cache Produces Worse Output Than Normal Generation – Why? Beginners	1	263	March 3, 2025
Outputs change if re-using KVCache (past_key_values) for model.forward and generation 🤗Transformers	5	283	January 22, 2025
Past_key_values - why not past_key_values_queries? Beginners	5	11192	October 15, 2023
KV cache sizing 🤗Transformers	0	769	August 24, 2023
How does attention key/value caching work with models that have learned absolute position embeddings? 🤗Transformers	0	1366	September 26, 2023