Hello Hugging Face Community! I’m currently working on a project where I frequently batch process with identical past_key_values (pkv) for all samples in a batch. I calculate past_key_values once, then duplicate it as many times as I need to save on compute. The typical scenario involves having som…

Is There a Way to Improve Memory Usage When Using Identical `past_key_values` for All Samples in a Batch?

RaushanTurganbay October 21, 2024, 5:31pm 4

this page in the docs should help

2 Likes

Topic		Replies	Views
Efficient batch inference using stacked past_key_values for multiple continuation candidates Models	1	27	June 10, 2025
Forge synthetic past_key_value batch from multiple outputs Intermediate	0	478	May 12, 2021
Outputs change if re-using KVCache (past_key_values) for model.forward and generation 🤗Transformers	5	242	January 22, 2025
KV caching for varying length texts 🤗Transformers	1	164	December 16, 2024
Past_key_value with multiple new tokens Intermediate	1	1364	August 10, 2023