Is There a Way to Improve Memory Usage When Using Identical `past_key_values` for All Samples in a Batch?

🤗Transformers
3

do you mean prompt cache, it can reuse the kv cache of some prefix prompt？https://discuss.huggingface.co/t/how-to-cache-common-instruction-prompt/101419

1 Like