Problem Description To feed a given sequence to a decoder-only model and obtain its past_key_values, I tried two approaches: Directly take the whole sequence as input_ids First feed part of the sequence and obtain its past_key_values, then feed the past_key_values and the rest of the sequence…

I have found a post that could explain this: Possible Bug with KV Caching in Llama (original) model · Issue #25420 · huggingface/transformers · GitHub In short: Using KV cache will change the logits, especially when the model is loaded in 16-bit precision.

Outputs change if re-using KVCache (past_key_values) for model.forward and generation

🤗Transformers

John6666 January 20, 2025, 4:38pm 2

I tried the code and got a warning, but I wonder if this is related to the change.

Topic		Replies	Views
Model.generate use_cache=True generates different results than use_cache=False Intermediate	3	347	March 4, 2025
Storing and loading KV cache 🤗Transformers	6	1566	October 21, 2024
Past_key_values - why not past_key_values_queries? Beginners	5	11249	October 15, 2023
The way to get Seq2SeqLM's `decoder_input_ids` to obtain `past_key_values` Beginners	0	1359	October 25, 2020
GPT-2 Forward w/ and w/o caching of past gives different results Beginners	0	421	May 31, 2022

Outputs change if re-using KVCache (past_key_values) for model.forward and generation

Related topics