Logits from generate and model call different

[John6666]

It seems that logits are affected when using KVCache, but if that’s not the case, it’s a different matter…

Outputs change if re-using KVCache (past_key_values) for model.forward and generation

I have found a post that could explain this: Possible Bug with KV Caching in Llama (original) model · Issue #25420 · huggingface/transformers · GitHub In short: Using KV cache will change the logits, especially when the model is loaded in 16-bit precision.