GPT2Model model output inconsistency between different transformers versions

[John6666]

Possibly related this phenomenon.

Ask for help: Output inconsistency when using LLM batch inference compared to single input

I found single LLM input get different output logits when merging into a batch for inference. Besides, I need to use inputs_embeds as model input. My test LLM is “Qwen/Qwen2.5-1.5B-Instruct” and the test code is below. from transformers import AutoModelForCausalLM, AutoTokenizer import torch # load model and tokenizezr model_name = "Qwen/Qwen2.5-1.5B-Instruct" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto", trust_remote_code=Tr…

Also, the part that has changed a lot recently is the KV cache-related area, which seems to have changed quite a bit.