AutoModelforCausalLM fails only on Cuda due to inf/nan/<0 tensors

And perhaps related this issue: Smolagents Error: probability tensor contains either `inf`, `nan` or element < 0