I was using attention_mask so it was working with 0 but for input_ids it’s 1. I checked config file, padding token id is 1.
new code would look like this:
labels = inputs['input_ids'].clone()
labels[inputs['input_ids'] == 1] = -100
inputs['labels'] = labels