TFT5ForConditionalGeneration generate returns empty output_scores

🤗Transformers
1

Hi everyone,
I’m fine-tuning the TFT5ForConditionalGeneration model (“t5-small”). Before doing model.fit() and save the model I set the output_score=True as well as the relevant parameters in the generation config. After fine-tuning the model I load it and I can see that in the loaded model the config.output_scores = True as well as the generation_config.output_scores=True.

However, when I generate the text, the scores returns empty.
How can I fix this please? I read documentation https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation.TFSampleEncoderDecoderOutput, https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/text_generation#transformers.TFGenerationMixin but wasn’t able to figure out how to get the scores in the generated text.

I’m using transformers version 4.36.0

from transformers import TFAutoModelForSeq2SeqLM, GenerationConfig, AutoConfig

config = AutoConfig.from_pretrained("t5-small")
config.output_scores = True
transformer_model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small", config=config)

generation_config = GenerationConfig(**transformer_model.generation_config.__dict__)
generation_config.max_new_tokens = 10
generation_config.return_dict_in_generate = True
generation_config.output_scores = True

transformer_model.save_pretrained(transformer_model_path)
generation_config.save_pretrained(transformer_model_path)

# text generation
input_ids = [[283, 1890, 6, 2733, 18, 715, 621, 29, 32, 106, 4804, 89, 17, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
generated_text = transformer_model.generate(input_ids, generation_config=generation_config, return_dict_in_generate=True, output_scores=True)

print(generated_text.scores)  # returns empty tuple
2

Hi!

Don’t know if you are still struggling with this, I stumbled upon this post because my fine-tuned BLIP2 model was returning empty strings.

I fixed it when I realized that I had to set the same data type for my processor and model which in my case was torch.bfloat16

My code looks like this

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
base_model = "path/models/blip2-opt-2.7b"
model = Blip2ForConditionalGeneration.from_pretrained(base_model,torch_dtype=torch.bfloat16)
adapter_model = "path/finetuned_models/finetuned_blip2-opt-2.7b.pt"
peft_model = PeftModel.from_pretrained(model, model_id=adapter_model, device=device)
peft_model.to(device)
processor = AutoProcessor.from_pretrained(base_model)

# generating captions
image = dataset[0]["image"] # using the first dataset item for experimenting
# <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1166x1750>
inputs = processor(image, return_tensors="pt").to(device, torch.bfloat16)
generated_ids = peft_model.generate(**inputs, max_length=50)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
1 Like