Hi everyone,
I’m fine-tuning the TFT5ForConditionalGeneration model (“t5-small”). Before doing model.fit() and save the model I set the output_score=True as well as the relevant parameters in the generation config. After fine-tuning the model I load it and I can see that in the loaded model the config.output_scores = True as well as the generation_config.output_scores=True.
However, when I generate the text, the scores returns empty.
How can I fix this please? I read documentation https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation.TFSampleEncoderDecoderOutput, https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/text_generation#transformers.TFGenerationMixin but wasn’t able to figure out how to get the scores in the generated text.
I’m using transformers version 4.36.0
from transformers import TFAutoModelForSeq2SeqLM, GenerationConfig, AutoConfig
config = AutoConfig.from_pretrained("t5-small")
config.output_scores = True
transformer_model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small", config=config)
generation_config = GenerationConfig(**transformer_model.generation_config.__dict__)
generation_config.max_new_tokens = 10
generation_config.return_dict_in_generate = True
generation_config.output_scores = True
transformer_model.save_pretrained(transformer_model_path)
generation_config.save_pretrained(transformer_model_path)
# text generation
input_ids = [[283, 1890, 6, 2733, 18, 715, 621, 29, 32, 106, 4804, 89, 17, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
generated_text = transformer_model.generate(input_ids, generation_config=generation_config, return_dict_in_generate=True, output_scores=True)
print(generated_text.scores) # returns empty tuple