which says: " Auto-regressive language generation is now available for GPT2 , XLNet , OpenAi-GPT , CTRL , TransfoXL , XLM , Bart , T5 in both PyTorch and Tensorflow >= 2.0!"
so I wanted to try to do the same, they just change the model to T5. However even though the model runs, the output is very strange. This is the code:
!pip install transformers
import tensorflow as tf
from transformers import TFT5ForConditionalGeneration, T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = TFT5ForConditionalGeneration.from_pretrained("t5-small", pad_token_id=tokenizer.eos_token_id)
input_ids = tokenizer.encode('Hello, my dog is cute', return_tensors='tf')
greedy_output = model.generate(input_ids, max_length=50)
print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))
Because of this, and taking into account that I have not found many text-generation examples with t5, I would like to ask if this is possible? if so, why my output is so strange?
Yes, using summarize task works. Here is an example I used:
from transformers import T5ForConditionalGeneration, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = T5ForConditionalGeneration.from_pretrained("t5-base")
input_ids = tokenizer.encode('summarize: I enjoy walking with my cute dog', return_tensors='pt')
greedy_output = model.generate(input_ids, num_beams=7, no_repeat_ngram_size=2, min_length=50, max_length=100)
print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))
Output:
I enjoy walking with my cute dog - he is a joy to walk with and is very affectionate with me. I like to spend time with him on walks with his kitty cat, leo, who is so cute!
I want to fine-tune the model for my task. but when I trying to use “transformers/examples/pytorch/language-modeling/run_clm.py” gives this error.
ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of RoFormerConfig, BigBirdPegasusConfig, GPTNeoConfig, BigBirdConfig, CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig, XLMProphetNetConfig, ProphetNetConfig, BartConfig, MBartConfig, PegasusConfig, MarianConfig, BlenderbotConfig, BlenderbotSmallConfig, MegatronBertConfig.
As the error indicates, you cannot use the run_clm script with a T5 model as there is no version of T5 with a causal LM head. The error listed all the possible architectures you can use instead.
I want to add some small rules when generating the output text.
For e.g:- First number should be larger than the second generating number in the generating sentence.
hello! @sgugger , I’m reproducing a model for a paper and need to use t5-base model for Casual Language Modeling. I find nothing about it, would it be possilble to ask you?