[RAG]AttributeError: 'RagTokenizer' object has no attribute 'convert_tokens_to_ids'

yuchaozhao · March 2, 2023, 9:13pm

Hi,

I tried to set a symbol to stop the RAG model’s generation. I saw there is ‘eos_token_id’ parameter in model.generate(), but the RagTokenizer doesn’t have ‘convert_tokens_to_ids’. How can I use the ‘eos_token_id’ to stop the generation with certain symbol like “%” or “*”? Thank you.

Topic		Replies	Views
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation Beginners	3	35375	August 29, 2022
Trying RAG with other Retriever Models 🤗Transformers	0	300	January 21, 2021
What is the difference between tokenizer.eos_token_id, model.config.eos_token_id and model.generation_config.eos_token_id? Beginners	1	293	April 22, 2024
min_length in generate method Models	0	322	October 7, 2021
Facing an error in Bert NLP "At most 4 tokens in tensor([ 2, 2, 2, 2, 44763, 44763, 2, 44763]) can be equal to eos_token_id: 2. Make sure tensor([ 2, 2, 2, 2, 44763, 44763, 2, 44763]) are corrected." Models	0	302	May 6, 2021

[RAG]AttributeError: 'RagTokenizer' object has no attribute 'convert_tokens_to_ids'

Related Topics