Chatbot using Reformer model. Not able to find "token_type_ids" like BERT

🤗Transformers
1

I want to use Reformer model as Chatbot.
So in order to train it along with the input_ids and lm_labels. I am looking for token_type_ids to encode the context, speaker1 and speaker2 information.

But from the documentation of Reformer

*class* `transformers.` `ReformerForMaskedLM` 
`forward` ( *input_ids=None* ,  *position_ids=None* ,  *attention_mask=None* ,  *head_mask=None* ,  *inputs_embeds=None* ,  *num_hashes=None* ,  *labels=None* ,  *output_hidden_states=None* ,  *output_attentions=None* ,  *return_dict=None* )

I do not see “token_type_ids”. How do I fine-tune this model to work as a chatbot?

Also, How do I train this model from scratch in some other language for example German.

2

there are no token_type_ids for reformer, you could some special delimiter tokens to separate context and speakers.

This article could help