I want to use Reformer model as Chatbot.
So in order to train it along with the input_ids and lm_labels. I am looking for token_type_ids to encode the context, speaker1 and speaker2 information.
But from the documentation of Reformer
*class* `transformers.` `ReformerForMaskedLM` `forward` ( *input_ids=None* , *position_ids=None* , *attention_mask=None* , *head_mask=None* , *inputs_embeds=None* , *num_hashes=None* , *labels=None* , *output_hidden_states=None* , *output_attentions=None* , *return_dict=None* )
I do not see “token_type_ids”. How do I fine-tune this model to work as a chatbot?
Also, How do I train this model from scratch in some other language for example German.