Hello,
I would like to build a translation pipeline with a Bert2Bert EncoderDecoder model.
Calling an input tokenizer, then the custom Bert2Bert model, and finally the output tokenizer goes fine. However building a pipeline is very hard because of the tokenizer: the pipeline function accepts only 1 tokenizer:
generator = pipeline(task=“text-generation”, model=model, tokenizer=tokenizer)
How can I build a “bilingual” BertTokenizerFast which encodes french inputs in regular context, and decodes english targets under the as_target_tokenizer() context ? (just like the MarianMT tokenizer)
Any help would be esteemed deeply