Do the tokenizers in the transformers package support tokenization of triplets?
Lets assume we’re dealing with a VQA dataset. Each entry in the dataset contains the following information:
- Image name
- 5 possible answers (1 is correct)
- Image captions
I would like to able to represent each input as:
[CLS] Q + [SEP] + A + [SEP] + CAPTIONS