Any examples on VisualBERTforMultipleChoice

[Petrus]

Hi,

Would anyone know any examples on how to use VisualBERTforMultipleChoice, or any similar examples? I am mostly looking for an example that can showcase how I need to tokenize my text data and perform visual feature extraction of my images, as well as how to input my multi-class labels to the model.

I would like to build something similar to this paper using radiology images and text reports and train a model to predict 14 classes (thoracic diagnosis):

2009.01523.pdf

Here is the hugging face transformer model I plan to use:

VisualBERT

If someone would have a good example on how to do this with hugging face please share. Thanks!

[Petrus]

Actually, if anyone would be able to share an example of a vision language transformer model? Preferably trained on a multi-classification problem, but any task would be helpful. Even better, if it uses the SageMaker API. Thank you!