While there are numerous examples and notebooks showing how to run and fine-tune pretrained models like Vision Transformers (ViT), I’m looking for a clear example of how to integrate a pretrained ViT into a PyTorch Lightning pipeline. Specifically:
- Should I instantiate the
AutoImageProcessor()
class within mypl.LightningModule
, or would it be better to do so in mypl.DataModule
? - Should I implement my own
forward
method in theLightningModule
, or should I simply call theforward
method of the pretrained model (which would be an attribute of my Lightning class)?