Converting SWIN Transformers from Pytorch through ONNX or Others

🤗Transformers
1

Context/Use case:
Thesis for University - Trying to Convert Model

Hi Everyone,

Thanks in advance for taking the time to read this and help.

I am a student at the University of Amsterdam who is currently writing my thesis about SWIN Transformers for Motion Amplification based off the Paper here. I want to work on making this model smaller e.g. TFLite.

What’s the Issue

Unable to Convert into ONNX and test.

Checkpoints and my code can be found here.

  • Currently in Possession of a PyTorch Model (Trained in PyTorch 1.11 but also a version with 10 Epochs in PyTorch 2.0.1) converted to ONNX with set magnification factors but: The input of this model is 3 input tensors (Two tensors that are images and a magnification tensor) All credits to Ricard Lato.
  • Unable to Convert this model into ONNX (unsure if ONNX is successful)

Possible Solutions / Suggestions from Me
(Open to any other ideas)

  1. Convert from PyTorch —> Tensorflow using HuggingFace such as done in the Transformer examples (Unable to link as do not have permission to add more than 2)
  2. Convert from PyTorch —> ONNX —> Tensorflow light
  3. Convert Model from Pytorch to TensorFlow by rewriting the model from the original model reference above and retraining

Uncertainties

  • Is this even possible in regards to converting SWIN Transformers from PyTorch —> ONNX —> TFlite