Converting finetuned Pytorch Whisper model to HF

I have a custom fine tuned pytorch based Whisper model. I would like to convert this model to HuggingFace format so that I can fine tune it further using the HuggingFace way.
Can you please let me know how to do this?

I understand that there is already a Whisper model in HuggingFace format on HuggingFace repository but I can not use that because of some restrictions on my end.