Late to the party, but it’s always recommended to do 1 pretrained model == 1 repo. It allows to have a download counter per model (allowing you to know which models are getting more traction), better discoverability for users on the Hub, dedicated community tabs per variant, etc.
(related: Add TorchGeo to libraries by isaaccorley · Pull Request #1464 · huggingface/huggingface.js · GitHub)