Hi @gruffgoran, your use cases sound like a perfect match for the ONNX format 
Having said that, you might be able to get a quick win by trying something like the following (see docs):
tf_model = TFBertForSequenceClassification.from_pretrained("KB/bert-base-swedish-cased", from_pt=True)
From here you can then run inference / fine-tune etc using TensorFlow.
If you want to go the ONNX route, the idea would be to convert PyTorch → ONNX and then load the ONNX model in TensorFlow. Details on doing the conversion can be found here: Exporting transformers models — transformers 4.3.0 documentation