Not sure if transformer models are required here. I think you should be fine with something like sent2vec and doing similarity searches with faiss.
This recent approach (labse) by Google might be useful, too as they do the steps above in one go, it seems.