For anyone wondering how to fix this issue, I found that when using the channel conda-forge there is no issue at all. So the fix:
conda install -c conda-forge transformers
conda install importlib-metadata
4 Likes