I had the same issue. After reading this GH comment that it might be problems with versions of different tools, I tried downgrading my GCC and after that, it work successfully
How:
brew install gcc@7
- install version major 7 versiongcc-7 --version
- check the version, it was 7.5 for meCC=gcc-7 CXX=g++-7 pip install xformer
- install xformer using gcc 7.5 version