I receive the following error when running my Trainer script via accelerate:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject
Seems to be an incompatibility with pyarrow and the installed arrow C binaries, but I’m unsure how to address this.
The following may be helpful in troubleshooting:
$ conda list | grep arrow
arrow 1.3.0 pyhd8ed1ab_0 conda-forge
arrow-cpp 11.0.0 h374c478_2 anaconda
pyarrow 15.0.2 pypi_0 pypi
pyarrow-hotfix 0.6 pyhd8ed1ab_0 conda-forge
$ conda list | grep datasets
datasets 2.12.0 py311h06a4308_0 anaconda