Unsupported numpy type when calling ds.add_column()

python 3.9.7
datasets 2.5.2

I am running into an issue whenever I try to add a column to a dataset I am working with. Below is some sample code:

import numpy as np

z_train = np.array(ds['train']['re_part'].numpy(), dtype=complex)
z_train.imag = ds['train']['im_part'].numpy()
ds['train'].add_column(name='complex_col', column=z_train)

ds['train']['re_part'] and ds['train']['im_part'] are initially pytorch tensors which is why I needed to convert them to numpy arrays first. I get the following error:

Exception has occurred: ArrowNotImplementedError
Unsupported numpy type 15

and I’m not entirely sure what that is referring to. I checked the types of z_train and they are all numpy arrays until you get to the individual elements, in which case they are numpy.complex128. Any ideas? Thanks in advance for your help!

Arrow doesn’t support complex numbers unfortunately.
I think you’ll need keep storing the real and imag parts separately