Unsupported numpy type when calling ds.add_column()

🤗Datasets
1

python 3.9.7
datasets 2.5.2

I am running into an issue whenever I try to add a column to a dataset I am working with. Below is some sample code:

import numpy as np

z_train = np.array(ds['train']['re_part'].numpy(), dtype=complex)
z_train.imag = ds['train']['im_part'].numpy()
ds['train'].add_column(name='complex_col', column=z_train)

ds['train']['re_part'] and ds['train']['im_part'] are initially pytorch tensors which is why I needed to convert them to numpy arrays first. I get the following error:

Exception has occurred: ArrowNotImplementedError
Unsupported numpy type 15

and I’m not entirely sure what that is referring to. I checked the types of z_train and they are all numpy arrays until you get to the individual elements, in which case they are numpy.complex128. Any ideas? Thanks in advance for your help!

2

Arrow doesn’t support complex numbers unfortunately.
I think you’ll need keep storing the real and imag parts separately