Dataset set_format

[dhruvgrammarly]

@lhoestq

Under the hood all the data are in Arrow format. Arrow has a good interoperability with numpy that allows to cast arrow objets to numpy fast and with zero-copy. Then from numpy the conversion to pytorch is also very fast.

This seems inconsistent with what the documentation at

to_numpy(*self*, *zero_copy_only=False* )

Return a NumPy copy of this array (experimental).

Parameters:

**zero_copy_only** [bool](), default `False`
Introduced for signature consistence with pyarrow.Array.to_numpy. This must be False here since NumPy arrays’ buffer must be contiguous. 

This suggests that it’s making a copy of the data and not doing a zero copy to numpy arrays. I’m also running into a problem when loading it as numpy array or python list seems equally slow. Maybe I’m doing something horribly wrong. Create batch from list of ids in the dataset is very slow - #4