What was the reason for NOT using Feather V2 file format for datasets on-disk?

🤗Datasets
1

As noted here Load

“For now only the Arrow streaming format is supported. The Arrow IPC file format (also known as Feather V2) is not supported.”

I am curious to know an engineering reason for that. After all, Feather V2 has an IPC bytestream embedded in the file at a certain offset, so using Feather V2 would allow to read in streaming format from that offset, but the reverse is not true (applications expecting Feather V2 would fail on pure IPC bytestream).

2

We started with the streaming format and IPC file format is simply not supported yet, but should be pretty easy to support given the formats are roughly the same :slight_smile: