How to load this simple audio data set and use dataset.map without memory issues?

[lhoestq]

@akt42

If you don’t mind me asking, how did you get this?

Since you loaded the dataset from memory using .from_pandas , then the dataset has no associated cache directory to save intermediate results.

I’ve read the docs for days but was never able to figure this out.

I guess we have to add it to the documention

Basically a Dataset os just a wrapper of an Arrow table. It can be a InMemoryTable, a MemoryMappedTable (i.e. from local files), or a combination of both as a ConcatenationTable. You can check the Arrow table with ds.data