How to load this simple audio data set and use dataset.map without memory issues?

No it looks fine (no need to provide batch_size though, since by default batched=False).

Are you running on colab ?

Maybe related to `load_dataset` consumes too much memory · Issue #4057 · huggingface/datasets · GitHub