Multiprocessing map taking too much memory footprint

🤗Datasets
3

Hi! What do you get when you access the cache_files attribute of your dataset? map with multiprocessing can be an issue for in-memory datasets due to data being copied to the subprocesses (more info).