I want to download a HuggingFace dataset, e.g.
from datasets import load_dataset ds = load_dataset("uonlp/CulturaX", "en")
However, it downloads on one thread at 50 MB/s, while my network is 10 Gbps. Since this dataset is 16 TB, I’d prefer to download it faster so that I don’t have to wait for a few days. How can I multithreadedly download a HuggingFace dataset?