I am using load_dataset to load the new Wikipedia dataset by:
load_dataset(“wikipedia”, language=“en”, date=“20230301”, beam_runner=“DirectRunner”)
I have successfully download the whole data, however, the procedure gets stuck after downloading it and I have to interrupt it:
Hi, just a hint from my experience preprocessing the
I successfully preprocessed the
es dump from
20230320, and that dataset is roughly 4 GB which is much smaller than the
en dataset you chose. The preprocessing took about 3 hours and consumed up to 35 GB of ram, but it worked. Currently, there is one annoyance: no indication of progress, except if you monitor the size of the temporary file being created in the folder
~/.cache/huggingface/datasets/wikipedia/en/2.0.0/.... You can see that file growing about every 2 minutes.
So I’m guessing in your case the processing will take at least 12 hours and consume much more ram than 35 GB.
A good alternative to preprocessing locally would be to use DataflowRunner, but I don’t precisely know how to craft the
beam_options arguments to do so.
My 2 cts!
Cheers and take care