Can I Use upload_folder in Multiprocessing Mode for Large Subfolders on Hugging Face Hub?

Hello Wauplin,

Thanks for the quick reply and the details about the script. It seems well-equipped for large datasets. How does its speed and robustness compare to the standard upload_folder method from Hugging Face Hub, which I thought might already handle large uploads effectively?