Hi, I am looking for a way to download a large dataset, transform it and then upload it to another location. Note that the transformations for each instance is independent of others. I can load the dataset in streaming mode and start the transformation but cannot find a way to write to huggingface…

Streaming for Saving

John6666 September 12, 2025, 11:37pm 4

Yeah. It is now possible to save parquet files per shard or upload them incrementally.

1 Like