What if I just want to transform the dataset and then save in a streaming way? If the dataset is large, the CPU occupied memory become larger when transfering the dataset. Or should I transfer and save the data in parts instead of waiting until the entire dataset has been transferred?
1 Like