Dear Team,
I use the map to process data, then 300GB dataset becomes 3TB cache, and run out of my device storage.
possible solutions:
- I understand we can use set_transform to process on the fly. But how could I do remove_columns after using set_transfom? as I have a remove_columns after map.
May I know if you have a good solution for this? thank you!