Extend load_from_disk and save_to_disk to remote storage

Hello everyone,

As datasets are sliced, diced, and transformed it makes sense to load/save them from disk for future use. But why just local disk when this load/save can be extended to any remote storage (S3, GS etc)?

It would be great to have remote storage load/save added in the next release :slight_smile:

Many thanks

1 Like

You’re right that would be cool :slight_smile:
Thanks for the suggestion

We think it could also be useful to have some sort of versioning for hosted datasets. Do you feel like it could be something you’d be interested to have ?

That would be nice to have. I guess you can have the mechanism for versioning currently available. I am more “concerned” about hooking up and changing different storage providers seamlessly and easily. Also, how to expose API to users for service authentication etc.

+1 for versioning.