I am trying to connect to MinIO using an S3 interface with the load_dataset() function. I noticed that the S3FileSystem class is marked as deprecated, and the code in similar topics uses it. I am hoping someone can point me in the right direction. If this is not yet a feature, I would be glad to help implement it (I work for MinIO) - if it is currently under development, I can help test it.
Hi! You can use s3fs.S3FileSystem instead of datasets.filesystems.S3FileSystem. datasets.filesystems.S3FileSystem is a dummy wrapper around s3fs.S3FileSystem with a different docstring.
Thanks @mariosasko - do you know if anyone is considering adding this capability to the load_dataset() function? What would be nice is if you could pass an S3FileSystem object to the load_dataset() function along with a dictionary of s3 based URLs - much like what you can do with files.
We will eventually support this, but this is not a high priority right now (we are more focused on improving the Hub integration).