I noticed that both load_from_disk and save_to_disk support the ‘fs’ parameters, but not load_dataset. I want to know if it is possible to load text data from hdfs via load_dataset Any suggestion will be appreciately.

Does huggingface support load raw text dataset from hdfs?

lhoestq January 9, 2022, 9:16pm 4

No it doesn’t, only lozd_dataset supports streaming mode

Topic		Replies	Views
Read data from hdfs Beginners	2	953	April 7, 2021
How to steaming .hf dataset 🤗Datasets	5	73	November 30, 2024
NotImplementedError when loading dataset with Streamlit 🤗Datasets	8	10371	June 16, 2025
Huggingface-cli to load_dataset 🤗Datasets	5	4026	July 23, 2025
Loading a dataset cached in a LocalFileSystem is not supported 🤗Datasets	3	899	July 23, 2025