Persistent disk + Spaces + Upload of data

Spaces
1

Hi all,

I’m using HF spaces + docker with persistent disk storage, and using the HfApi to upload a folder of large data files:

hf_api = HfApi()
dataset_path = os.path.join('data', 'datasets', d)
hf_api.upload_folder(
  folder_path=os.path.abspath(dataset_path),
  path_in_repo=dataset_path,
  repo_id=hf_space,
  repo_type='space',)

Which seems to get uploaded correctly:
image

However when I try to read files from /data once the docker image boots up they don’t seem to exist, using os.listdir I see an empty ‘/data’ dir once the docker image boots up.

Is there something I’m misunderstanding?

1 Like