Push datasets from multi machines at the same time

🤗Datasets
1

My datasets have many folders contain large files located on many different machines. These folders are separate and have nothing to do with each other.

Because my folder contains many heavy files, I want to push it from different machines instead of sending it to a single machine and pushing it.

I tried push from one machine, then go to other and run git push again, however it says that I need to pull before push. If I do a force push it will overwrite the first commit.

Is there a way I can push these folders to the same Huggingface Git repository at the same time (running git push from multiple machines at the same time)?

2

You can run git lfs install --skip-smudge to avoid downloading large files on clone/pull. This forces Git LFS to download a small pointer file instead.

Another option is to use different branches.

1 Like