Hi all, I am very new to this space (programming in general) and so learning about models and pipelines is quite fascinating.
I am also new to this ‘git’ concept which I find quite confusing. When I upload a model to HF it gets stored as a “commit” or “revision”. That is pretty cool, since I can go back to an earlier commit and retrieve older model files before it got ruined by overtraining.
But I am then concerned about the storage I am taking up on HF with some of these older versions.
How do I permanently delete old commits or revisions through CLI, GUI or python? So specifically if I have 5 commits to date on model, i may want to delete commits 3 and 4, but keep 1, 2 and 5.
Hope someone can help.
Note that git only stores the difference (diff) between files, not the files themselves. However, if you update an entire neural network (all the weights), then each commit will have its own pytorch_model.bin file, which can be several gigabytes large.
In git, you can squash commits together. So in your case, you could squash commits 3, 4 and 5 into a single commit, which will only store the latest version (commit 5). So to do that, first clone a repository from the hub to your local computer:
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone <URL>
You can add the GIT_LFS_SKIP_SMUDGE=1 to not load the big files.
Next, squash the last 3 commits into one as explained here:
git reset --soft HEAD~3 && git commit
then you can push the updates back to the HF hub:
git push
Hi and thank you for your reply!
However, if you update an entire neural network (all the weights), then each commit will have its own
pytorch_model.binfile, which can be several gigabytes large
Exactly! that was my concern
In git, you can squash commits together. So in your case, you could squash commits 3, 4 and 5 into a single commit, which will only store the latest version (commit 5).
Understood. I appreciate the info and the related code/link! I will try it out on some dummy files before squashing the actual model folder.