Cannot do git pull from a repo at HF: lfs fails

NickKolok · May 5, 2024, 4:30pm

Yesterday and today I got this error while trying to pull from my HF repo:

$ LC_ALL=EN_US git pull
Updating 5035f4707f..60dd6f92e8
Downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (554 KB)
Error downloading object: 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf): Smudge error: Error downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf3e88271904e81b5b31fdf8e795d589cf2de2a89c7c9925abaf33b42cc): batch request: unexpected end of JSON input

Errors logged to /media/work2/git/nn/generated/Meryl_generated/.git/lfs/objects/logs/20240505T192401.594816374.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png: smudge filter lfs failed

(the previous successful pull was on 30 April UTC).

Additional information:

$ git remote -v
origin  git@hf.co:NickKolok/Meryl_generated (fetch)
origin  git@hf.co:NickKolok/Meryl_generated (push)

$ git --version
git version 2.20.1

Can you please help me with this issue? I’m a bit frustrated because it looks like smth is broken on HF side.

NickKolok · May 5, 2024, 4:40pm

UPD: the log file mentioned above.

$ cat /media/work2/git/nn/generated/Meryl_generated/.git/lfs/objects/logs/20240505T192401.594816374.log
git-lfs/2.3.4 (GitHub; linux amd64; go 1.8.3)
git version 2.20.1

$ git-lfs filter-process
Error downloading object: 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf): Smudge error: Error downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf3e88271904e81b5b31fdf8e795d589cf2de2a89c7c9925abaf33b42cc): batch request: unexpected end of JSON input

unexpected end of JSON input
batch request
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrap
        github.com/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
        github.com/git-lfs/git-lfs/tq/api.go:55
github.com/git-lfs/git-lfs/tq.Batch
        github.com/git-lfs/git-lfs/tq/api.go:37
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:426
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
        github.com/git-lfs/git-lfs/tq/transfer_queue.go:348
runtime.goexit
        /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197
Error downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf3e88271904e81b5b31fdf8e795d589cf2de2a89c7c9925abaf33b42cc)
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.Wrapf
        github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.downloadFile
        github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:96
github.com/git-lfs/git-lfs/lfs.PointerSmudge
        github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:61
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
        github.com/git-lfs/git-lfs/lfs/pointer.go:64
github.com/git-lfs/git-lfs/commands.smudge
        github.com/git-lfs/git-lfs/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/commands.filterCommand
        github.com/git-lfs/git-lfs/commands/command_filter_process.go:116
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra/command.go:653
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra/command.go:728
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra/command.go:687
github.com/git-lfs/git-lfs/commands.Run
        github.com/git-lfs/git-lfs/commands/run.go:68
main.main
        github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
        /usr/lib/go-1.8/src/runtime/proc.go:185
runtime.goexit
        /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
        github.com/git-lfs/git-lfs/errors/types.go:170
github.com/git-lfs/git-lfs/errors.NewSmudgeError
        github.com/git-lfs/git-lfs/errors/types.go:256
github.com/git-lfs/git-lfs/lfs.PointerSmudge
        github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:70
github.com/git-lfs/git-lfs/lfs.(*Pointer).Smudge
        github.com/git-lfs/git-lfs/lfs/pointer.go:64
github.com/git-lfs/git-lfs/commands.smudge
        github.com/git-lfs/git-lfs/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/commands.filterCommand
        github.com/git-lfs/git-lfs/commands/command_filter_process.go:116
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra/command.go:653
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra/command.go:728
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra/command.go:687
github.com/git-lfs/git-lfs/commands.Run
        github.com/git-lfs/git-lfs/commands/run.go:68
main.main
        github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
        /usr/lib/go-1.8/src/runtime/proc.go:185
runtime.goexit
        /usr/lib/go-1.8/src/runtime/asm_amd64.s:2197

Current time in UTC: 
2024-05-05 16:24:01

ENV:
LocalWorkingDir=/media/work2/git/nn/generated/Meryl_generated
LocalGitDir=/media/work2/git/nn/generated/Meryl_generated/.git
LocalGitStorageDir=/media/work2/git/nn/generated/Meryl_generated/.git
LocalMediaDir=/media/work2/git/nn/generated/Meryl_generated/.git/lfs/objects
LocalReferenceDir=
TempDir=/media/work2/git/nn/generated/Meryl_generated/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/media/work2/git/nn/generated/Meryl_generated/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_EXEC_PATH=/usr/lib/git-core
GIT_PREFIX=
GIT_REFLOG_ACTION=pull

Client IP addresses:

(I would prefer not to post my IP address)

NickKolok · May 5, 2024, 4:55pm

Also, I tried to add another remote repo - https variant. However, it doesn’t help at all!

$ git remote -v
https   https://huggingface.co/NickKolok/Meryl_generated (fetch)
https   https://huggingface.co/NickKolok/Meryl_generated (push)
origin  git@hf.co:NickKolok/Meryl_generated (fetch)
origin  git@hf.co:NickKolok/Meryl_generated (push)

$ git pull https main
Username for 'https://huggingface.co': NickKolok
Password for 'https://NickKolok@huggingface.co': 
Из https://huggingface.co/NickKolok/Meryl_generated
 * branch                  main       -> FETCH_HEAD
Обновление 5035f4707f..fbca585935
Downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (554 KB)
Error downloading object: 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf): Smudge error: Error downloading 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png (bc3c9bf3e88271904e81b5b31fdf8e795d589cf2de2a89c7c9925abaf33b42cc): batch request: unexpected end of JSON input

Errors logged to /media/work2/git/nn/generated/Meryl_generated/.git/lfs/objects/logs/20240505T194554.33615992.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: 20231104-23-lametta-7k-2400/600x1000px_50s_EulerA_lpw_528178e/diffused__2024-05-02---05-59-58_.png: smudge filter lfs failed

The same thing happens with at least two more of my repos. It looks likea bug or unannounced breaking change on the HF side. Please help me!

NickKolok · May 5, 2024, 7:20pm

The discussion is continued here: Git lfs fetch with git protocol is failing - #10 by NickKolok