LFS no longer works for my Huggingface models

🤗Hub
1

I’ve been able to use LFS fine so far on a whole bunch of models on Huggingface, but since tuday I’m getting this error when trying to push data with LFS :

Uploading LFS objects:   0% (0/1), 0 B | 37 KB/s, done.
LFS: Authorization error: https://s3.us-east-1.amazonaws.com/lfs.huggingface.co/repos/d7/38/d738d10b0bff88e0df117384720684e38a23e633549f988ab9a015c4f0066648/6a96896ad631726021758af7de2f6076ab54ee698300ceae28c5f1adf2450e82?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGO27GPWFUO%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T115458Z&X-Amz-Expires=900&X-Amz-Signature=f9e4f7fefab5ffec5c565438d9c1c99876a137b7618e8ad7d175e7ae7cbf3c1f&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject
Check that you have proper access to the repository

The error message contains the following XML :

<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <AWSAccessKeyId>AKIA4N7VTDGO27GPWFUO</AWSAccessKeyId>
  <StringToSign>AWS4-HMAC-SHA256
20230128T114614Z
20230128/us-east-1/s3/aws4_request
ddf0c59292a0789b110f5c5749d7ae8c430cc6e281b41394f0f72b988e2e0dab</StringToSign>
  <SignatureProvided>0dce350ebe9bd071eb56a82b83781c20cbb3504b5fdd0731a31668f36dcad0f0</SignatureProvided>
  <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 31 32 38 54 31 31 34 36 31 34 5a 0a 32 30 32 33 30 31 32 38 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 64 66 30 63 35 39 32 39 32 61 30 37 38 39 62 31 31 30 66 35 63 35 37 34 39 64 37 61 65 38 63 34 33 30 63 63 36 65 32 38 31 62 34 31 33 39 34 66 30 66 37 32 62 39 38 38 65 32 65 30 64 61 62</StringToSignBytes>
  <CanonicalRequest>GET
/lfs.huggingface.co/repos/d7/38/d738d10b0bff88e0df117384720684e38a23e633549f988ab9a015c4f0066648/459ab0e6771c051601a4999452f9fa7bb67f44a893d4d4b2de9b2da806828c3b
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&amp;X-Amz-Credential=AKIA4N7VTDGO27GPWFUO%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20230128T114614Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=host&amp;x-amz-storage-class=INTELLIGENT_TIERING&amp;x-id=PutObject
host:s3.us-east-1.amazonaws.com host UNSIGNED-PAYLOAD</CanonicalRequest>
  <CanonicalRequestBytes>47 45 54 0a 2f 6c 66 73 2e 68 75 67 67 69 6e 67 66 61 63 65 2e 63 6f 2f 72 65 70 6f 73 2f 64 37 2f 33 38 2f 64 37 33 38 64 31 30 62 30 62 66 66 38 38 65 30 64 66 31 31 37 33 38 34 37 32 30 36 38 34 65 33 38 61 32 33 65 36 33 33 35 34 39 66 39 38 38 61 62 39 61 30 31 35 63 34 66 30 30 36 36 36 34 38 2f 34 35 39 61 62 30 65 36 37 37 31 63 30 35 31 36 30 31 61 34 39 39 39 34 35 32 66 39 66 61 37 62 62 36 37 66 34 34 61 38 39 33 64 34 64 34 62 32 64 65 39 62 32 64 61 38 30 36 38 32 38 63 33 62 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 6f 6e 74 65 6e 74 2d 53 68 61 32 35 36 3d 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 4e 37 56 54 44 47 4f 32 37 47 50 57 46 55 4f 25 32 46 32 30 32 33 30 31 32 38 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 33 30 31 32 38 54 31 31 34 36 31 34 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 39 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 26 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 3d 49 4e 54 45 4c 4c 49 47 45 4e 54 5f 54 49 45 52 49 4e 47 26 78 2d 69 64 3d 50 75 74 4f 62 6a 65 63 74 0a 68 6f 73 74 3a 73 33 2e 75 73 2d 65 61 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
  <RequestId>G81B83SMG39M9736</RequestId>
  <HostId>mIp/18+bzGVnGr3LupEn9Iixd3Zegf/mtmmXD/DbSItokeQt9XNft2+AH7fqqWDFVQGr16/iJw0=</HostId>
</Error>

I figured I may have met some hidden data storage limit, but after deleting one of my CKPT files I kept getting the same error, even when trying to push very small files.

I’m getting this for any of my models I try this with, whether hidden or public.

I’m getting the same issue both when I push from my local machine or when I push from a hosted environment like Google Colab.

Any idea what might be causing this or how I can fix this? I was about to officially release at least one new Stable Diffusion model in the near future. This totally blocks me both from releasing these models as well as from creating any new ones.

2 Likes
2

I think it has to do with the backend of HF.

Can’t even upload through the browser

image
image3518×1280 329 KB

3

The issue definitely doesn’t seem to be specific for my account or my IP. I tried creating a new account with a new email address and move a model to my new account from a friend’s home and I got the exact same error.

4

I just noticed the issue has been logged on Github :

5

Maybe the token you’re using to authenticate is a Read only token? Can you try login-cli again?

6

I’ve been using the same token for weeks. It worked flawlessly until today.

Either way, several people seem to experience the same issue. See the Github link I posted in the previous comment!

7

Thanks for reporting it. I can reproduce the issue as well. I’ve already notified the infra team. I’ll ping you back.

8

ok @johnslegers @jed351 the error is fixed now. Sorry for the inconvenience.

9

I noticed.

I can confirm that I no longer have this issue.

Thanks!

1 Like