My Space suddenly went offline. The CPU cannot restart

It was running normally before, then suddenly disappeared, showing the Huggingface icon and a message saying “Building Space.”

I checked the backend logs, and before the logs stopped, there were several instances of “reloading database.” I tried restarting the Space, but it didn’t work. I tried rebuilding the Space, but it also didn’t work. Then I noticed my CPU is stuck in a spinning state. What should I do now?

1 Like

The cause is unknown and cannot be resolved by the user at this time.

The dirty but quickest workaround is as follows.

  • Rename the current space to something appropriate and set it to Private (for safekeeping in case the issue is resolved in the future).
  • Create a new space with an available name.
  • Upload the same source code.

What a tragedy. From the posts you shared, I see many people are in the same situation. No idea how long it will take to recover. I even saw some people stuck on this issue for weeks…

1 Like

Exactly. Even a Hugging Face staff member who was maintaining Spaces couldn’t solve the problem on his own… :nauseated_face:

It probably requires quite high-level permissions…

Me too. Python Gradio space. Was working fine yesterday. Committed modified app.py that works perfectly on my home PC in VS2022 . Even after factory rebuild, just sitting on “Building” while logs just look normal. Pushed and started.

EDIT: After about 1.5 hours this additional error message appeared int Build log following the normal messages that looked like everything was OK:

ERROR: failed to push spaces-registry.huggingface.tech/spaces/6801b2253a3d2135e30da61a:cpu-08475b3-7x848txl: unexpected status from HEAD request to https://spaces-registry.huggingface.tech/v2/spaces/6801b2253a3d2135e30da61a/manifests/cpu-08475b3-7x848txl: 401 Unauthorized

3 Likes

It must be an error for so many to suddenly appear at the same time… @meganariley @pierric @hysts

I finally created a new space, same configuration and same files as the space that was stuck building. It built and ran just fine. Deleted the stuck space.

1 Like

Thanks for reporting! I shared this internally.

5 Likes

Thank you, hysts!

1 Like

I’m having the same issue. Stuck in building until I get a build error that says unexpected status from HEAD request

1 Like

Same here, stuck at “Building” until…:

--> ERROR: failed to push spaces-registry.huggingface.tech/spaces/66a915c181dd5b0fe315302a:cpu-0ada85f-8cwhnd27: unexpected status from HEAD request to https://spaces-registry.huggingface.tech/v2/spaces/66a915c181dd5b0fe315302a/manifests/cpu-0ada85f-8cwhnd27: 401 Unauthorized
4 Likes

+1, I’m running into the same issue.

1 Like

Possibly related:

Suggests a problem with docker vs. AWS perms:

“Today I stumbled upon the same issue. The docker buildx build … --push command failed with the same error message (unexpected status from HEAD request to : 403 Forbidden). But docker push was working uninterrupted. It turns out that buildix required one additional AWS ECR permission - ecr:BatchGetImage. :upside_down_face:

I know nothing about this stuff, but hope that clue might help those who do :slight_smile:

2 Likes

Same issue. Over the past 3-4 days, 2 of my spaces went offline due to “Build error”. They were working fine for the last 1 year.

2 Likes

Same issue here. Any update is appreciated!

1 Like

Wow, you’re really having a rough time. Hope they can fix this error. I haven’t been using Huggingface for long, so I don’t have much data, and I had to rebuild after careful selection.

2 Likes

Was just having the same issue. What ended up working for me is to rebuild the image using a different Space Hardware. Then rebuild it back to the original hardware.

3 Likes

I confirm that this also worked for me. What a relief.

1 Like

They have fixed the issue, it seems. All my gradio spaces are back. Great news.

2 Likes

The infra team has resolved the issue. We are still investigating the root cause, but restarting the Space should fix it.

2 Likes