Was there a HF update that affects spaces container networking?

Spaces
1

Have there been any changes in the Spaces infrastructure (e.g. container networking) in the last day or so?

I had a stock docker space running jupyter notebook that worked fine yesterday. Today it doesn’t: the web app itself works - I can login, open existing notebooks, etc - but it can’t connect to the kernel. Terminal sessions don’t work either.

This doesn’t seem to be limited to me - the jupyter-lab space here also fails to run a notebook or terminal once you login.

The only logs I can access without getting into the container are from HF’s container logs:

...
[I 2023-06-07 10:58:32.025 LabApp] Build is up to date
[I 2023-06-07 10:58:45.444 ServerApp] Creating new notebook in 
[I 2023-06-07 10:58:45.493 ServerApp] Writing notebook-signing key to /home/admin/.local/share/jupyter/notebook_secret
[I 2023-06-07 10:58:46.333 ServerApp] Kernel started: 3771ed7c-815b-4f72-ab69-517418d244be
[W 2023-06-07 10:58:46.800 ServerApp] Blocking Cross Origin WebSocket Attempt.  Origin: https://xxxxxxxxxx-jupyter.hf.space, Host: 10.19.179.226:7860
[W 2023-06-07 10:58:46.800 ServerApp] 403 GET /api/kernels/3771ed7c-815b-4f72-ab69-517418d244be/channels?session_id=74366e13-3bd4-4835-986f-ea2475e1c542 (5b91f8020966424798c59b4c87658dc1@10.16.32.186) 9.79ms referer=None
...

My gut feeling based on that is that something changed in the container networking or addressing, but this is just an educated guess.

2

Hi @ebudan,

Indeed, we have made some internal infrastructure changes over the past day or so. A fix for the websocket issue was deployed approximately 8 hours ago. Could you please try perform a factory reboot on your Space? For instance, the jupyter-lab space is functioning again.

3

Confirmed, my space works now.

I would be curious to know what the culprit was, in case I run into something similar again on a cluster of my own. Is there a changelog you can link to?