Oh… It looks like you may have run into an unknown Hugging Face-side bug here. I couldn’t find a practical workaround at the moment, so I think the best next step is to report it to Hugging Face. Details:
just in case, @hysts
My current reading
The failure appears to happen in the Dev Mode image layer generated by Hugging Face, before your application or its GPU workload starts.
The relevant generated step is trying to install both:
ms-python.python
ms-python.debugpy
ms-python.python installs successfully, but resolving ms-python.debugpy fails, and that one failure terminates the entire Dev Mode image build.
That distinction matters because the Spaces Dev Mode documentation says that the Dev Mode image starts all of the following together:
- your application;
- the SSH server;
- the VS Code server.
I could not find a documented option to:
- enable SSH without the VS Code server;
- skip the Python/debugpy extension installation;
- pin a specific debugpy extension version;
- treat an extension-installation failure as non-fatal.
So, unfortunately, even though you only need SSH, the failed editor-extension setup appears to prevent the SSH-enabled Dev Mode container from being created at all.
I would not treat the T4 as the main suspect yet. The build is failing while preparing OpenVSCode extensions, before CUDA initialization or your application’s performance-sensitive code should be running.
More importantly, the remaining useful questions concern information that only Hugging Face can inspect internally: the exact OpenVSCode build they injected, the registry endpoint and response seen by the builder, any proxy/cache layer, and whether this is isolated to a particular builder pool or environment.
Recommended next route
I would stop looking for increasingly indirect workarounds for now and turn this into a small, actionable Dev Mode reproduction for Hugging Face.
The default path I would use is:
- Confirm whether the same commit builds normally with Dev Mode disabled.
- Try Dev Mode once on Hugging Face’s official minimal Python Dev Mode example, duplicated into your own namespace.
- Save the complete generated Dev Mode build step, not only the final error line.
- Add the result to this thread.
- Cross-link the report in Hugging Face’s official pinned Spaces Dev Mode feedback discussion.
You do not need to test every hardware type or every possible configuration. Even one clean comparison would make the report substantially more useful.
A minimal result matrix would be:
| Result |
What it would suggest |
| Normal build succeeds; Dev Mode fails |
The failure is probably in the additional Dev Mode layer rather than the application build itself |
Official minimal example fails at the same ms-python.debugpy step |
Strong evidence for an HF-side Dev Mode builder, registry-integration, or account/builder-path issue |
| Official minimal example succeeds; the original Space fails |
Compare the Space SDK, base image, Docker layout, architecture, and generated OpenVSCode version |
| CPU minimal example also fails |
The T4 is probably irrelevant |
| CPU succeeds but T4 fails |
Useful evidence for a hardware-specific builder path or builder-pool difference |
| A later retry succeeds without repository changes |
A transient registry, cache, proxy, or builder issue becomes more plausible |
Information that would make the report actionable
The most useful bundle would be:
- the original Space ID or URL;
- the minimal reproduction Space ID, if you create one;
- the commit SHA;
- SDK type, such as Docker or Gradio;
- hardware type;
- whether the same commit works with Dev Mode disabled;
- the approximate failure time, preferably in UTC;
- confirmation that Factory Rebuild produced the same result;
- the full generated OpenVSCode installation step;
- the exact OpenVSCode Server version and platform, if shown in the log.
In particular, look slightly earlier in the build log for something resembling:
openvscode-server-vX.Y.Z-linux-x64.tar.gz
The exact version and target platform may be much more useful to HF than the final Extension not found line alone.
You can retrieve build logs from the CLI with the officially documented command:
hf spaces logs <namespace>/<space> --build
Before posting logs publicly, remove access tokens, temporary signed URLs, private repository details, or other secrets if any appear.
Why this looks like a generated Dev Mode-layer failure
The important ordering in the log is approximately:
Copy OpenVSCode Server into the image
Run OpenVSCode Server with extension-install arguments
Install ms-python.python successfully
Fail to resolve ms-python.debugpy
Exit with status 1
Abort the Dev Mode image build
This is different from a failure in:
- your Python dependencies;
- your application startup;
- model loading;
- CUDA initialization;
- inference;
- the SSH client on your local machine.
The Space does not appear to reach the point where those later components can be tested.
The fact that ms-python.python installs successfully also makes a complete DNS or total registry outage less obvious, although it does not rule out:
- a different API endpoint being used for debugpy;
- per-extension cache state;
- platform filtering;
- version-selection behavior;
- a partial registry outage;
- a proxy or mirror inconsistency.
The safest description is therefore:
The Dev Mode builder can reach enough of the extension service to install the Python extension, but it cannot resolve the requested debugpy extension in that build environment.
That is an observation. The root cause remains unknown.
The part that only Hugging Face can realistically verify
At this point, several of the most important checks are internal to HF’s infrastructure.
Useful questions for the Spaces/Dev Mode team would be:
-
Which exact OpenVSCode Server version was injected into this build?
-
Which platform target did the builder use?
For example, linux-x64, linux-arm64, or another target.
-
Does the generated build contact Open VSX directly, or does it use an HF proxy, cache, or mirror?
-
What raw registry response did the builder receive for ms-python.debugpy at the failure time?
-
Did version selection, compatibility filtering, or platform filtering remove all candidate versions?
-
Was the issue limited to a particular builder pool, region, account, SDK, or hardware path?
-
Is this already a known Dev Mode incident?
-
Can debugpy installation be retried or pinned to a known-good version?
-
Can optional editor-extension failures be made non-fatal?
-
Can Dev Mode provide an SSH-only path when the user does not need the browser VS Code server?
Those questions are probably more useful now than trying to infer the exact root cause from outside HF’s build infrastructure.
Why the failure handling may matter as much as the registry error
There may be two separate issues here:
- Why
ms-python.debugpy could not be resolved.
- Why failure to install it aborts the entire SSH-capable Dev Mode image.
Microsoft’s Python extension documentation describes Python Debugger, Pylance, and Python Environments as optional dependencies. It explicitly says that the Python extension remains functional if those optional extensions fail to install.
That does not tell us what caused this HF build failure, but it does suggest a useful resilience question:
Should failure to install an optional editor feature prevent the SSH server and the rest of Dev Mode from starting?
Possible platform-side mitigations could include:
- retrying the extension query;
- pinning a known-good debugpy version;
- installing Python and debugpy in separate build steps;
- treating debugpy installation as best-effort;
- allowing users to disable editor extensions;
- providing an SSH-only Dev Mode option.
Any one of those could prevent a temporary editor-extension problem from blocking users who only need shell access.
Public sanity checks, and what they do not prove
The ms-python.debugpy extension currently exists on Open VSX, so this does not look like a simple typo in the extension identifier.
That weakens explanations such as:
- the publisher/extension ID never existed;
- the extension was entered without its publisher;
- there is categorically no debugpy package in Open VSX.
However, the extension being visible now does not prove that it was resolvable:
- at the exact build time;
- from HF’s builder network;
- through any HF-side cache or proxy;
- for the builder’s selected platform;
- through the particular OpenVSCode client version;
- after compatibility and version filtering.
There are also precedents showing that Extension not found can be a client/registry-resolution symptom rather than literal nonexistence.
For example, the code-server 4.104.0 changelog includes a fix for:
“extension not found” errors from Open VSX when trying to install the latest version of an extension.
This is only a clue, not a diagnosis:
- code-server and OpenVSCode Server are different projects;
- their patches and registry clients may differ;
- we do not yet know which OpenVSCode version HF injected;
- there is no evidence yet that this is the same bug.
It does, however, make the exact injected OpenVSCode version and its extension-resolution behavior worth checking.
Open VSX has also had unrelated cases where already-published extension versions were temporarily inactive or invisible and the API returned Extension not found, such as this Open VSX issue. Again, that issue concerns a different extension and does not establish the cause here. It only illustrates why the error text alone is not enough to conclude that the extension ID is invalid.
A small reproduction that can be handed to Hugging Face
The cleanest control appears to be Hugging Face’s own minimal Python Dev Mode example, which is linked from the official Dev Mode documentation.
A practical minimal test would be:
- Duplicate that example into your namespace.
- Keep it private if preferred.
- Use CPU Basic initially; the extension-install step should not require a GPU.
- Enable Dev Mode.
- Capture the build log.
The relevant commands after duplication would be:
hf spaces dev-mode <namespace>/<minimal-space>
hf spaces logs <namespace>/<minimal-space> --build
Interpretation:
If the minimal Space fails in exactly the same place
Add both Space IDs and the matching log fragments to the report. This would remove most application-specific variables and give HF a compact reproduction.
If the minimal Space succeeds
Compare the generated portions of both logs:
- OpenVSCode Server version;
- target platform;
- SDK;
- base image;
- user and UID;
/app ownership;- required Dev Mode packages;
- any difference in the generated extension-install command.
For Docker Spaces, the official documentation lists requirements including bash, curl, wget, procps, an /app directory owned by UID 1000, and a CMD instruction. Those are worth checking if the minimal control succeeds. However, the current log already gets far enough to launch OpenVSCode and install one extension, so a basic missing-package problem does not appear to be the best first explanation.
If you only run one test
The official minimal example is probably the highest-value one.
There is no need to spend paid T4 time testing many combinations unless the CPU result indicates a hardware-path difference.
Why the obvious workarounds do not appear to bypass this failure
Factory Rebuild
You already tried it and reproduced the same failure. Repeating it many more times is unlikely to add much unless the failure later disappears without repository changes, in which case the timestamps become useful evidence for a transient service issue.
hf spaces ssh --auto
The HF CLI documentation describes this as a way to enable Dev Mode automatically and then connect to the Dev Mode container.
It does not bypass the Dev Mode image build.
The sequence is still effectively:
Enable Dev Mode
Build/start the Dev Mode container
Expose its SSH endpoint
Connect
Your failure is occurring before the container reaches the SSH-connectable stage.
Manual VSIX installation
Manual installation would normally require access to the running container or control over the generated build command. Here, the build fails before SSH is available, and I could not find a documented user setting for replacing the generated extension command.
Switching away from the T4
This is useful as a control, not as an established fix.
If a CPU minimal example fails identically, there is little reason to suspect the GPU. If only the T4 path fails, that gives HF a useful builder-pool comparison.
Disabling Dev Mode
This should avoid the generated Dev Mode layer, but it also removes the SSH access you wanted. It is therefore a diagnostic comparison rather than a complete workaround.
Why reporting this as a possible infrastructure issue is reasonable
There is at least one previous, non-identical example where a blank/minimal Space failed inside an HF-generated Dev Mode step.
In this earlier HF Forum thread, the user reproduced the failure after:
- a Factory Reset;
- deleting and recreating the Space;
- changing between CPU and ZeroGPU;
- using an otherwise empty Space.
An HF contributor described that case as an infrastructure issue, escalated it to the infrastructure team, and later reported that it had been resolved.
That does not mean the current issue has the same root cause. The earlier error involved a different generated operation.
What it demonstrates is narrower:
- HF’s generated Dev Mode layer can fail independently of user application code;
- a blank or official minimal Space can be valuable evidence;
- changing hardware does not necessarily resolve generated-layer failures;
- some failures can only be investigated or corrected by HF internally.
That is why a minimal reproduction plus complete generated-build logs is probably the most productive next step here.
Where I would report it
I would keep the main reproduction details in this existing Forum thread so the context and original log remain together.
I would then add a short cross-link in Hugging Face’s official pinned Spaces Dev Mode feedback discussion, containing only:
- a link to this Forum thread;
- the original Space ID;
- the minimal reproduction Space ID, if available;
- the UTC failure time;
- the matching
ms-python.debugpy error;
- the injected OpenVSCode version, if visible.
That discussion is explicitly dedicated to Dev Mode feedback, and HF contributors have previously used it to request specific SSH/VS Code reproduction details and confirm platform-side fixes.
I would not open an upstream OpenVSCode issue first. Before doing that, HF should ideally confirm:
- the exact OpenVSCode version;
- the endpoint it queried;
- whether a proxy/cache was involved;
- the raw response;
- whether the failure reproduces outside the HF builder.
Without those details, an upstream report would contain too much speculation.
Current public service status
At the time of writing, the Hugging Face status page reports all services online.
That only means there is no currently published broad incident. It does not rule out a narrower problem involving:
- Dev Mode image generation;
- a particular builder pool;
- a registry proxy or cache;
- one extension;
- one account path;
- one region;
- a short-lived incident that has not been posted to the status page.
So the status page does not materially change the recommendation to provide HF with a minimal reproduction.
Bottom line
I do not see a supported way to recover SSH from this failed Dev Mode build while simply ignoring the VS Code server.
The strongest next step is therefore:
- one normal-build-versus-Dev-Mode comparison;
- one official minimal Dev Mode reproduction, if practical;
- complete generated-build logs;
- then hand the remaining questions to HF.
If the official minimal Space fails at the same extension step, provide HF with both Space IDs and the matching logs.
If the minimal Space succeeds, compare the generated OpenVSCode version, platform target, and Dev Mode prerequisites against the original Space.
Either result is more actionable than trying to guess externally whether the hidden cause is OpenVSCode version selection, Open VSX visibility, an HF proxy/cache, or a particular builder pool.