Personally, this (and Space-to-Space requests to *.hf.space return 503 from awselb (since Jul 8)) looks more like a regression to me. Details below:
just in case, @hysts
Direct answer: the general architecture does not appear to be inherently unsupported.
Hugging Face documents Docker Spaces as suitable for FastAPI and other API endpoints, documents authenticated access to private Space endpoints, and also documents calling one Space from another Space. I could not find one official reference that combines this exact topology—private Docker caller, private Docker target, arbitrary FastAPI route, and server-side Bearer authentication—but the individual pieces are documented or have prior working examples.
For a private Docker Space, the route I would expect to use is still:
caller Space
→ HTTPS on port 443
→ https://<target-subdomain>.hf.space/<route>
→ Authorization: Bearer <token that can access the target Space>
→ target application
The Docker Spaces documentation describes exposing an application through app_port, and the Spaces networking documentation allows outbound HTTP/HTTPS traffic on ports 80 and 443. The official private Tabby Docker Space example also uses the direct *.hf.space URL as an API endpoint and adds an authorization header when the Space is private.
The strongest reason I would currently suspect a regression is the independent report in Space-to-Space requests to *.hf.space return 503 from awselb. That report describes:
- the same target URLs returning
200 outside Hugging Face but 503 from inside a Space;
- failures against both private Spaces and public Gradio Spaces;
- the same result with and without a valid token;
- reproduction from multiple caller Spaces and multiple egress IPs;
- identical DNS resolution inside and outside HF.
If those controls are accurate, this is difficult to explain purely as a Dockerfile, FastAPI route, token-scope, private-visibility, DNS, or single-Space deployment problem.
For now, I would pause large application-side changes. Rebuilding all five Spaces again, changing the model-loading code, or increasing retries is unlikely to isolate a source-dependent platform issue. A small inside-vs-outside comparison with exact timestamps is likely to be more useful.
Why this does not look like an unsupported architecture
There are several separate pieces of evidence here. None is a perfect one-to-one match for the full private-Docker-to-private-Docker topology, but together they show that API access and Space-to-Space use are established patterns.
1. Docker Spaces are explicitly intended to expose APIs
The official Docker Spaces documentation says that Docker Spaces can be used for “FastAPI and Go endpoints,” among other applications.
A Docker Space exposes one external application port through the README metadata:
---
sdk: docker
app_port: 7860
---
The container may use additional ports internally, but clients outside that container normally reach the application through the configured app_port and the Space’s *.hf.space endpoint.
Your reported configuration—Uvicorn listening on 0.0.0.0:7860—matches that documented pattern.
2. A private Docker Space can serve as an authenticated API endpoint
The official Tabby on Spaces guide describes a private Docker Space whose direct URL serves as an API endpoint:
https://<owner>-<space-name>.hf.space
It then requires an authorization header because the Space is private.
There is also a 2025 forum example involving a private Docker/FastAPI endpoint. The suggested request was essentially:
import os
import requests
response = requests.post(
"https://<owner>-<space-name>.hf.space/api/predict",
headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"},
json={"text": "hello"},
timeout=60,
)
The original poster later confirmed that it worked after correcting the token issue.
That example was an external caller rather than another Space, so it does not prove that the current Space-originated path is healthy. It does support the direct URL and Bearer-authentication part of the design.
3. Space-to-Space access is a documented and intentionally implemented pattern
The current Spaces as API endpoints documentation includes a section titled “Calling Spaces from Another Space.”
Its explicit example is a Gradio application calling a ZeroGPU Space, so it should not be stretched into a complete guarantee for every possible Docker-to-Docker REST topology. Still, it establishes that server-side Space-to-Space communication through HF Space endpoints is not generally an alien or forbidden pattern.
Historically, Gradio also merged support for loading private Spaces using an HF token in November 2022. That change explicitly added support for private Spaces and updated Space URLs to the *.hf.space format.
Later issues such as a public Space loading a private Space with hf_token concern downstream state-management behavior, not whether cross-Space access existed at all. Those reports are useful evidence that authenticated Space-to-Space configurations were being used in practice.
4. Official HF course code sends HTTP requests from a Space to another *.hf.space service
The official Agents Course final-assignment template uses ordinary requests.get() and requests.post() calls against another *.hf.space service.
That is a public course API rather than a private Docker worker, but it is a concrete official example of a Space-hosted application making server-side HTTP requests to another Space endpoint.
5. The remaining limitation in the public evidence
I could not find a single official example combining all of the following:
- a private Docker Space as the caller;
- another private Docker Space as the target;
- an arbitrary FastAPI REST endpoint;
- server-side Bearer authentication;
- documented long-running production use.
So I would not claim that this exact topology has a dedicated formal support guarantee.
However, each relevant part—Docker API serving, direct *.hf.space endpoints, private authentication, outbound HTTPS, and Space-to-Space calls—is documented, implemented, or represented by previous examples. That makes “this architecture was never supposed to work” a less convincing explanation than a current regression or an undocumented policy change.
Why the current evidence points more toward a regression
The important part is not any single response header or log line. It is the comparison between callers and targets.
Evidence from this report
You have already reported that:
- the same Docker images and code work locally;
- every Space builds and appears as Running;
- the workers start Uvicorn on
0.0.0.0:7860;
- the models finish loading;
- the worker health endpoints are accessible;
- the backend receives repeated
503 Service Temporarily Unavailable responses;
- recreating and redeploying the Spaces did not resolve the issue.
A Running state does not by itself prove that every ingress path is working, and “health endpoint is accessible” can mean different things depending on where it was tested from. Still, these observations make a simple container-startup failure less likely.
Evidence from the independent report
The related Space-to-Space 503 report provides stronger controls:
| Test |
Reported result |
What it reduces |
External machine → same .hf.space URL |
200 |
A generally dead target |
| Space container → same URL |
503 |
Points to a caller-dependent path |
| Space → public Gradio Space |
503 |
Private visibility alone |
| Request with valid token |
503 |
Missing authentication alone |
| Request without token |
503 |
Token parsing alone |
| Multiple caller Spaces |
503 |
One broken caller deployment |
| Multiple egress IPs |
503 |
One blocked source IP |
| Same DNS result inside and outside |
Same address resolution |
Basic DNS divergence |
If that reproduction applies to your environment, it substantially reduces the likelihood of:
- one malformed worker URL;
- one incorrect Dockerfile;
- one broken FastAPI route;
- a problem limited to private visibility;
- a problem limited to a specific token;
- a problem limited to one caller Space;
- a problem limited to one caller egress IP;
- a simple DNS-resolution failure.
It does not reveal the internal root cause. It only moves the likely failing boundary away from one application and toward the HF-managed path used when a Space calls a *.hf.space endpoint.
About the missing worker log entry
The absence of the POST in the visible worker logs is useful supporting evidence, but it is not conclusive by itself.
HF Spaces Run Logs have previously had cases where an application was serving requests but stdout/stderr request logs were incomplete or absent. A stronger application-side control would be a unique request ID recorded by middleware, a counter, or another independently observable mechanism.
Therefore I would phrase the conclusion as:
The request does not appear in the available target logs.
rather than:
It has been proven that the request never reached any application process.
The outside-200 versus inside-503 comparison is a stronger signal than the absence of a line in the Run Logs.
About Server: awselb/2.0
The Server: awselb/2.0 response header shows that an AWS Application Load Balancer is present in the response path. It does not, by itself, identify which internal component generated the 503.
AWS documents that ALBs can generate 503 responses in some target-availability situations, but AWS also documents that the awselb/2.0 server header is normally added to responses passing through an ALB. HF’s load-balancer metrics, target state, routing rules, and internal proxy logs are not publicly visible.
So the header is useful context, but not proof of a specific ALB, target-group, proxy, or routing failure.
What only Hugging Face can confirm — and what may help them investigate
There is a practical boundary here: the public observations can narrow down the failing path, but only Hugging Face can inspect the managed infrastructure behind it.
What can be observed from outside HF
Users can compare:
- the same URL from an external machine and from a Space;
- public and private targets;
- authenticated and unauthenticated requests;
- GET and POST requests;
- DNS resolution;
- caller egress IPs;
- response status, body, headers, and timing;
- whether application-controlled instrumentation records the request.
These observations can show that a problem is source-dependent or occurs before an expected application response. They cannot identify the exact internal service responsible.
What only Hugging Face can confirm
Only HF can determine:
- whether server-side requests from one Space to another
*.hf.space endpoint are currently intended to work;
- whether this behavior is an intentional policy change or a regression;
- whether the issue is global or limited to particular clusters, accounts, namespaces, regions, or request paths;
- where the
503 is generated inside the managed Spaces ingress/proxy path;
- whether a routing, security, entitlement, or infrastructure change was deployed around July 8;
- whether the target application was selected or contacted internally;
- whether an internal incident exists that is not represented on the public status page.
At the time of writing, the public HF status page lists both Spaces and Spaces Proxy as operational. That is worth recording, but it does not rule out a partial, source-dependent, account-specific, or newly discovered regression.
I would avoid guessing further about HF’s internal listener rules, target groups, routing policies, or rollout configuration without HF telemetry.
A compact evidence package for HF
One minimal failing request is probably more useful than another full rebuild. Ideally, preserve:
- exact UTC timestamp;
- caller Space ID;
- target Space ID;
- target visibility: public or private;
- HTTP method and route;
- whether an Authorization header was included;
- status code;
- response body;
- complete response headers, after removing credentials;
- request duration;
- caller egress IP;
- caller DNS result;
- result from an external machine against the same URL;
- result from the caller Space against one public control Space;
- a unique application correlation ID, if available;
- whether that correlation ID appeared in target-side instrumentation.
Useful response headers, when present, may include:
DateServerViaX-Request-IdX-Amzn-Trace-IdX-Amz-Cf-Id- any HF-specific request or proxy identifier
Do not post the token itself. In particular, sanitize verbose curl output before pasting it into the forum.
A compact report could look like this:
UTC time:
Caller Space:
Target Space:
Target visibility:
Method/path:
Authorization supplied: yes/no
Caller egress IP:
DNS result:
External caller → target:
status:
body:
request duration:
Space caller → same target:
status:
body:
request duration:
Space caller → public control Space:
status:
body:
Relevant response headers:
Target-side correlation result:
The platform-level clarification needed
The central questions for HF are:
- Are server-side requests from one Space to another Space’s
*.hf.space endpoint still supported?
- If yes, is the current
503 behavior a known regression?
- If no, what is the supported replacement architecture for Space-to-Space APIs?
Those answers would be more useful than additional speculation about an internal component that users cannot inspect.
Minimal checks and how to interpret the results
You have already done substantial troubleshooting, so I would not treat all of the following as mandatory homework. This is a small menu of controls that can make the report easier to classify.
Minimal comparison matrix
| Caller |
Target |
Authentication |
Main purpose |
| External machine |
Same private worker |
Yes |
Check target URL and token |
| Main backend Space |
Same private worker |
Yes |
Reproduce the failing path |
| Main backend Space |
Public control Space |
No |
Remove private-authentication variables |
| Main backend Space |
Ordinary external HTTPS endpoint |
No |
Check general outbound HTTPS |
| External machine |
Public control Space |
No |
Verify the control target |
The most informative result is:
external machine → same target = 200
Space container → same target = 503
because it holds the target constant and changes the caller environment.
Start with the smallest route
Where possible, compare these separately:
GET /health- a small
POST /debug
- the real inference request
This separates basic routing from:
- large request bodies;
- JSON parsing;
- file uploads;
- model inference;
- long processing times;
- application-level timeouts.
Possible interpretations:
| Result |
More consistent with |
GET and POST both return the same immediate 503 |
Failure before route-specific application behavior |
| GET works but POST fails |
Method, route, body, content type, or request-duration difference |
Public target also returns 503 from the Space |
Not limited to private authentication |
| Public target works but private target fails |
Recheck private access and token behavior |
| Ordinary external HTTPS also fails |
Broader outbound-networking problem |
503 later changes to 401 or 403 |
Routing may be working again, exposing a separate auth issue |
| Correlation ID reaches the worker |
Focus again on FastAPI middleware, route handling, or downstream processing |
| Correlation ID does not reach independent target instrumentation |
Stronger evidence for failure before the application |
Observe redirects separately
For the first comparison, it can help not to follow redirects automatically:
curl -sS \
--max-redirs 0 \
--max-time 20 \
-D response-headers.txt \
-o response-body.txt \
-H "Authorization: Bearer ${HF_TOKEN}" \
"https://<target-subdomain>.hf.space/health"
printf '\nStatus and headers:\n'
cat response-headers.txt
printf '\nBody:\n'
cat response-body.txt
Then repeat with -L in a separate test.
This distinguishes:
- an immediate
503;
- a redirect to another host or path;
- a login/authentication redirect;
- a failure that occurs only after following the redirect.
For Python:
import os
import requests
url = "https://<target-subdomain>.hf.space/health"
headers = {"Authorization": f"Bearer {os.environ['HF_TOKEN']}"}
response = requests.get(
url,
headers=headers,
allow_redirects=False,
timeout=20,
)
print("status:", response.status_code)
print("headers:", dict(response.headers))
print("body:", response.text[:1000])
Optional correlation header
A caller-controlled identifier can make logs easier to match:
X-Debug-Request-ID: 2026-07-09T07:30:00Z-test-001
For example:
import os
import requests
request_id = "2026-07-09T07:30:00Z-test-001"
response = requests.get(
"https://<target-subdomain>.hf.space/health",
headers={
"Authorization": f"Bearer {os.environ['HF_TOKEN']}",
"X-Debug-Request-ID": request_id,
},
timeout=20,
)
print(response.status_code)
print(response.headers)
print(response.text)
The target can log or echo the same identifier in middleware.
This remains a supporting control: an intermediate proxy is not guaranteed to preserve every arbitrary header. It is most useful when the identifier does appear at the target, or when the target uses a separate observable counter.
Related cases and timing context
These links are useful for comparison, but I would not assume they share the same root cause.
Closest current report
The closest case is the independent Space-to-Space 503 report.
It is unusually useful because it includes controls involving:
- inside versus outside HF;
- public versus private targets;
- token versus no token;
- multiple caller Spaces;
- multiple egress IPs;
- DNS comparison.
That report is the strongest external support for treating the current behavior as a platform-path regression rather than a problem specific to this five-Space application.
Earlier Docker proxy-like symptom
In April 2026, another user reported Docker Spaces returning {"data":[]} while the applications appeared to start normally.
Similarities:
- Docker Spaces;
- Uvicorn startup completed;
{"data":[]} appeared;- the user did not see incoming requests in the container logs;
- the managed proxy/application boundary was suspected.
Important difference:
- that report said the Space viewer and external direct URL also failed;
- the current July report describes external access succeeding while Space-originated access fails.
Therefore it is a useful historical comparison, not evidence of the same bug.
Similar architecture, different failure types
Gradio issues involving public Spaces loading private Spaces—such as state separation when a public Space accesses a private Space—show that authenticated cross-Space configurations have been used.
However, Gradio loading, queue protocols, state handling, and file proxying are not necessarily the same path as a raw FastAPI REST request. A Gradio protocol or file-authentication issue should not be treated as the same cause as an immediate Docker-to-Docker 503.
Other reports around July 8
There were other Spaces-related reports in approximately the same time window:
The timing is worth recording, but these affect different platform layers:
| Report |
Main platform area |
Space stuck in APP_STARTING |
Scheduling / lifecycle |
| PRO requirement during creation |
Entitlement / admission / billing |
| Running Space cannot call another Space |
Runtime communication / ingress path |
Their proximity does not establish a common cause. Whether they correspond to one deployment or several unrelated changes is something only HF can verify.
Bottom line
- The general design—Docker Spaces exposing APIs, private endpoints using authentication, and one Space calling another—is documented or represented by prior examples.
- I could not find a dedicated official reference for this exact private-Docker-to-private-Docker REST topology, so that limitation should be kept explicit.
- The strongest current evidence is not the
awselb header or the missing Run Log entry. It is the independently reported difference between an external caller receiving 200 and a Space caller receiving 503 from the same target.
- The public-target and token controls make a private-authentication-only explanation less likely.
- The exact internal cause cannot be determined from outside Hugging Face.
- Preserving one small, timestamped reproduction is probably more useful than rebuilding the full application again.
The key question for HF is whether server-side requests from one Space to another *.hf.space endpoint are still intended to work. If they are, the current evidence appears to warrant platform-side investigation. If they are not, the supported replacement route needs to be documented.