For now, I’m not very familiar with Azure, so there’s a good chance I’m missing something, but from what I found:
Short version
I have not reproduced this exact deployment myself, but the error text you pasted looks very close to the one shown in the official Hugging Face guide for Deploy Meta SAM 3 on Microsoft Foundry.
I would first debug this as an Azure ML / Foundry secret-injection and endpoint-identity issue, not as a SAM3 runtime issue.
The likely failing layer is roughly:
Hugging Face token
-> Azure ML / Foundry workspace connection
-> endpoint managed identity
-> secret injection into deployment environment
So I would branch the debugging like this:
| If this is true |
I would check next |
| The endpoint uses system-assigned identity, SAI |
Was it created with enforce_access_to_default_secret_stores=enabled, and did the endpoint SAI actually receive Azure Machine Learning Workspace Connection Secrets Reader? |
| The endpoint uses user-assigned identity, UAI |
Was that UAI manually granted Azure Machine Learning Workspace Connection Secrets Reader on the workspace/project scope? |
| The deployment was started from the Foundry / Azure ML UI |
Try the programmatic path from the official SAM3 docs, because the docs call this out for gated models |
| The connection exists but deployment still fails |
Check whether the connection is in the exact same workspace / Foundry hub-based project used by the endpoint |
| The Azure secret-injection layer passes |
Then check HF gated approval, token scope, model download logs, SKU/quota, and runtime logs |
The high-signal checks I would do first:
1. Is the endpoint SAI or UAI?
2. Is the connection named exactly HuggingFaceTokenConnection?
3. Is the key named exactly HF_TOKEN?
4. Is the connection in the same workspace / Foundry hub-based project as the endpoint?
5. If SAI: was the endpoint created with enforce_access_to_default_secret_stores=enabled?
6. If SAI: did the creating user/service principal have permission to read workspace connection secrets?
7. If UAI: was the UAI manually granted Azure Machine Learning Workspace Connection Secrets Reader?
8. Is the HF account behind the token approved for facebook/sam3?
Please do not paste the Hugging Face token or any populated connection secret into the thread. Sanitized endpoint/deployment snippets should be enough for others to help narrow this down.
Why I would look at Azure secret injection before SAM3 runtime
The important part of the error is not just that the deployment failed, but that it failed with SecretsInjectionError and says:
Failed to retrieve and inject workspace connection secrets
That points to the layer where Azure ML / Foundry is trying to retrieve a secret from a workspace connection and inject it into the deployment environment.
The SAM3 guide itself mentions this kind of failure in the gated-model path: Deploy Meta SAM 3 on Microsoft Foundry.
The Microsoft docs describe secret injection as the process where an online deployment retrieves secrets from secret stores and exposes them to the user container as environment variables: What is secret injection in online endpoints?
So I would separate the layers like this:
1. Model availability
- facebook-sam3 exists in the Azure / Foundry model catalog
2. Hugging Face gated access
- the HF account behind the token must be approved for facebook/sam3
3. Azure workspace connection
- connection name: HuggingFaceTokenConnection
- key name: HF_TOKEN
- value: HF read or fine-grained token, stored as secret
4. Endpoint identity and RBAC
- SAI or UAI
- Workspace Connection Secrets Reader role
- correct workspace/project scope
5. Secret injection into deployment
- Azure retrieves the connection secret and injects it into the deployment environment
6. Later layers
- model download
- SKU/quota
- container startup
- SAM3 runtime
- request payload
For this specific error, I would not start with layers 5-6 until layers 2-4 are confirmed.
That does not mean SAM3 runtime, Transformers, CUDA, quota, or request format cannot fail later. It only means this particular error message looks earlier than that.
SAI vs UAI: the part I would check most carefully
The most important branch seems to be whether the endpoint uses system-assigned identity or user-assigned identity.
Microsoft’s secret-injection docs explain this distinction here: Access secrets from online deployment using secret injection.
If the endpoint uses system-assigned identity, SAI
For SAI, the docs show this endpoint-level property:
properties:
enforce_access_to_default_secret_stores: enabled
If the required conditions are met, Azure can automatically assign the endpoint SAI permission to read workspace connection secrets.
The conditions I would check are:
- endpoint identity is SAI
- endpoint was created with enforce_access_to_default_secret_stores=enabled
- the user/service principal creating the endpoint had permission to read workspace connection secrets
- the endpoint SAI actually received Azure Machine Learning Workspace Connection Secrets Reader
The confusing part is that the creating user/service principal and the endpoint managed identity are not the same thing.
So even if your user has some permissions, I would still check whether the endpoint identity itself got the required role.
If the endpoint uses user-assigned identity, UAI
For UAI, I would not rely on the SAI auto-assignment path.
The Microsoft docs say that if the endpoint uses UAI, the UAI must be manually assigned the needed role.
So if this is UAI, I would check:
- what UAI is attached to the endpoint
- whether that UAI has Azure Machine Learning Workspace Connection Secrets Reader
- whether the role is assigned at the correct workspace/project scope
I would avoid simplifying this to “just add Contributor” or “just add Owner”. The error is pointing at a specific secret-reader path. The useful thing to verify is the exact identity, role, and scope.
Workspace connection checks
The SAM3 docs and the Azure model card both point to the same expected connection shape.
The connection should be:
name: HuggingFaceTokenConnection
key: HF_TOKEN
value: Hugging Face read or fine-grained token
Relevant links:
The details I would verify:
- connection exists in the same workspace / Foundry hub-based project used by the endpoint
- connection name is exactly HuggingFaceTokenConnection
- key name is exactly HF_TOKEN
- token is stored as a secret
- token belongs to the HF account approved for facebook/sam3
One small thing that may be easy to miss: the SAM3 docs mention that, for Microsoft Foundry, the workspace_name parameter should be the Azure AI Foundry hub-based project name. So I would double-check that the connection, endpoint, and deployment are all targeting the same workspace/project context.
If the connection was created in one workspace/project and the endpoint is being created in another, the setup can look correct at a glance while the endpoint identity still cannot read the secret used by the deployment.
Small local checks that may help narrow it down
These are examples of the kind of checks I would run locally. I would not paste any secret values into the forum.
Check endpoint identity:
az ml online-endpoint show \
--name "$ENDPOINT_NAME" \
--resource-group "$RESOURCE_GROUP" \
--workspace-name "$WORKSPACE_NAME" \
--query "identity"
Check endpoint properties:
az ml online-endpoint show \
--name "$ENDPOINT_NAME" \
--resource-group "$RESOURCE_GROUP" \
--workspace-name "$WORKSPACE_NAME" \
--query "properties"
For SAI, I would look for the default secret-store access setting in the endpoint properties. I would avoid relying too much on one exact JSON path in a forum reply, because CLI output shape can vary by version. The main question is whether the endpoint was created with secret-store access enabled.
Check whether the connection exists in the same workspace/project:
az ml connection show \
--name HuggingFaceTokenConnection \
--resource-group "$RESOURCE_GROUP" \
--workspace-name "$WORKSPACE_NAME"
If you already know the endpoint principal ID, check role assignments at the workspace scope:
WORKSPACE_SCOPE="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE_NAME"
az role assignment list \
--assignee "$ENDPOINT_PRINCIPAL_ID" \
--scope "$WORKSPACE_SCOPE" \
--include-inherited \
--output table
The role I would look for is:
Azure Machine Learning Workspace Connection Secrets Reader
If a custom role is used instead, the error message itself mentions these actions:
Microsoft.MachineLearningServices/workspaces/connections/listsecrets/action
Microsoft.MachineLearningServices/workspaces/metadata/secrets/read
The Azure permission reference describes Microsoft.MachineLearningServices/workspaces/connections/listsecrets/action in the AI / machine learning permission set: Azure AI + machine learning permissions.
Again, I would not paste populated secrets or token values.
HF gated access is still worth checking, but I would treat it as a separate layer
facebook/sam3 is gated, so the HF account behind the token must be approved for access.
I would verify:
- the token belongs to the HF account that requested access
- the request for facebook/sam3 is approved, not pending/rejected
- the token is read or fine-grained and has access to the model
- the token is stored under the HF_TOKEN key in the Azure connection
But I would treat this as the next layer after the Azure secret-injection path.
If the endpoint identity cannot retrieve and inject the workspace connection secret at all, the deployment may fail before the container even reaches the stage where it tries to download SAM3 weights from the Hub.
So my ordering would be:
1. Azure workspace connection / secret injection / endpoint identity
2. HF gated approval and token scope
3. model download logs
4. quota / SKU availability
5. SAM3 runtime or request payload
Relevant links:
If you want to share more information safely
The most useful sanitized details would be:
- Did you deploy from UI, Python SDK, or CLI?
- Is this a new endpoint or an existing endpoint?
- Is the endpoint identity SAI or UAI?
- Sanitized endpoint creation snippet/YAML
- Sanitized deployment snippet/YAML
- Whether endpoint properties include default secret-store access, if SAI
- Whether the connection is named HuggingFaceTokenConnection
- Whether the key is named HF_TOKEN
- Whether the connection is in the same workspace / Foundry hub-based project
- Whether the endpoint identity has Azure Machine Learning Workspace Connection Secrets Reader
- Whether the HF account behind the token is approved for facebook/sam3
- Azure ML CLI / azure-ai-ml SDK version, if relevant
Please do not share:
- the HF token
- populated connection secrets
- full subscription IDs if you do not want them public
- tenant-specific private identifiers unless sanitized
A good sanitized shape would be something like:
Deployment path: Python SDK / CLI / UI
Endpoint identity: SAI or UAI
Connection name: HuggingFaceTokenConnection
Key name: HF_TOKEN
Same workspace/project: yes/no
Endpoint secret-store access flag, if SAI: enabled/disabled/unknown
Endpoint identity has Workspace Connection Secrets Reader: yes/no/unknown
HF access to facebook/sam3 approved: yes/no/pending/unknown
My current guess
Without seeing your endpoint/deployment config, my best guess would be:
Most likely:
endpoint identity cannot read the HuggingFaceTokenConnection/HF_TOKEN secret
Possible reasons:
- deployment was created through a UI path instead of the programmatic SAM3 path
- SAI endpoint was not created with enforce_access_to_default_secret_stores=enabled
- SAI auto role assignment did not happen because the creator identity lacked permission
- UAI endpoint is missing manual Workspace Connection Secrets Reader assignment
- connection exists but in a different workspace / Foundry project
- connection name or key name differs from HuggingFaceTokenConnection / HF_TOKEN
- role assignment was added but not at the right scope or not yet propagated
Worth checking, but probably a later layer for this exact error:
- HF token owner is not approved for facebook/sam3
- token scope is wrong
- model download fails after secret injection
- quota / SKU issue
- SAM3 runtime issue
- request payload issue
So I would first make the Azure secret-injection path observable, especially SAI vs UAI and the endpoint identity’s access to the workspace connection secret. After that is confirmed, the next layer would be HF gated access and model download/runtime logs.