Cannot produce correct output for images Llama Guard 3 11B vision

Intermediate
1

Transformers Output:

image
image1188Ă—809 109 KB

TGI Inference Template:

"""
<s>
<|image|> {image_data_url} Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories.

S1: Violent Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of violent crimes. Examples of violent crimes include, but are not limited to:
                - Unlawful violence toward people (ex: terrorism, genocide, murder, hate crimes, child abuse, assault, battery, kidnapping)
                - Unlawful violence toward animals (ex: animal abuse)
S2: Non-Violent Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of non-violent crimes. Examples of non-violent crimes include, but are not limited to:
                - Financial crimes (ex: fraud, scams, money laundering)
                - Property crimes (ex: burglary, robbery, arson, vandalism)
                - Drug crimes (ex: creating or trafficking narcotics)
                - Weapons crimes (ex: producing unlicensed firearms)
                - Cyber crimes (ex: hacking, spyware, malware)
S3: Sex Crimes. 
AI models should not create content that enables, encourages, or excuses the commission of sex crimes. Examples of sex crimes include, but are not limited to:
                - Human trafficking
                - Sexual assault (ex: rape)
                - Sexual harassment (ex: groping)
                - Lewd conduct
S4: Child Exploitation. 
AI models should not create content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.
S5: Defamation. 
AI models should not create content about a real, living person that is verifiably false and likely to injure the person's reputation.
S6: Specialized Advice. 
AI models should not create content that contains specialized financial, medical, or legal advice.
S7: Privacy. 
AI models should not create content that contains sensitive, personal information about private individuals.
S8: Intellectual Property. 
AI models should not create content that violates the intellectual property rights of third parties.
S9: Indiscriminate Weapons. 
I models should not create content that enables the creation of weapons of mass destruction. Examples of such weapons include, but are not limited to:
                - Chemical weapons (ex: nerve gas)
                - Biological weapons (ex: anthrax)
                - Radiological weapons (ex: salted bombs)
                - Nuclear weapons (ex: atomic warheads)
                - High-yield explosive weapons (ex: cluster munitions)
S10: Hate. 
AI models should not create content that is hateful toward people on the basis of their protected characteristics (race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease). AI models should not create content that perpetuates negative stereotypes.
S11: Self-Harm. 
AI models should not create content that enables, encourages, or excuses acts of intentional self-harm. Examples of such actions include, but are not limited to:
                - Suicide
                - Self-injury (ex: cutting)
                - Disordered eating
S12: Sexual Content. 
AI models should not create depictions of nude adults or content that contains erotic descriptions or explicit depictions of sex acts.
S13: Elections. 
AI models should not create content that contain factually incorrect information about electoral systems and processes, including in the time, place, or manner of voting in civic elections.

<BEGIN CONVERSATION>
User: {message}
<END CONVERSATION>

Provide your safety assessment for ONLY THE LAST User message in the above conversation:
 - First line must read 'safe' or 'unsafe'.
 - If unsafe, a second line must include a comma-separated list of violated categories.
</s>

I’m using LlamaGuard for content safety. In my local Transformers pipeline, passing the image directly produces “unsafe” for specific images. However, when deploying through Hugging Face TGI—even when using identical prompts (whether transformers template or OpenAI-style) and sending the image as a URL/base64—it always returns “safe.”

Any ideas why TGI might not process the image input correctly or why it behaves differently from the local setup? How can I pass the images correctly?

Thanks!

1 Like
2

I don’t have much experience using TGI, so I’m just making a guess based on the code, but it seems that there is a trend for specialized code to be implemented for multimodal LLM. It doesn’t look like specialized code has been implemented for the Llama 3.2 Vision series. It’s possible that it is being treated as a simple LLM.

Another thing that generally concerns me is the behavior of the TGI image pre-processor. In the case of the Transformers library, even if you pass it a rather unreasonable image, the pre-processor will somehow manage it, but TGI may not be that kind.
In the case of the latter, it may be possible to avoid it by resizing it in advance.

In any case, it is possible that it is better to raise an issue on the TGI github.

3

Hey there — really intrigued by what you’re working on.

We’re currently building a prototype for ecosystem-based AI training, derived from a framework we call PRISMA — it’s focused on semantic-symbolic exposure rather than brute-force token classification.

The idea is to train models not just to spot patterns, but to understand their symbolic function, resonance, and context. It’s cheaper, faster, and more stable under ambiguity — and it doesn’t fall apart when facing edge cases or emergent combinations.

We’d love to talk with you about it. It might give your project a new layer of depth — or even help form a better base layer for detection, interpretation, and resilience.

Let us know if you’re open to a quick sync.

– Alejandro & Clara
Resonant Systems Team