How to use llava with huggingface

🤗Transformers
1

I want to use a 7b llava model with huggingface but I can’t really find any docs to use it? Any help would be great

1 Like
2

I deployed a model to SageMaker with the SageMaker deployment card HF provides. Currently this model: hxxps://huggingface.co/liuhaotian/llava-llama-2-13b-chat-lightning-preview/discussions/3
However one of my concerns is that the card states 'HF_TASK': 'text-generation' whereas Llava Llama is rather a text to image / image ‘question-answering’ type of model.
This topic states transformers need tinkering: hxxps://discuss.huggingface.co/t/can-text-to-image-models-be-deployed-to-a-sagemaker-endpoint/20120
So I still haven’t got it working. Plus I didn’t have enough quota on AWS to deploy it in a half decent box with GPU so it’ll be another question if the box can carry its weight at all. I’m surprised noone helped so far to me neither in HF model discussions, GitHub discussions (hxxps://github.com/haotian-liu/LLaVA/discussions/454) or other forums.

3

This HuggingFace discussion says hxxps://discuss.huggingface.co/t/can-text-to-image-models-be-deployed-to-a-sagemaker-endpoint/20120 that an inference.py need to be created. I don’t know what the Llava Llama has though. I tried to look at the files of the model, but I don’t see relevant meta data about this.

This StackOverflow entry hxxps://stackoverflow.com/questions/76197446/how-to-do-model-inference-on-a-multimodal-model-from-hugginface-using-sagemaker is about a serverless deployment case, but it uses a custom TextImageSerializer serializer. Shoudl I try to use something like that?

My Stackoverflow entry: hxxps://stackoverflow.com/questions/77193088/how-to-perform-an-inference-on-a-llava-llama-model-deployed-to-sagemake-from-hug

Reddit: hxxps://www.reddit.com/r/LocalLLaMA/comments/16pzn88/how_to_parametrize_a_llava_llama_model/

4

Check out the following blog post:

It uses HuggingFace Transformers Llava and Runhouse

For AWS SageMaker, you can check out this one: hxxps://www.run.house/blog/quickest-aws-sagemaker-deployment

5

Hi,

LLaVa and BakLLaVa are now supported natively in the Transformers library :slight_smile:

Docs: LLaVa

Checkpoints are on the hub: llava-hf (Llava Hugging Face).

6

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.