Multimodal datasets and corresponding models

[wRise]

Is there a model that can process multi-modal data like “CMU-MOSI” on huggingface? I’ve just learned, please advise.

[John6666]

If you search for models that can handle images and text using keywords such as VQA or VL, you should be able to find many. There are still very few models that can handle audio, but the following are some well-known, recent examples.

microsoft/Phi-4-multimodal-instruct · Hugging Face

Qwen/Qwen2-Audio-7B-Instruct · Hugging Face

[wRise]

@John6666

If you search for models that can handle images and text using keywords such as VQA or VL, you should be able to find many. There are still very few models that can handle audio, but the following are some well-known, recent examples.

appreciate