Looking at the list of supported models for TGI, I do not see Qwen 2.5 listed.
Could I host an inference server for a fine-tuned model whose base is Qwen 2.5 Coder 7B in BF16 using the TGI offer?
1 Like
I think it’s probably just that the manual hasn’t been updated.
Qwen 2.5 works with the Serverless Inference API and is the most popular.
On the other hand, it is currently almost impossible to run models that you have improved yourself with the Serverless Inference API. I think it’s because the HF server doesn’t have enough resources.
Of course, there is no problem with running it locally or using it from Spaces or other virtual environments after placing it on HF.
Yes Qwen 2.5 is supported, since it is identical to Qwen 2.
This is similar to the llama models (the docs only list llama, but it supports any version of llama).
2 Likes