MaziyarPanahi's Mistral 0.2 Merges

Models
1

Hey, I was just looking for derivatives of Mistral 7b Instruct v0.2 and found that @MaziyarPanahi has made like 30 of them.

Wondering if anyone has tried any of these, or if Maziyar themself could comment.

I’m intrigued if a little intimidated to download and test all of these.

Would be great to get a TLDR on which are worth investigating further, and for which purposes.

1 Like
2

Hi @cognitivetech

So I go first by why I did it all at once in a batch:

  • First, I love Mistral 7B the v0.2! The quality, the speed, the multiple language support for something in that size, and last not least the 32k context even if only 16k is solid is everything to me! So many targeted use-cases where the agent can use this model without hitting that 70B you have on expensive hardware
  • I decided to experiment with merging it (both v0.1 and v0.2) with other 7B models fine-tuned on top of Mistral
  • It was time consuming to do this one by one, so I automated the top 200 models baed on mistral 7B models and pulled the trigger.

The results for some merges are not bad but not better due to confusing prompt template mostly. Some merges are very interesting in terms of keeping the same quality of original Mistral instruct models, but also adding a bit of the other domain-specific model.

I have been trying to find a good eval package that is fast and also useful to run over all of this and update the README for each model. Specially now that I find this library being open-sourced recently: GitHub - huggingface/lighteval

I am trying to figure out how to use lighteval and once it’s done I guess I will automate the eval right after my merges to be more intuitive.

In the meantime, my advice is, if you see a model that you like and it is fine-tuned based on Mistral 7b and I have a merge of that, you should try that to see how it performs given you know both models well. (but not downloading them all and trying them at once)

1 Like
3

first I have to make my own eval, then we can talk :smiley:

1 Like
4

oh yeah, don’t forget about Snorkel :smiley:

1 Like
5

I’ll keep track of models I like, here:

1 Like