Efficiently Merge, then Fine-tune LLMs with mergoo

Show and Tell
1

:star: Efficiently Merge, then Fine-tune LLMs with Mergoo

:rocket: In Mergoo, developed by Leeroo team, you can:

  • Easily merge multiple open-source LLMs
  • Efficiently train a MoE without starting from scratch
  • Compatible with :hugs: :hugs:Transformers and #trainers
  • Supports various merging methods e.g. MoE and Layer-wise merging

In Leeroo (https://www.leeroo.com/), our vision is to democratize AI for individuals everywhere, and open-sourcing is the foundational step toward realizing this vision. Join Leeroo, where we can work together to make this vision a reality!

mergoo: GitHub - Leeroo-AI/mergoo: A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

2 Likes