Multiple tasks for one fine-tuned LLM

Beginners
1

Hi there! If you want to fine-tune an LLM on your own dataset but want it to perform multiple tasks, what would the best procedure be?

  1. Fine-tune a pre-trained model on your own dataset for one of the specific tasks.
  2. Fine-tune your model.base_model of your own fine-tuned model on another specific task (and repeat) (see reference question).

OR

  1. Fine-tune a pre-trained model on your own dataset for one of the specific tasks.
  2. Fine-tune that same pre-trained model on your own dataset with another one of the specific tasks (and repeat)

Or would it be better to create one model with multiple task heads (see this Medium blog) instead of fine-tuning the model for each task?

Thanks!