Need help fine-tuning Llama3 for log anomaly detection

Hello everyone.

I need help fine-tuning Llama3 to analyse exception messages from log files generated by a Windows application.

I created a dataset in huggingface with a lot of possible inputs, which are the exception messages, and the outputs, which are divided into 7 categories:

  • Coding issue
  • Network issue
  • Database issue
  • Infrastructure issue
  • Memory issue
  • Service issue
  • Irrelevant

However, the dataset is very small and when I’m testing the model I’m noticing either wrong answers or a lot of hallucinations.
For example: If I ask it to tell me the type of problem a certain ExcpMessage represents it starts answering with the “instruction” and “input” fields over and over, instead of only the “output”

I’m also using Unsloth and the Colab code to fine-tune Llama3.

The dataset: https://huggingface.co/datasets/agoncalves/log_excp_messages . There are only 146 rows as I can not find anymore examples to feed the model.

Important to note that the only thing I’ve changed in the Colab code was removing the “instruction” variable, adding temperature=0 and changing the dataset from alpaca to mine.

Should I change something else? Or perhaps changing the dataset? I’m still a beginner and there are a lot of things I don’t understand.

Any tips are welcome.

Thank you