Need help fine-tuning Llama3 for log anomaly detection

Beginners
1

Hello everyone.

I need help fine-tuning Llama3 to analyse exception messages from log files generated by a Windows application.

I created a dataset in huggingface with a lot of possible inputs, which are the exception messages, and the outputs, which are divided into 7 categories:

  • Coding issue
  • Network issue
  • Database issue
  • Infrastructure issue
  • Memory issue
  • Service issue
  • Irrelevant

However, the dataset is very small and when I’m testing the model I’m noticing either wrong answers or a lot of hallucinations.
For example: If I ask it to tell me the type of problem a certain ExcpMessage represents it starts answering with the “instruction” and “input” fields over and over, instead of only the “output”

I’m also using Unsloth and the Colab code to fine-tune Llama3.

The dataset: https://huggingface.co/datasets/agoncalves/log_excp_messages . There are only 146 rows as I can not find anymore examples to feed the model.

Important to note that the only thing I’ve changed in the Colab code was removing the “instruction” variable, adding temperature=0 and changing the dataset from alpaca to mine.

Should I change something else? Or perhaps changing the dataset? I’m still a beginner and there are a lot of things I don’t understand.

Any tips are welcome.

Thank you

2

Hi,

I don’t know whether I clearly understand your question.
If you want to reduce hallucination, usually it is good that if you can have some external “oracle”, that you can validate the answer of the LLM efficiently (Although the oracle cannot generate the answer itself). For example, maybe you can check one of our work to get a sense of how it works: Assuring LLM-Enabled
Cyber-Physical Systems. I’m happy to discuss more with you.

Best,
Mengyu