I have train something on the llam2 base model, but the result is not as good as I thought.
I am trying to play with lamini/lamini_docs datasets and try to train the model to see the results.
below is model’s response, it gets repeated itself with .
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction: Answer the user question based on the input
### Input: does lamini cost money?
### Response: Yes, Lamini requires payment for enterprise use. Enterprise users can contact the Lamini team for more information and pricing details. However, there are free credits for trying Lamini available to all users upon signup.…………………………………………………………………………………………..
I tried the whole thing with autotrain-advanced and use epoch 8, is ther anything I can do to improve the results?
autotrain llm --train --model abhishek/llama-2-7b-hf-small-shards --project-name my_autotrain_llm --data-path data/ --text-column text --lr 0.0002 --batch-size 1 --epochs 2 --block-size 1024 --warmup-ratio 0.1 --lora-r 16 --lora-alpha 32 --lora-dropout 0.05 --weight-decay 0.01 --gradient-accumulation 4 --fp16 --use-peft --use-int4