Research on Hyperparameters for Fine Tuning

Research
1

I fine-tuned the databricks/Dolly-v2-3b with the b-mc2/sql-create-context dataset in order to get the SQL queries for the given context in response, but after Fine Tuning the model even gave worse results instead of SQL queries it gave random statements as a response. And also in SQL queries it is missing the conditions.

SELECT count(*)
FROM head
WHERE age

So, how should we configure the Hyperparameters and what is the relation between Hyperparameters and the model and also what is the best approach to do Fine-Tuning?

2

you should start with low learning rates and also the amount of dataset depends

3

Hi @Pekka10 , you can try using https://huggingface.co/defog/sqlcoder-7b-2 to see if fits your needs. We use this particular prompt format: sql-eval/prompts/prompt.md at main · defog-ai/sql-eval · GitHub.
p.s. I work for defog and am aiming to improve our OSS model so feel free to send any bugs my way.