Research on Hyperparameters for Fine Tuning

I fine-tuned the databricks/Dolly-v2-3b with the b-mc2/sql-create-context dataset in order to get the SQL queries for the given context in response, but after Fine Tuning the model even gave worse results instead of SQL queries it gave random statements as a response. And also in SQL queries it is missing the conditions.

SELECT count(*)
FROM head

So, how should we configure the Hyperparameters and what is the relation between Hyperparameters and the model and also what is the best approach to do Fine-Tuning?

you should start with low learning rates and also the amount of dataset depends

Hi @Pekka10 , you can try using to see if fits your needs. We use this particular prompt format: sql-eval/prompts/ at main · defog-ai/sql-eval · GitHub.
p.s. I work for defog and am aiming to improve our OSS model so feel free to send any bugs my way.