Need Advice on Fine-tuning for Specific Medical Domains to Reduce Loss Stagnation
Hi everyone,
I’m currently working on fine-tuning a pre-trained model for a medical application, and I’ve encountered a bit of a challenge that I hope you can help me with. I started with a dataset of approximately 16k rows, which covers a broad range of medical topics. Initially, the training loss decreased from 0.9 to 0.5, which was encouraging. However, it has since stagnated at 0.5, and I’m not seeing the improvement I was hoping for.
I’m beginning to wonder if the breadth of the medical field covered by my dataset is too wide for effective fine-tuning, especially considering its relatively small size. Would it be more beneficial to narrow down the focus of my dataset to a more specific area within the medical field, such as oncology or another specialized area? I’m particularly concerned about whether the current scope is too generalist and causing the model to miss out on deeper, more nuanced understanding that could be achieved with a more focused approach.
Has anyone here experienced similar challenges with fine-tuning in broad vs. specific domains, especially in the context of specialized fields like medicine? How did you address this, and did focusing on a narrower field help improve your model’s performance and reduce loss stagnation?
Any insights, experiences, or advice would be greatly appreciated.
Thanks in advance for your help!
Best regards,