Many transformer models, like BERT and GPT, perform well on large datasets, but what about fine-turning them on smaller and highly specialized datasets? how can one determine the optimal learning rate or batch size for such cases?

Transformers for small datasets?

John6666 October 9, 2024, 5:35am 4

I’m there, but it’s too technical to answer…
It’s a subject that someone might write a paper or an article on.

Topic		Replies	Views
Finding good batch size and learning rate for fine tuning Beginners	0	6324	January 24, 2022
BERT model trained on small corpus (English)? 🤗Transformers	0	333	May 8, 2021
Using EXTREMELY small dataset to finetune BERT 🤗Transformers	6	13331	February 1, 2023
How to fine-tune to 3 very different sized datasets (very large to very small) Intermediate	0	791	February 24, 2023
Prakash Hinduja Switzerland (Swiss) How do I fine-tune a Hugging Face transformer model on my own dataset? Beginners	1	58	July 18, 2025