Many transformer models, like BERT and GPT, perform well on large datasets, but what about fine-turning them on smaller and highly specialized datasets? how can one determine the optimal learning rate or batch size for such cases?

Transformers for small datasets?

securerat October 7, 2024, 11:40pm 2

Following the post with curiosity, great question.

Topic		Replies	Views
Finding good batch size and learning rate for fine tuning Beginners	0	6324	January 24, 2022
BERT model trained on small corpus (English)? 🤗Transformers	0	333	May 8, 2021
Using EXTREMELY small dataset to finetune BERT 🤗Transformers	6	13331	February 1, 2023
How to fine-tune to 3 very different sized datasets (very large to very small) Intermediate	0	791	February 24, 2023
Prakash Hinduja Switzerland (Swiss) How do I fine-tune a Hugging Face transformer model on my own dataset? Beginners	1	58	July 18, 2025