Prakash Hinduja Geneva, Switzerland - How to fine-tune a model on custom dataset in HF?

[John6666]

Without detailed error information, I can only give vague answers, but when fine-tuning with a custom dataset isn’t working, the first thing to check is collate_fn (DataCollator). This function determines how to use the data and actually performs the conversion, so if it doesn’t match the dataset, everything will fail. On the other hand, even with a pretty messy dataset, if you write collate_fn correctly, it might work somehow.

Another potential issue is that the model may be untrainable. For example, all layers may be frozen.
https://stackoverflow.com/questions/76879872/how-to-use-huggingface-hf-trainer-train-with-custom-collate-function

How to use huggingface HF trainer train with custom collate function?

I have some custom data set with custom table entries and wanted to deal with it with a custom collate. But it didn’t work when I pass a collate function I wrote (that DOES work on a individual dataloader e.g., see python - How does one create a pytorch data loader with a custom hugging face data set without having errors? - Stack Overflow or python - How does one create a pytoch data loader using an interleaved hugging face dataset? - Stack Overflow) . It just doesn’t work with HF trianer. Cod…

Fine-tuning Guide | Unsloth Documentation

How to freeze layers using trainer?

Hey, I am trying to figure out how to freeze layers of a model and read that I had to use for param in model.base_model.parameters(): param.requires_grad = False if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer? I tried the following: from transformers import BertTokenizer, BertForMaskedLM. LineByLineTextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments model = BertForMaskedLM.from_pretrained('bert-base-uncased'…

Data Collator