Implementation of Two Distinct Datasets with HuggingFace Trainer Module

[Mdrnfox]

@rkabir

Hi, I was wondering if anyone could provide any insight into the most effective/efficient way of implementing finetuning with the HuggingFace Trainer module while incorporating two distinct datasets. Basically, I am looking for a way to have it so that the finetuning process goes over two distinct sets of text such that each epoch guarantees traversing over the entirety of both of them. Moreover, the implementation should allow for distinct batch sizes for each set (e.g. 32 and 128).

You could try this I think…

Override the Trainer.get_train_dataloader() method to return a custom iterator that uses two different dataloader, one for each dataset. Iterate over each set per epoch and you can handle the batch size difference… This would give you a lot of the flexibility that you can control and implement using the given library code. Then you would just have to extend out the trainer class with a multi dataset trainer class