Alternating between batches of different datasets

Intermediate
1

Hi,

I have two datasets, English and French.
During training, I like each batch to contain samples from only one of the datasets.

For this, I have created a dataloader for each one of the datasets.

For training loop I tried something like:

for batch_en, batch_fr in zip(english_dataloader, french_dataloader):

	outputs_en = model(**batch_en)
	loss_en = outputs_en.loss

	outputs_fr = model(**batch_fr)
	loss_fr = outputs_fr.loss
	
	accelerator.backward(loss_en)
    accelerator.backward(loss_fr)

    accelerator.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    lr_scheduler.step()
    optimizer.zero_grad()
    if accelerator.is_main_process:
        progress_bar.update(1)

However, this causes issues with Accelerator.
Saying RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.

Interleave Datasets is not very useful to me here, as the batches that it creates, contain data from all datasets. Whereas my goal is to have each batch contain data/samples from either English dataset or French dataset.
Any help is greatly appreciated.