Data collator for training bart from scratch

zuujhyt · November 13, 2020, 9:13am

Hello,
I would like to train bart from scratch.
It seems the official example script is not available yet (if any, please tell me!).
So I try to have one by modifying the example scripts run_mlm.py and run_clm.py.
And not sure how to set the data collator part for bart.

In run_mlm.py , DataCollatorForLanguageModeling is used:

github.com

huggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_mlm.py#L342


    # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map
    tokenized_datasets = tokenized_datasets.map(
        group_texts,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
    )
# Data collator
# This one will take care of randomly masking the tokens.
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=data_args.mlm_probability)
# Initialize our Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
    eval_dataset=tokenized_datasets["validation"] if training_args.do_eval else None,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

In run_clm.py , default_data_collator is used:

github.com

huggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_clm.py#L310


)
# Initialize our Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"] if training_args.do_train else None,
    eval_dataset=lm_datasets["validation"] if training_args.do_eval else None,
    tokenizer=tokenizer,
    # Data collator will default to DataCollatorWithPadding, so we change it.
    data_collator=default_data_collator,
)
# Training
if training_args.do_train:
    trainer.train(
        model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
    )
    trainer.save_model()  # Saves the tokenizer too for easy upload
# Evaluation

Can someone give some advice about how to set the data collator for bart?
Thanks.

dhruvramani · December 6, 2020, 1:19pm

@zuujhyt Maybe try creating different Dataset classes using different DataCollators. You could then use PyTorch Lightning to create a Dataloader w/ multiple Dataset classes like this, and train Bart on that. I’ll try doing this.

Let me know if you have any updates on this.

Topic		Replies	Views
Create custom data_collator for Huggingface Trainer 🤗Transformers	1	2742	July 22, 2022
Training RoBERTa from scratch: error? 🤗Transformers	0	482	August 26, 2021
Pretraining BART for conditional generation 🤗Transformers	1	641	May 30, 2022
Getting started with community BART model with no documentation Beginners	0	295	May 5, 2021
Tokenizer to dataset to datacollator Beginners	1	1185	April 28, 2022

Data collator for training bart from scratch

Related Topics