Evaluating on MMLU while finetuning using Trainer

🤗Transformers
1

Hi all, I’m trying to finetune llama on custom text dataset and would want to check how it’d perform if I evaluated it on MMLU dataset. Here’s what I’m using rn ( the eval_data and train_data are subsets of custom dataset).

trainer = Trainer(
    model = model,
    train_dataset = train_data,
    eval_dataset = eval_data,
    args = **training_args
    data_collator=DataCollatorForLanguageModeling(tokenizer,pad_to_multiple_of=8, mlm=False)
)

If I replace eval_data with MMLU dataset (aka by doing
eval_data = load_datset('mmlu','abstract_algebra')
), how do I tell the trainer to use Binary loss ( 1 or 0) instead of Cross Entropy loss (this is what it’d use normally ig)?