Customizing generation config for Trainer's training loop evaluation

🤗Transformers
1

When using Trainer’s training loop, I’d like to be able to customize the parameters for generation (used for evaluation). I tried the following approach:

gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config.validate()
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config,
)
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    eval_dataset=...,
    ....
)
trainer.train()

However, the values are not used (for example, max_new_tokens is not respected). This warning is generated:

Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (Text generation strategies) instead. This warning will be raised to an exception in v4.41.
Non-default generation parameters: {‘early_stopping’: True, ‘num_beams’: 4, ‘no_repeat_ngram_size’: 3, ‘forced_bos_token_id’: 0, ‘forced_eos_token_id’: 2}

along with:

/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:1128: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.

I tried to follow the advice from the first warning by saving the modified configuration into a file and then loading it:

gen_config = copy.deepcopy(model.generation_config)
gen_config.update(
    max_new_tokens=10,
    early_stopping=False,
    num_beams=1,
    no_repeat_ngram_size=0,
)
gen_config_file = "gen_config.json"
gen_config.to_json_file(gen_config_file)
training_args = Seq2SeqTrainingArguments(
    ....,
    predict_with_generate=True,
    generation_config=gen_config_file,
)
...

but the warning stays the same.

How can I customize the evaluation generation when using Trainer? Thank you for any advice.

2

I found the problem. The warning was actually triggered by Trainer saving the model via save_pretrained, not by generation. I submitted an issue: BART-base save_pretrained triggers a warning about GenerationConfig · Issue #28793 · huggingface/transformers · GitHub