P-tuned Phi-2 produces open ended, repeating, responses

[jpodivin]

I’ve worked with the guide from peft docs only slightly adjusting the preprocessing function to resolve one bug and substituting phi-2 for the original model. The trained model produces repeating output, for example:

[‘Tweet text : @NYTsupport i have complained a dozen times & yet my papers are still thrown FAR from my door. Why is this so hard to resolve? Label : complaintaintcomplaintaint’]

I’m wondering what could be the cause and if there is anything that can be done about it?

[jpodivin]

I found out where the problem was. Turns out the eos token set in config doesn’t actually influence what’s going on during inference. It has to be set as an argument of the generate method. [0] All credit for this solution should go to Eugenio-Schiavoni who first wrote about it on github.com

tldr:
Instead of:

model.config.eos_token_id = tokenizer.eos_token_id

use:

model.generate(input_tokens, eos_token_id=tokenizer.eos_token_id)

[0]When using model.generate, it does not stop at eos_token, but instead continues until the maximum length. · Issue #23175 · huggingface/transformers · GitHub