Generate constraint words within the output sentence and not at its start/end

I am new to the transformers-library and I want to generate text using GPT2 while influencing its output. This I do via the PhrasalConstraint that lets me add certain tokens that are to apear in the sentence generated by GPT2.
Mainly I oriented myself on this blogpost.

Now the problem is, that the constraint words will almost certainly appear after the input or at the end of the generated sentence. Never in the middle of the sentence.
→ I suspect that the constraints do not fit the generated sentence, and are therfore “discarded” to the end/ beginning, to keep up the overall probability of the generation.

input: I like to
constraint: France
output_1 → I like to France is a beautiful county
output_2 → I like to become a carpenter when I grow up France

Does anybody know how I can improve this? Is there a way to make the constrains fit more natural into the generated sentence? Or is there another way to infuse constraints into the generation, that you are aware of?

Thanks for any help!


from transformers import AutoModelForCausalLM, AutoTokenizer, PhrasalConstraint
tokenizer = AutoTokenizer.from_pretrained("dbmdz/german-gpt2",add_prefix_pace=True)
# the model is a german version of gpt2
model = AutoModelForCausalLM.from_pretrained("dbmdz/german-gpt2")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})

encoder_input_str = " Im Krankenhaus musste ihr ein"
force_words = [" Morphium"]

constraints = []
for constraint_entitiy in force_words:

input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
force_words_ids = tokenizer(force_words, add_special_tokens=False).input_ids

outputs = model.generate(
    constraints = constraints,

There seems to be a bug that causes this above transformers==4.20.1
If you downgrade it should work as described in the tutorial linked above.
I will write an Issue on this…

→ Issue: PhraseConstraints apearing only directly after input or at the end of the generated sentence · Issue #19070 · huggingface/transformers · GitHub