Using .generate() with CodeParrot

I’m trying to run prediction using CodeParrot. I’d like to use generate() because pipeline is too high-level but __call__ is too low-level:

import transformers
from   transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot")
model = AutoModelForCausalLM.from_pretrained("codeparrot/codeparrot")

config = GenerationConfig(

inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model.generate(


but I get:

ValueError: If `eos_token_id` is defined, make sure that `pad_token_id` is defined.

This makes no sense to me. Isn’t generate() supposed to basically do the work for me? I can run inference manually with this model without having to change model details like the vocabulary… so how come generate() isn’t able to figure it out?

Having the same problem. It seems to be related to using GenerationConfig. Passing parameters directly to generate works fine.

Doing this solved the problem for me, although it still feels like a bug related to GenerationConfig:

config = transformers.GenerationConfig() 
config.eos_token_id = var.tokenizer.eos_token_id

output_ids = var.model.generate(
  generation_config = config,

This should fix your issue

generation_output = model.generate(

1 Like