Infinity output from gpt2 model?

Beginners
1

New to hugging face. Trying to use gpt2 model to generate some text, why the following code yields infinity result?

chatgpt says it might due to EOS not correct. I tried a few but does not help.

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained(“gpt2”, torchscript=True).eval()

tokenizer

tokenizer = GPT2Tokenizer.from_pretrained(“gpt2”)
in_text = “Lionel Messi is something”
in_tokens = torch.tensor(tokenizer.encode(in_text))

inference

#token_eos = torch.tensor([198]) # line break symbol
token_eos = torch.tensor([50256])
out_token = None
i = 0
with torch.no_grad():
while out_token != token_eos:
logits, _ = model(in_tokens)
out_token = torch.argmax(logits[-1, :], dim=0, keepdim=True)
in_tokens = torch.cat((in_tokens, out_token), 0)
text = tokenizer.decode(in_tokens)
print(f’step {i} input: {text}', flush=True)
i += 1

out_text = tokenizer.decode(in_tokens)
print(f’ Input: {in_text}‘)
print(f’Output: {out_text}’)

2

Hey HuangLED,

It looks like your code might be getting stuck in an infinite loop because out_token may never match token_eos. You can try adding a maximum iteration limit to your loop to prevent it from running indefinitely. Here’s a modified version of your loop:

max_iterations = 100  # or any suitable number
i = 0
while out_token != token_eos and i < max_iterations:
    logits, _ = model(in_tokens.unsqueeze(0))  # Add batch dimension
    out_token = torch.argmax(logits[0, -1, :], dim=0, keepdim=True)
    in_tokens = torch.cat((in_tokens, out_token), 0)
    text = tokenizer.decode(in_tokens)
    print(f'step {i} input: {text}', flush=True)
    i += 1
3

Thanks for the discussion, Matt.

I am not sure if it is the right thing to do. A hard cut off like this, will just lead to an incomplete sentence at last.

That does not seem a nature response to get I assume?