GPT2 returns sequence of <|endoftext|> after finetuning

[tempdeltavalue]

Here’s I tried to finetune gpt2

tempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb

but after few iterations I started to get same sequences, for ex.
prompt: Prime number is

model output: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

By the way, my text divided into sentences without [EOS] tokens, in other words every item is separate sentence

[tempdeltavalue]

upd:

It’s reproducible even without finetuning

[tempdeltavalue]

one top screenshot - correct output

on bottom - eos sequence

But code generally the same