I was using this to finetune a GPT medium.
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', bos_token='<|startoftext|>', eos_token='<|endoftext|>', pad_token='<|pad|>') #gpt2-medium
configuration = GPT2Config.from_pretrained('gpt2', output_hidden_states=False)
model = GPT2LMHeadModel.from_pretrained("gpt2", config=configuration)
model.resize_token_embeddings(len(tokenizer))
# after finetuning
model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)
But when I upload the model and tokenizer on the model hub (ritwikm/gandhi-gpt), I see the following error
Can't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)
On my local machine, I am loading the same tokenizer and model using the following lines:
model = model.from_pretrained(output_dir).to(device)
tokenizer = tokenizer.from_pretrained(output_dir)
And it works fine. It just fails on the inference.
I have tried many solutions like using only those tokenizer files which are available in the official repo (gpt2).
I saw the post by bala1802, even that didn’t help.
What can be done?