Tokenizer is not being loaded on Huggingface Inference

I was using this to finetune a GPT medium.

tokenizer = GPT2Tokenizer.from_pretrained('gpt2', bos_token='<|startoftext|>', eos_token='<|endoftext|>', pad_token='<|pad|>') #gpt2-medium
configuration = GPT2Config.from_pretrained('gpt2', output_hidden_states=False)
model = GPT2LMHeadModel.from_pretrained("gpt2", config=configuration)

# after finetuning


But when I upload the model and tokenizer on the model hub (ritwikm/gandhi-gpt), I see the following error

Can't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)

On my local machine, I am loading the same tokenizer and model using the following lines:

model = model.from_pretrained(output_dir).to(device)
tokenizer = tokenizer.from_pretrained(output_dir)

And it works fine. It just fails on the inference.

I have tried many solutions like using only those tokenizer files which are available in the official repo (gpt2).

I saw the post by bala1802, even that didn’t help.

What can be done?