Issue in loading the saved tokenizer

[hadnoor]

[hadnoor]

i was using the following code :

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(“Salesforce/xgen-7b-8k-base”,trust_remote_code=True)

tokenizer.pad_token = tokenizer.eos_token

tokenizer.save_pretrained(“toksav”)

loaded_tokenizer = AutoTokenizer.from_pretrained(“toksav”)