Tokenizer for 'sshleifer/distilbart-xsum-12-6'?

I try to generate outputs using sshleifer/distilbart-xsum-12-6 but it gives me the following error:

OSError: Model name 'sshleifer/distilbart-xsum-12-6' was not found in tokenizers model name list (facebook/bart-base, facebook/bart-large, facebook/bart-large-mnli, facebook/bart-large-cnn, facebook/bart-large-xsum, yjernite/bart_eli5). We assumed 'sshleifer/distilbart-xsum-12-6' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'], but couldn't find such vocabulary files at this path or url.

I am thinking instead of tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-xsum-12-6") we should change it to another tokenizer for xsum? Thank you!

(Again, thank you Sam for your wonderful work.

1 Like

Hi @chz816, this call should work. You can also use BartTokenizer

1 Like

I can download the tokenizer with your code. strange.

1 Like