[image] google/reformer-enwik8 · Hugging Face We’re on a journey to advance and democratize artificial intelligence through open source and open science. It seems that the language model does not have a token Is this correct?

Reformer-enwik8 do not have <EOS> token

Flash321 July 28, 2021, 12:34pm 1

It seems that the language model does not have a token Is this correct?

Topic		Replies	Views
Retrieval Augmented Generation using Transformer Eco System 🤗Transformers	0	246	October 12, 2023
Huggingface t5 models seem to not download a tokenizer file 🤗Tokenizers	0	536	December 16, 2022
Incorporating my tokenizer into huggingface 🤗Tokenizers	0	125	February 15, 2024
Suggestions for hugging face transformer models for Code and Formal Languages Intermediate	2	1241	May 3, 2022
Question about the license and tokens 🧨 Diffusers	6	1520	September 1, 2022