Hi, there, I try to train a RoBERTa model from scratch in the Chinese language. The first step is to build a new tokenizer. First, I followed the steps in the quicktour . After the tokenizer training is done, I use run_mlm.py to train the new model. However, the RoBERTa model training fails and …

Build a RoBERTa tokenizer from scratch

valhalla December 11, 2020, 10:05am 2

Topic		Replies	Views
BartTokenizer with vocab.json and merge.txt which were created by ByteLevelBPETokenizer encode <s> into 3 tokens Beginners	1	5701	January 27, 2021
Pipeline fill-mask error with custom Roberta tokenizer Beginners	1	864	February 8, 2022
Load tokenizer from file : Exception: data did not match any variant of untagged enum ModelWrapper 🤗Tokenizers	3	9604	August 1, 2023
Training RoBERTa from scratch: error? 🤗Transformers	0	603	August 26, 2021
Tokenized sequence lengths 🤗Tokenizers	6	2118	March 10, 2022