Hello everyone, I’ve been referencing this paper on training transformer based models using metadata enhanced MIDI and was thinking about implementing this using the huggingface transformers and tokenizer libraries as an introduction to these libraries beyond the basic language modeling examples. As…

Tokenizing Domain Specific Text

jodiak November 20, 2020, 4:57am 6

Thank you this is incredibly helpful.

Topic		Replies	Views
Use a pretrained ByteLevelBPETokenizer on text 🤗Tokenizers	1	3833	July 17, 2020
How tokenize natural words by using Tokenizer from transformer pretrained models 🤗Transformers	0	223	November 23, 2022
Using HuggingFace Tokenizers Without Special Characters 🤗Tokenizers	2	1959	November 2, 2022
Tokenizer from own vocab 🤗Tokenizers	0	460	July 11, 2022
wordLevel tokenizing Beginners	0	256	July 4, 2024