Using customized algorithm

ockapuh August 17, 2022, 4:38pm 1

Hi everyone.

Is it possible to use a customized algorithm for tokenizing and building the vocab during Language Modeling?

Thanks!

1 Like

Topic		Replies	Views
Customized tokenizers Beginners	0	203	August 18, 2022
Tokenizer effect on the fine-tuning Research	0	260	October 6, 2023
Building Own Knowledge Base LLM Beginners	1	140	April 6, 2024
Creating custom language model for sentence classification Beginners	0	199	November 23, 2020
Keyword generation using T5 Models	4	1485	November 2, 2022