When fitting a custom tokenizer, I am preprocessing raw text. I remove punctuation, but then there is no indication of the end/start of a sentence, therefore if I keep the “.” as an example that would be understood by the tokenizer model? Do you have any suggestions?

Preprocessing raw text

antoine2323231 October 26, 2022, 3:21pm 3

Hey bro,
Thanks a lot! Gonna try this out.

Topic		Replies	Views
Tokenizer post_processor help 🤗Tokenizers	1	1378	October 27, 2022
Issue with post-processing 🤗Tokenizers	1	1105	June 15, 2022
Add BOS and EOS when encoding a sentence 🤗Tokenizers	2	14861	August 22, 2022
BOS tokens for mBERT tokenizer 🤗Tokenizers	1	634	April 14, 2021
TemplateProcessing for encoder-decoder 🤗Tokenizers	0	506	November 16, 2022