Tokenizer method inference

[jinoooooooooo]

This is a beginner question but

How does huggingface infer what type of method to use for tokenizing the sentence? Can you point me where the implementation could be?

Thanks

[John6666]

Maybe simply written in a json file in the same folder.

tokenizer_config.json · unsloth/Llama-3.2-1B-Instruct at main

[jinoooooooooo]

not really, i couldnt find it

[John6666]

Oh, you’re looking for code. The following is called based on the YAML set in the model repo and the json as above. in Llama’s case, this is it.

huggingface/transformers/blob/main/src/transformers/models/llama/tokenization_llama.py