Which tokenization step do you mean for training or inference?
Inference step.
I want to replicate the same tokenization performed in the preprocessing notebook but in inference time.
I am blocked with this problem at this moment. It seems that my input data for inference is not being tokenized as I indicated in the preprocessing notebook. how can I indicate to the inference step to make the proper tokenization?