Special tokens and inference

I have created a token classification model, and I was thinking during inference is it better to remove special tokens on encoding or the prediction will be affected by this change !