Whisper fine tuning

I fine-tuned the whisper-small checkpoint, but in inference I am getting this warning: ‘Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.’ I’m getting this warning after fine-tuning whisper. How can I avoid this error or how can I add special tokens so that I do not get this error?