Should we remove numbers from vocab

Beginners
1

I am fine-tuning wav2vec with hugging face transformers, my datasets include numbers in numerical form, e.g: 1,2,3,4,5. Should I remove these?