Dealing with Decimal and Fractions

Hi Team,

I’m currently dealing with dataset which contains decimal and fraction and my task have big impact because of those numbers.

When I’m trying Bert based tokenizer, its tokenizing “1-1/2” as [1,-,1,/,2] but I want it as single token i.e [“1-1/2”]
Something similar happening with decimals too.

Please suggest the possible solutions to tokenize it properly.

Thanks
Ashish