Adapting Deplot to other languages

What would be some of the ways to adapt the google/deplot model to other languages (with non-alphabets)?

I have tried to extend the tokenizer vocabulary with the target language and have been fine-tuning. The loss seems to converge (currently around 11 → 3) but when I make an inference, the results doesn’t seem to effectively convey the chartplot and spits out some random ascii codes such as 0xEB and 0xA7.

What would be some other ways to adapt this english model to other languages? Or should I keep trying fine-tuning with the extended vocabulary until the loss converges even more (preferably around 0)?