Hi,
I have an issue with German umlauts python in deepseek-ai/deepseek-coder-1.3b-instruct tokenizer.
If I run the following script I got � für every German umlaut.
from transformers import AutoTokenizer
import torch
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("Target Device: ", DEVICE)
MODEL = "deepseek-ai/deepseek-coder-1.3b-instruct"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
print (tokenizer)
inputs = tokenizer("A", return_tensors="pt").to("cuda")
inputs.input_ids[0][1] = 32009
print (inputs)
decoded = tokenizer.decode(inputs.input_ids[0])
print (decoded)
How to fix this?