Ran into the same issue and this post was helpful. Looked into AutoTokenizer, which addressed the issue.