I’ve exported a custom PyTorch-based Transformer model into ONNX to run it on NodeJS. However, the exported model seems to expect input_ids
directly (and not raw text).
Is there any way I can perform tokenization in JS?
Or is there something I’m missing, wherein the ONNX model itself is capable of performing the tokenization as well?