Hello!
I am using Transformers and Tokenizers on Amazon Web Services (AWS), however I am struggling to import the “transformers” or “tokenizers” library into my AWS Lambda function. I have tried methods with Docker and by zipping the transformers package into Lambda, however none of these approaches have worked for me. I have already trained my model on my local computer, so I do not require any training functionality of transformers on AWS; I only require inference from my trained transformer on AWS.
So far, I have successfully converted my trained transformer model to an .onnx file. This way, I do not have to import “transformers” into AWS. I followed the tutorial here. However, I still require my tokenizer to feed data into my .onnx file, and I am not able to import the “tokenizers” library.
Could I convert my tokenizer to an .onnx file as well? Or is there another way that I could use my tokenizer without requiring the “tokenizer” library?
Thank you very much for all of your help!