I have quantized the meta-llama/Llama-3.1-8B-Instruct model using BitsAndBytesConfig. However when i try deploying it to sagemaker endpoint, it throws error. I traiged this in a docker container locally and see the same issue reproduced. Sharing the code of my docker file and the working snippets of…

Error loading tokenizer: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

John6666 October 10, 2024, 10:25pm 4

ModuleNotFoundError: No module named ‘transformers’

Definitely something wrong in sagemaker side…