RuntimeError on trying to create Inference Endpoint

sungkim11 · August 2, 2023, 7:17pm

I am getting this error when trying to create an Inference Endpoint for fine-tuned LLM. RuntimeError(f"weight {tensor_name} does not exist")\n\nRuntimeError: weight model.layers.0.self_attn.rotary_emb.inv_freq does not exist

Topic		Replies	Views
TGI version 0.9.3 llama2 13B deployment sagemaker RuntimeError Amazon SageMaker	2	668	September 12, 2023
Key Error when trying to deploy inference endpoint Inference Endpoints on the Hub	2	776	December 3, 2023
Model won't load on custom inference endpoint Inference Endpoints on the Hub	2	329	June 13, 2024
Error while trying to host finetuned model on inference endpoint Inference Endpoints on the Hub	2	394	May 22, 2024
"Unknown error" in the hub, but model works locally 🤗Hub	0	968	June 24, 2022

RuntimeError on trying to create Inference Endpoint

Related topics