Hey everyone! I am running into an issue when running inference on Falcon 40B Instruct through SageMaker. I’m trying to generate ~50K datapoints (based on 50K different prompts) but after every couple hundred the model errors out with: ModelError: An error occurred (ModelError) when calling the Inv…

CPU/Memory Utilization Too High When Running Inference on Falcon 40B Instruct

cvetanovskaa June 26, 2023, 2:38pm 2

Hi @philschmid do you happen to have any ideas what might be going on?

Topic		Replies	Views
Sagemaker Serverless Inference Amazon SageMaker	22	8999	May 22, 2024
Fail predict using Falcon-7B-Instruct 🤗Transformers	0	658	June 1, 2023
Deploy falcon 7b problems Beginners	4	2782	June 6, 2023
Getting error in the inference stage of Transformers Model (Hugging Face) 🤗Transformers	0	782	October 11, 2022
CUDA error for inference on GPU instance Amazon SageMaker	2	761	May 16, 2023