Hey everyone! I am running into an issue when running inference on Falcon 40B Instruct through SageMaker. I’m trying to generate ~50K datapoints (based on 50K different prompts) but after every couple hundred the model errors out with: ModelError: An error occurred (ModelError) when calling the Inv…

CPU/Memory Utilization Too High When Running Inference on Falcon 40B Instruct

enniop August 31, 2023, 8:57am 5

Hi @philschmid , how were you able to find the list of all the environment variables supported in the container?

Topic		Replies	Views
Inference error for FLAN-UL2 on AWS SageMaker Amazon SageMaker	1	957	April 3, 2023
503 No worker is available when calling single huggingface endpoint Amazon SageMaker	11	4361	April 7, 2022
Sagemaker Serverless Inference Amazon SageMaker	22	9037	May 22, 2024
Impossible to use flan-t5-xxl in a batch-transform job Amazon SageMaker	3	1150	May 23, 2023
Inference failed for FLAN-UL2(20B) on SageMaker Amazon SageMaker	6	2167	April 4, 2023