Deploying Fine-Tune Falcon 40B with QLoRA on Sagemaker Inference Error

philschmid · July 18, 2023, 8:51am

Hello,

We updated the script and requirements to merge and save the weights in the safetensors format when training. Meaning there is no conversion needed on the LLM inference container side.

You can update the script and requirements, and it should work when deploying after training.

Topic		Replies	Views
QLoRA trained Mixtral 8x7B deployment error on Sagemaker using text generation inference image Amazon SageMaker	0	143	April 10, 2024
Falcon 40B instruct training with QLora, Sagemaker model artifact location Amazon SageMaker	3	372	September 21, 2023
Sagemaker Pipelines with fintuned llama2 Amazon SageMaker	0	738	September 12, 2023
Use my finetuned Bert Model in SageMaker BatchTransform Amazon SageMaker	4	2663	April 30, 2022
CPU/Memory Utilization Too High When Running Inference on Falcon 40B Instruct Amazon SageMaker	4	1227	August 31, 2023

Deploying Fine-Tune Falcon 40B with QLoRA on Sagemaker Inference Error

Related Topics