Hi, I’m working on fine-tuning bloom and deploying the model on Sagemaker and I wanted to know if it’s possible to stream the output generated text, with directly modifying the inference functions? I already tried with applying the TextIteratorStreamer within predict_fn function, but this doesn’t …

Streaming output text when deploying on Sagemaker

Ludivine May 12, 2023, 2:39pm 3

Topic		Replies	Views
Streaming output text When deploying a finetuned (SFT, DPO) model with custom inference script Amazon SageMaker	1	32	November 8, 2024
AWS Sagemaker doesn't return the full response Amazon SageMaker	1	131	July 17, 2024
Deploying a conversational pipeline on AWS Amazon SageMaker	9	4302	July 13, 2023
Model Stream Error - Streaming times out after 60 seconds Amazon SageMaker	0	342	May 15, 2024
Deploying Open AI’s whisper on Sagemaker for audio streaming Amazon SageMaker	2	1524	June 9, 2023