Sagemaker MultiRecord Inference Not Completing

Amazon SageMaker
1

Hi,
I’m working on using batch transform for a HuggingFace-hosted LLM. I can’t use the MultiRecord strategy for some reason, as it just ends up only performing inference on a few entries, and skips the rest in my .jsonl.

huggingface_model = HuggingFaceModel(
    env=hub,  # configuration for loading model from Hub
    role=role,
    image_uri=get_huggingface_llm_image_uri("huggingface", version="2.0.2"),
)

batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g5.2xlarge',
    output_path=s3_output_data_path,
    strategy='MultiRecord'  # or 'SingleRecord' depending on your use case
)

batch_job.transform(
    data=s3_input_data_path,
    content_type='application/json',
    split_type='Line',
    wait=True
)

I’d greatly appreciate it if someone could provide some feedback on this issue.