Using pipeline parameters with Sagemaker

Amazon SageMaker
1

I’m struggling to pass pipeline parameters to a model hosted on sagemaker. I’m trying to deploy a token-classification model which is stored in S3 as a model.tar.gz with the ‘stride’ parameter so that it does the automatic batching of large text. I appear to be bumping into some issue where it thinks the parameter isn’t valid. When I use “stride” at either pipeline creation or invocation I get a _sanitize_parameters() error. Weirdly I don’t get the error in the local version.

my inference script that doesn’t work:

def model_fn(model_dir):

    pipe = pipeline("token-classification", model=model_dir, aggregation_strategy='simple') #stride here doesnt work
    pipe.tokenizer.model_max_length = 512

    return pipe 
 
 
def predict_fn(data, pipe):

    input_data = data.pop('inputs')

    parameters = data.pop('parameters', None)

    if parameters:
        predictions = pipe(input_data, **parameters) #stride in parameters doesn't work either
    else:
        predictions = pipe(input_data)


    return [{"generated_text": predictions}]

Equivalent local version that does work:

pipe = pipeline("token-classification", model="hf-model", aggregation_strategy='simple',stride=25)

error:

  "code": 400,
  "type": "InternalServerException",
  "message": "_sanitize_parameters() got an unexpected keyword argument \u0027stride\u0027"

Would appreciate any guidance with this!