Not providing them for Inference will definitely increase the inference speed, since the input won’t be padded and be used as it is.
@Oigres since the underlying original model was properly pre-trained on 512 tokens it should perform decently well Yes.
You can add it into your model.tar.gz but I would do some tests in advance before deploying.
2 Likes