Ah, actually, I think I see the issue, right after posting it.
I think it’s about setting up the tokenizer? Could that be it?
In preprocess I make sure to have overlap, pad and max length = model.max_length
I’m guessing by default when I just pass tokenizer to the pipeline, none of these get set.
Can I pass a pre setup tokenizer?
Figured it out, there are two parameters, max_seq_len and max_answer_len that can set during the call but not initialization.
thanks a lot!