Tokenizing two sentences with the tokenizer

Hi,

As explained in the docs, you can specify several possible strategies for the truncation parameter, including 'only_first'. Also, the encode_plus method is outdated actually. It is recommended to just call the tokenizer, both on single sentence or pair of sentences. TLDR:

inputs = tokenizer(text_a, text_b, truncation='only_first', max_length=max_length)

2 Likes