Is_split_into_words in encode_plus

gaussclb · February 27, 2023, 11:32am

See code, transformers/tokenization_utils.py at ae54e3c3b18bac0832ad62ea9b896dfd52a09850 · huggingface/transformers · GitHub

In _encode_plus function, it invokes self.tokenize(t, is_split_into_words=True, **kwargs), but tokenize() doesn’t use is_split_into_words.

See code,

github.com

huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/tokenization_utils.py#L481


      
                      Whether the number of added tokens should be computed in the case of a sequence pair or a single
                      sequence.
          
          
    Returns:
                  `int`: Number of special tokens added to sequences.
              """
              token_ids_0 = []
              token_ids_1 = []
              return len(self.build_inputs_with_special_tokens(token_ids_0, token_ids_1 if pair else None))
          
          
def tokenize(self, text: TextInput, **kwargs) -> List[str]:
              """
              Converts a string in a sequence of tokens, using the tokenizer.
          
          
    Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies
              (BPE/SentencePieces/WordPieces). Takes care of added tokens.
          
          
    Args:
                  text (`str`):
                      The sequence to be encoded.
                  **kwargs (additional keyword arguments):

It’s very weird.

Topic		Replies	Views
Space token ' ' cannot be add when is_split_into_words = True 🤗Tokenizers	1	401	March 11, 2021
Is_pretokenized argument for tokenizer doesn't work? 🤗Transformers	1	1535	September 18, 2020
Question on splitting input sequence Beginners	3	4283	June 14, 2022
Tokenizer splits up pre-split tokens 🤗Tokenizers	9	5005	February 9, 2024
Tokenizing two sentences with the tokenizer Beginners	1	2194	October 18, 2021

Is_split_into_words in encode_plus

Related Topics