Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length

Kowsher · October 22, 2022, 7:06pm

To train a Bert LM, I updated the DataCollatorForLanguageModeling function, and it creates another input_ids as new_input_ids which is the same size as input_ids. To train bert model it accepts input_ids and new_input_ids in BertForPreTraining class. But facing issues in batch size, getting error
as ’ Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length’
I get this issue, comes from this 699 line of this file

github.com

huggingface/transformers/blob/bd9871657bb9500a9f4437a873db6df5f1ae6dbb/src/transformers/tokenization_utils_base.py#L715


      
                          # elif tensor.ndim < 2:
                          #     tensor = tensor[None, :]
          
          
                self[key] = tensor
                  except:  # noqa E722
                      if key == "overflowing_tokens":
                          raise ValueError(
                              "Unable to create tensor returning overflowing tokens of different lengths. "
                              "Please see if a fast version of this tokenizer is available to have this feature available."
                          )
                      raise ValueError(
                          "Unable to create tensor, you should probably activate truncation and/or padding "
                          "with 'padding=True' 'truncation=True' to have batched tensors with the same length."
                      )
          
          
    return self
          
          
@torch_required
          def to(self, device: Union[str, "torch.device"]) -> "BatchEncoding":
              """
              Send all values to device by calling :obj:`v.to(device)` (PyTorch only).

Now how can i keep the same size of new_input_ids same as input_ids for every batch

Topic		Replies	Views
It asks to add padding or truncation but I have already done it Beginners	1	750	October 6, 2023
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ 🤗Transformers	1	662	November 22, 2023
DeBERTa - ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length 🤗Tokenizers	2	1439	October 3, 2023
Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' 🤗Transformers	2	123	July 24, 2024
ValueError in using DataCollator: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length 🤗Transformers	1	6667	January 26, 2023

Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length

Related topics