Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True'

šŸ¤—Transformers
3

Hereā€™s a post that looks relevant. Itā€™s possible that the relevant sections of several ā€œutilsā€ libraries should change:

return torch.tensor(value)

to

return torch.tensor(value, dtype=torch.float16)

or to something similar in the function:

        def as_tensor(value, dtype=None):
            if isinstance(value, list) and isinstance(value[0], np.ndarray):
                return torch.tensor(np.array(value))
            return torch.tensor(value)

In the post, the author edited line 141 of feature_extraction_utils.py. I found identical code at line 724 of tokenization_utils_base.py. Identical code would cause identical errors.

According to the author, this solution worked for him. For me, all I got was another error.