Hi,
How can I see the tokens with tokenizer()
? In the example:
raw_inputs = [
"I've been waiting for a HuggingFace course my whole life.",
"I hate this so much!",
]
inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="pt")
print(inputs)
The inputs
contains the unique numbers. I would like to see the tokens. For example, I want to know if ['I've']
is the token or ['i',''','ve']
is the token.