Try it out! Replicate the two last steps (tokenization and conversion to input IDs) on the input sentences we used in section 2 (“I’ve been waiting for a HuggingFace course my whole life.” and “I hate this so much!”). Check that you get the same input IDs we got earlier!
decoded_string = tokenizer.decode([101, 146, 112, 1396, 1151, 2613, 1111, 170, 20164, 10932,
2271, 7954, 1736, 1139, 2006, 1297, 119, 102])
print(decoded_string)
[CLS] I’ve been waiting for a HuggingFace course my whole life. [SEP]
decoded_string = tokenizer.decode([146, 112, 1396, 1151, 2613, 1111, 170, 20164, 10932, 2271, 7954, 1736, 1139, 2006, 1297, 119])
print(decoded_string)
I’ve been waiting for a HuggingFace course my whole life.
what’s the difference?