Gpt2 token of specific string

🤗Transformers
1

I have following texts to tokenize:

tokenizer = GPT2Tokenizer.from_pretrained(‘gpt2’, bos_token=‘<|startoftext|>’, eos_token=‘<|endoftext|>’, pad_token=‘<|paf|>’)

x =

text1 = tokenizer(‘<|startoftext|>’ + ‘Question. A’ + ‘<|endoftext|>’, truncation=True, max_length=20, padding=“max_length”)
x.append(text1[‘input_ids’])

text2 = tokenizer(‘<|startoftext|>’ + ‘Question. B’ + ‘<|endoftext|>’, truncation=True, max_length=20, padding=“max_length”)
x.append(text2[‘input_ids’])

text3 = tokenizer(‘<|startoftext|>’ + ‘Question. C’ + ‘<|endoftext|>’, truncation=True, max_length=20, padding=“max_length”)
x.append(text3[‘input_ids’])

print(x)

and this is what I got:
[[50257, 24361, 13, 317, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 347, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 327, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258]]

text1, text2, text3 always have the same length. Is there anyway to output the token of ‘A’, ‘B’, ‘C’? Is it right before 50256 which is eos_token i guess?