Hi, I am learning Chapter07-Token classification. The content returned by running the following code is inconsistent with the content of the tutorial, and the content of the context is not decode. Could you please tell me the reason?I run on IDEA
from datasets import load_dataset
from transformers import AutoTokenizer
raw_datasets = load_dataset(“squad”)
raw_datasets[“train”].filter(lambda x: len(x[“answers”][“text”]) != 1)
model_checkpoint = “bert-base-cased”
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
context = raw_datasets[“train”][0][“context”]
question = raw_datasets[“train”][0][“question”]
inputs = tokenizer(question, context)
tokenizer.decode(inputs[“input_ids”])
my output:
‘[CLS] to whom did the virgin mary allegedly appear in 1858 in lourdes france? [SEP] [UNK] [SEP]’
output from tutorial:
'[CLS] To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? [SEP] Architecturally, ’
'the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin ’
'Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms ’
'upraised with the legend " Venite Ad Me Omnes ". Next to the Main Building is the Basilica of the Sacred ’
'Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a ’
'replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette ’
'Soubirous in 1858. At the end of the main drive ( and in a direct line that connects through 3 statues ’
‘and the Gold Dome ), is a simple, modern stone statue of Mary. [SEP]’