Hi,
In the Metrics for the Text Summarization part, the rouge results are very detailed but when I call the compute, it only gives 4 numbers like below: 
{'rouge1': 0.923076923076923,
'rouge2': 0.7272727272727272,
'rougeL': 0.923076923076923,
'rougeLsum': 0.923076923076923}
Are these numbers mid-fmeasure scores?
And how can I get all the scores as explained in the course?
(I have checked the rouge space of the Huggingface and the results there look like as above, also.)
Hey! Im having the exact same issue, have you found a fix?
Hi! This course is wonderful. 
Iâm hoping Iâm asking my question in the right place. I am interesting in using a dataset to fine tune a model (but likely for âQuestion Answeringâ rather than âFill-Maskâ).
Are there guidelines or a reference on how to best create a dataset from documents? Like what to keep/throw away? (i.e., sentences less than 4 words?) Should I take my document and turn each individual sentence into a line of text? Does the ordering of those lines of text matter? Should each line of text be a paragraph from my document to maintain context? Those kinds of thingsâŚ
Thanks in advance!
Hello, during translation can a pretrained model of en-to-fr be used for fine-tuning model of spanish to german? Also can someone tell me how translation datasets are created, suppose we have 2 columns: english and french parallel translation. How can this dataset be converted into translation containing dictionary of the translations, like the ones most of the translation dataset have?
Hi, this is about Summarization sub-chapter. In this we are first importing rouge_score using !pip install rouge_score statement and then creating a variable rouge_score using rouge_score = evaluate.load("rouge") statement. So, I would like to understand in the statement scores = rouge_score.compute(predictions=[generated_summary], references=[reference_summary]), will the rouge_score class installed will be used or the evaluate.load() method created object will be used? I am wondering the need for !pip install rouge_score, when we are actually not importin it. Appreciate clarification.
Did you get an answer to this or find a solution? They seem to have reduced teh complexity in the output. I guess this rouge is a real rougue. 
I am currently following the Summarization sub-chapter in Chapter.7. I am using Google Colabs notebook and my network speed is very reasonable. When I execute the trainer.train() step as per the course, it seems it is taking time for ever to complete the execution. Even after almost 30 mts, it shows another 30+ hours as expected execution time ([ 127/9672 24:08 < 30:43:33, 0.09 it/s, Epoch 0.10/8]). What is wrong here, any inputs?
I could not find a solution to calculate mid-fmeasure. The return values of the method is changed so there is no solution for this.
I think, it is impossible with huggingface rouge anymore.
I did not searched in detail to find another solution. 
I think, you are using CPU (not GPU).
First, you must connect to a environment with GPU support. You can select it under Runtime/Change runtime type menu.
Second, you must use âdevice = 0â parameter while creating your model to use the GPU.
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint, device = 0)
or
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)
model.to(0)
Thanks for the suggestion. Nevertheless, I used GPU only and not CPU. I think it was an internet speed issue. Today it got executed, in fact the whole of the chapter in couple of hours. By the way, as far as the rogue is concerned, since there are only one set of values (instead of high, mid and low) for all types, I made small change to the code lines wherever necessary to get everything executed successfully.
what is the purpose of using clone in âpredictions.detach().cpu().clone().numpy()â? isntdetach().cpu().numpy() enough? thx
Hi, Iâm having some trouble downloading the datasets. When I use the 'load_dataset(âamazon_reviews_multiâ,âenâ) command, I receive the following error:
ConnectionError: Couldnât reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_en_train.json (SSLError(MaxRetryError(âHTTPSConnectionPool(host=âamazon-reviews-ml.s3-us-west-2.amazonaws.comâ, port=443): Max retries exceeded with url: /json/train/dataset_en_train.json (Caused by SSLError(SSLCertVerificationError(1, â[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)â)))â)))
Iâm not sure how using the âload_datasetâ function I can pass on the verify=ignore variable to ignore certificate verification, or of I download the datasets individually I can rebuild them as load_dataset(âtextâ).
Is there a package dependency issue Iâm facing?
Ok, I think there is an issue with how ârequestsâ handles SSL certificates or something. Using comments from another issue in huggingface if you downgrade requests to 2.27.1 and in the code:
os.environ[âCURL_CA_BUNDLEâ]=ââ
dataset = load_dataset(âamazon_reviews_multiâ,âenâ)
ahead of whatever request you are making it should work
Hi, I think there is a bug in Training a CLM from scratch tutorial.
It contains the following snippet:
from transformers import DataCollatorForLanguageModeling
tokenizer.pad_token = tokenizer.eos_token
data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)
out = data_collator([tokenized_datasets["train"][i] for i in range(5)])
for key in out:
print(f"{key} shape: {out[key].shape}")
DataCollatorForLanguageModeling will set the labels of the pad token to be -100 and hence the loss will ignore them. This is as expected but note that pad and eos are the same. So, what ends up happening is that even eos token gets ignored by the loss which is not what we want.
Furthermore, in this tutorial, the attention_mask is not corrected to have 0 for the padding tokens. Does that part happen automatically? Could you please provide the code for where that happens?
Please let me know if I missed anything or misinterpreted the code. Thanks!
Did someone get good results at âTraining a causal language model from scratchâ? For me it runs, but the result is not satisfying eg:
# create scatter plot with x, y
plt.figure()
The Amazon reviews amazon_reviews_multi ¡ Datasets at Hugging Face dataset isnât available anymore and this section needs to be updated.
Refer to this discussion amazon_reviews_multi ¡ DatasetGenerationError: Broken Links ?
Any suggestions as to which dataset can be used instead ?
@sgugger
Thank you
I am training bert-base-uncased not (distilbert-base-uncased) MLM on domain specific data. I am using paperspace gradient notebooks, I get this error
TypeError: forward() got an unexpected keyword argument âmasked_token_type_idsâ . training stops in between (I think after 1st epoch). I am trying to fine tune bert-base-uncase from this example Fine-tuning a masked language model - Hugging Face NLP Course . Here is the full stack:
TypeError Traceback (most recent call last)
Cell In [23], line 25
23 for step, batch in enumerate(eval_dataloader):
24 with torch.no_grad():
â> 25 outputs = model(batch)
27 loss = outputs.loss
28 losses.append(accelerator.gather(loss.repeat(batch_size)))
File /usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, kwargs)
1126 # If we donât have any hooks, we want to skip the rest of the logic in
1127 # this function, and just call forward.
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1129 or _global_forward_hooks or _global_forward_pre_hooks):
â 1130 return forward_call(*input, **kwargs)
1131 # Do not call functions when jit is used
1132 full_backward_hooks, non_full_backward_hooks = ,
TypeError: forward() got an unexpected keyword argument âmasked_token_type_idsâ
works for distilbert-base-uncased though!
So I am following the Huggingface course for Question answering and I am getting this error
AttributeError : âTFBertForQuestionAnsweringâ object has no attribute âprepare_tf_datasetâ
from the code given in the course
tf_train_dataset = model.prepare_tf_dataset(
train_dataset,
collate_fn=data_collator,
shuffle=True,
batch_size=16,
)
tf_eval_dataset = model.prepare_tf_dataset(
validation_dataset,
collate_fn=data_collator,
shuffle=False,
batch_size=16,
)
Please what can be done to solve this issue?
package versions:
transformers â 4.18.0
datasets- â 2.14.4
tensorflow â 2.10.0
Here is the course link: Question answering - Hugging Face NLP Course