libraries version issue?
I am trying to load -
from datasets import load_dataset
raw_datasets = load_dataset(“glue”, “mrpc”)
raw_datasets
but I get this error message : /usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning:
The secret HF_TOKEN does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (Hugging Face – The AI community building the future.), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
warnings.warn(
Downloading readme:
35.3k/? [00:00<00:00, 445kB/s]
ValueError Traceback (most recent call last)
/tmp/ipython-input-1-3226990657.py in <cell line: 0>()
1 from datasets import load_dataset
2
----> 3 raw_datasets = load_dataset(“glue”, “mrpc”)
4 raw_datasets
11 frames
/usr/local/lib/python3.11/dist-packages/fsspec/utils.py in glob_translate(pat)
729 continue
730 elif “" in part:
→ 731 raise ValueError(
732 "Invalid pattern: '’ can only be an entire path component”
733 )
ValueError: Invalid pattern: ‘**’ can only be an entire path component
What worked for me was to run
!pip install --upgrade datasets
raw_datasets = load_dataset(“glue”, “mrpc”)
I am getting this error :
/usr/local/lib/python3.11/dist-packages/fsspec/utils.py in glob_translate(pat) 729 continue 730 elif “" in part: → 731 raise ValueError( 732 "Invalid pattern: '’ can only be an entire path component” 733 )
ValueError: Invalid pattern: ‘**’ can only be an entire path component
In google collab, I tried reinstalling the libraries, clearing the .cache/huggingface/datasets/glue
still this error persists
Oh… So we need !pip install -U datasets huggingface_hub[hf_xet] fsspec …
The issue was caused by an incompatibility between the versions of
datasets,huggingface-hubandfsspec.
what is the problem about this code? it gives me an error
from datasets import load_dataset
from transformers import (
AutoTokenizer,
AutoModelForSequenceClassification,
TrainingArguments,
Trainer,
)
import numpy as np
import evaluate
# 1) Data & tokenizer
raw_datasets = load_dataset(“glue”, “mrpc”)
checkpoint = “bert-base-uncased”
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize_function(example):
return tokenizer(example\["sentence1"\], example\["sentence2"\], truncation=True)
tokenized_datasets = raw_datasets.map(
tokenize_function,
batched=True,
)
def compute_metrics(eval_preds):
logits, labels = eval_preds
preds = np.argmax(logits, axis=-1)
metric = evaluate.load("glue", "mrpc")
return metric.compute(predictions=preds, references=labels)
training_args = TrainingArguments(“test_trainer”)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets\["train"\],
eval_dataset=tokenized_datasets\["validation"\],
tokenizer=tokenizer,
compute_metrics=compute_metrics,
)
trainer.train()
How about this…? (Trainer)
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import numpy as np
import evaluate
# 1) Data & tokenizer
raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize_function(example):
return tokenizer(example["sentence1"], example["sentence2"], truncation=True)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
metric = evaluate.load("glue", "mrpc") # metric loaded once outside for efficiency
def compute_metrics(eval_preds):
preds = np.argmax(eval_preds.predictions, axis=-1)
labels = eval_preds.label_ids
return metric.compute(predictions=preds, references=labels)
training_args = TrainingArguments(output_dir="test_trainer")
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
eval_dataset=tokenized_datasets["validation"],
tokenizer=tokenizer,
compute_metrics=compute_metrics,
)
trainer.train()
I am getting an error on the trainer.train() step.
Error Traceback (most recent call last)
/tmp/ipython-input-4032920361.py in <cell line: 0>()
----> 1 trainer.train()
7 frames
/usr/local/lib/python3.12/dist-packages/wandb/sdk/lib/preinit.py in preinit_wrapper(*args, **kwargs)
34 ) -> Callable:
35 def preinit_wrapper(*args: Any, **kwargs: Any) -> Any:
---> 36 raise wandb.Error(f"You must call wandb.init() before {name}()")
37
38 preinit_wrapper.__name__ = str(name)
Error: You must call wandb.init() before wandb.log()
What is wanb.init(), how do I fix it ?
Try adding report_to="none" like args = TrainingArguments(..., report_to="none"). About Wandb
Or try Trackio.
Hi there!
I have a question regarding A full training loop - Hugging Face LLM Course
The “outputs.loss” returns “grad_fn=NllLossBackward0”, from which I understand that the BertForSequenceClassification uses a negative log likelihood loss. However, the modeling_bert .py seems to use a CrossEntropyLoss ( transformers/src/transformers/modeling_bert.py at 9aeacb58bab321bc21c24bbdf7a24efdccb1d426 · huggingface/transformers · GitHub ).
What is the loss function really used in this model?
Thank you very much!
Riad
Maybe:
The loss function is torch.nn.CrossEntropyLoss (for num_labels > 1), which PyTorch implements internally as LogSoftmax + NLLLoss. That is why you see grad_fn=NllLossBackward0. Mathematically, for single-label classification, this is exactly a negative log-likelihood loss.
1. What the model code uses
For BertForSequenceClassification, the loss logic is:
- If
num_labels == 1: regression, usesMSELoss. - If
num_labels > 1: classification, usesCrossEntropyLoss.
So in the typical classification case, the actual loss module is:
torch.nn.CrossEntropyLosson the logits and the integer class labels.
That is the high-level answer.
2. Why you see NllLossBackward in grad_fn
PyTorch defines CrossEntropyLoss as a combination of two operations:
- Apply a log-softmax to the logits.
- Apply a negative log-likelihood loss to those log-probabilities.
Conceptually:
- Let logits be a vector z \in \mathbb{R}^C for C classes.
- Softmax probabilities arep_i = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}.
- Log-softmax is\log p_i = z_i - \log \left( \sum_{j=1}^C e^{z_j} \right).
Internally, CrossEntropyLoss does something equivalent to:
- Compute \log p_i via a log-softmax.
- Feed those log-probabilities into
NLLLoss(negative log-likelihood loss).
Autograd tracks the low-level ops. The last primitive that creates the scalar loss is the NLL part, so the gradient function shows up as something like:
grad_fn=<NllLossBackward0>
even though at the API level you called CrossEntropyLoss.
So there is no contradiction: the HF model calls CrossEntropyLoss, and PyTorch’s implementation of that is what produces NllLossBackward in the computation graph.
3. Cross-entropy vs negative log-likelihood
The course text calling it a “negative log-likelihood loss” is also correct. For standard single-label classification with one correct class per example, cross-entropy equals negative log-likelihood.
Assume:
- There are C classes.
- Model outputs logits z \in \mathbb{R}^C.
- Softmax probabilities:p_i = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}.
- True label index is y \in {0, \dots, C-1}.
- One-hot target vector t has t_y = 1 and t_i = 0 for i \neq y.
Cross-entropy
The cross-entropy between target t and prediction p is
H(t, p) = -\sum_{i=1}^C t_i \log p_i.
For a one-hot target, only t_y = 1 contributes, so
H(t, p) = -\log p_y.
Negative log-likelihood
If we view p_y as the model’s probability for the true class, then the log-likelihood of the data point is
\log p_y,
and the negative log-likelihood (NLL) is
\text{NLL} = -\log p_y.
So for single-label classification,
\text{cross-entropy} = \text{NLL}.
This is exactly what NLLLoss is computing when you supply it with log-probabilities and the class index.
Since CrossEntropyLoss = log-softmax + NLL, for the single-label case it is just a convenient wrapper for computing that negative log-likelihood from raw logits.
4. How this ties back to your observation
Putting it all together:
-
The Hugging Face model code:
- Calls
CrossEntropyLossfor classification.
- Calls
-
PyTorch implementation:
CrossEntropyLossinternally doeslog_softmaxthenNLLLoss.- The backward node is therefore named
NllLossBackward.
-
The math:
- For a one-hot target, cross-entropy equals negative log-likelihood of the true class.
So:
- The HF course saying “negative log-likelihood loss”
- The Transformers code using
CrossEntropyLoss - The
grad_fn=NllLossBackward0
are three consistent descriptions of the same underlying loss for classification.
Short bullet summary
-
For
BertForSequenceClassification:num_labels > 1⇒torch.nn.CrossEntropyLosson logits and labels.num_labels == 1⇒torch.nn.MSELossfor regression.
-
PyTorch’s
CrossEntropyLossis implemented as log-softmax +NLLLoss. -
Autograd exposes the low-level op, so you see
grad_fn=NllLossBackward0. -
For single-label classification, cross-entropy equals negative log-likelihood, so calling it an NLL objective is mathematically accurate.
Thank you very much: this makes more sense now!
R
wandb.init(project="trasformer-fine-tuning", name="bert-mrpc-analysis")
loads infinitely on Kaggle. Am I understanding correct that it tries to connect to server?
When I login
from huggingface_hub import login
wandb.login()
I get
/usr/local/lib/python3.12/dist-packages/notebook/notebookapp.py:191: SyntaxWarning: invalid escape sequence '\/'
| |_| | '_ \/ _` / _` | _/ -_)
and infinite waiting
Do I need API key from wandb.ai?
it tries to connect to server?
seems so.
In your Kaggle notebook, wandb.init(...) is very likely trying to start an online W&B run and set up syncing to the W&B backend. If Kaggle Internet is off or restricted, that can look like “loads forever.” W&B documents that mode="online" is the default for wandb.init, and that offline and disabled exist specifically for air-gapped or restricted environments. (Weights & Biases Documentation)
That SyntaxWarning from notebookapp.py is almost certainly unrelated noise. It does not explain the hang. Your hang is about networking or authentication, not that warning.
What is happening in the background
1) What wandb.init(...) does
Think of W&B as having two steps:
- Auth step (optional): “Who are you?” via
wandb.login()or an API key already stored. - Run step: “Create a run and start logging.” That is
wandb.init(...).
In W&B’s own docs, wandb.init defaults to online syncing, and it is the point where W&B sets up the run and syncing behavior. (Weights & Biases Documentation)
So your understanding is correct: in default mode it is set up to connect to W&B’s service.
2) Why Kaggle triggers the “infinite loading” pattern
Kaggle notebooks have an “Internet” setting. Kaggle explicitly notes (a) you may need Internet enabled, and (b) some competitions require Internet to be disabled for submission. (Kaggle)
If Internet is disabled, any library that tries to reach an external service can hang, retry, or time out in ways that look like a stuck cell.
Also, some users cannot even see the “Internet on” toggle until phone verification is done. (Stack Overflow)
Why wandb.login() can hang in notebooks
wandb.login() often expects either:
- an existing configured API key, or
- a key you pass programmatically, or
- an interactive prompt where you paste the key
In notebook environments, the interactive prompt can fail or never complete. W&B’s login() docs mention a UsageError when an API key cannot be configured and there is “no tty,” which is exactly the class of issue you can hit in hosted notebook environments. (Weights & Biases Documentation)
So “infinite waiting” usually means: it is waiting for key input or stuck in an auth flow that is not completing.
Do you need an API key from wandb.ai?
You need a W&B API key only if you want online logging to wandb.ai.
- Online logging: yes, you need a key (or other auth) and you need Kaggle Internet enabled. W&B references the API key in their offline/online guidance too. (Weights & Biases Documentation)
- Offline logging: no key needed. You log locally and can sync later from a machine with Internet.
mode="offline"exists for this use case. (Weights & Biases Documentation) - Disabled: no key needed. W&B becomes a no-op. (Weights & Biases Documentation)
Clean fixes for Kaggle (pick one)
Option A: Internet must stay OFF (common in competitions)
Do not call wandb.login() at all. Force offline mode.
import os
os.environ["WANDB_MODE"] = "offline" # set before importing/using wandb
import wandb
wandb.init(project="transformer-fine-tuning", name="bert-mrpc-analysis", mode="offline")
Why this works: W&B explicitly supports offline for air-gapped environments. (Weights & Biases Documentation)
Option B: You do not need W&B at all
Disable it.
import os
os.environ["WANDB_MODE"] = "disabled"
import wandb
wandb.init(mode="disabled")
W&B documents disabled as a supported mode (no-op). (Weights & Biases Documentation)
Option C: You want to use wandb.ai from Kaggle (Internet ON)
- Enable Kaggle Internet in Notebook settings. Kaggle documents the Internet toggle and warns it must be off for some competitions. (Kaggle)
- Ensure you can see the toggle. If you cannot, phone verification can be the blocker. (Stack Overflow)
- Use a non-interactive login (avoid notebook prompts). Pass a key explicitly.
import os, wandb
# Put your key in Kaggle Secrets or env vars
wandb.login(key=os.environ["WANDB_API_KEY"], verify=True, timeout=60)
wandb.init(project="transformer-fine-tuning", name="bert-mrpc-analysis")
Why this is the right shape:
- W&B
login()supportskey=...and hasverifyandtimeoutparameters. (Weights & Biases Documentation) verify=Trueforces an immediate server check, so you fail fast if Internet is still blocked.
Two easy-to-miss context points
-
Hugging Face login is unrelated to W&B.
huggingface_hub.login()authenticates to Hugging Face. It does not authenticate to W&B. You still need a W&B key for wandb.ai online logging. -
Your project name has a typo:
"trasformer-fine-tuning"vs"transformer-fine-tuning".
This does not cause hanging, but it will create a project with the misspelled name if online logging succeeds.
If it still hangs after forcing offline
If mode="offline" still hangs, the next suspect is a local W&B background service startup problem (separate from Internet). W&B has many reports around “timed out waiting for wandb service to start.” (GitHub)
In that case, mode="disabled" is the fastest way to confirm whether W&B itself is the source of the stall.
Summary
- Yes, default
wandb.init()is online-mode behavior and can try to reach W&B servers. (Weights & Biases Documentation) - Kaggle often has Internet disabled (or required disabled for competitions), which makes W&B look stuck. (Kaggle)
wandb.login()can “wait forever” in notebooks because it is waiting for key input or has no TTY. (Weights & Biases Documentation)- You need a W&B API key only for wandb.ai online logging. Offline or disabled modes do not need it. (Weights & Biases Documentation)
Thank you very much for your very quick and detailed response, John!
Getting following errors ( Processing the data - Hugging Face LLM Course ) on executing this, any help
from transformers import AutoTokenizer
checkpoint = “bert-base-uncased”
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
tokenized_sentences_1 = tokenizer(raw_datasets[“train”][“sentence1”])
tokenized_sentences_2 = tokenizer(raw_datasets[“train”][“sentence2”])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[30], line 7
3 checkpoint = "bert-base-uncased"
4 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 7 tokenized_sentences_1 = tokenizer(raw_datasets["train"]["sentence1"])
8 tokenized_sentences_2 = tokenizer(raw_datasets["train"]["sentence2"])
File ~/dev/meetme2meat/course/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_base.py:2456, in PreTrainedTokenizerBase.__call__(self, text, text_pair, text_target, text_pair_target, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, tokenizer_kwargs, **kwargs)
2454 if not self._in_target_context_manager and hasattr(self, "_switch_to_input_mode"):
2455 self._switch_to_input_mode()
-> 2456 encodings = self._encode_plus(
2457 text=text,
2458 text_pair=text_pair,
2459 padding_strategy=padding_strategy,
2460 truncation_strategy=truncation_strategy,
2461 max_length=max_length,
2462 **all_kwargs,
2463 )
2464 if text_target is not None:
2465 if hasattr(self, "_switch_to_target_mode"):
File ~/dev/meetme2meat/course/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_tokenizers.py:804, in TokenizersBackend._encode_plus(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, padding_side, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens, **kwargs)
801 return False
803 if not _is_valid_text_input(text):
--> 804 raise ValueError(
805 "text input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) "
806 "or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs)."
807 )
809 if text_pair is not None and not _is_valid_text_input(text_pair):
810 raise ValueError(
811 "text input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) "
812 "or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs)."
813 )
ValueError: text input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).