How can I know what loss function I am using?

[joaorr13]

Hello,
I am finetunning my first model and I have to write a report on that, but I am having a little trouble finding what my loss function is.
This is my setup for training:
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, seed=42)
from transformers import EarlyStoppingCallback

early_stopping_callback = EarlyStoppingCallback(early_stopping_patience=3)

trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=train_tokenized_dataset,
eval_dataset=valid_tokenized_dataset,
data_collator=data_collator,
callbacks=[debug_callback, early_stopping_callback],
args=SFTConfig(
output_dir=“./output”,
eval_strategy=“epoch”,
save_strategy=“epoch”,
learning_rate=2e-4,
num_train_epochs=200,
per_device_train_batch_size=1800,
per_device_eval_batch_size=1800,
metric_for_best_model=“eval_loss”,
bf16=True,
),
)

How can I get my loss function expression?

[John6666]

Hmm… Maybe this?

SFTTrainer Loss function

The loss function being used is the cross-entropy loss. It is defined within the model, e.g. here for llama. In case you want to use your own custom loss function, you can overwrite the compute_loss method of the Trainer as explained here. For training an LLM the loss function is computed on the whole concatenated prompts, how to alter this and make loss function only compute on the output prompts Indeed as recommended above, the DataCollatorForCompletionOnlyLM can be used for this purpose…

https://stackoverflow.com/questions/71581197/what-is-the-loss-function-used-in-trainer-from-the-transformers-library-of-huggi
https://stackoverflow.com/questions/72350835/how-to-plot-loss-when-using-hugginfaces-trainer

Supervised Fine-tuning Trainer