Having troubel in understanding what loss is currently in use

aquorio15 · November 24, 2023, 10:23am

I was going through this hugging face code and I am having trouble understanding what loss the model is currently using. Although I know most seq2seq models uses CrossEntrophy loss but I don’t see the definition anywhere in the code

github.com

huggingface/transformers/blob/aca6288ff42cebded5421020f0ff088adeb446dd/examples/language-modeling/run_clm.py

#!/usr/bin/env python
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.

Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=causal-lm

This file has been truncated. show original

Actually I want to train the model with a new custom loss. I have trained a baseline model and its working fine.

Thank You

panigrah · November 24, 2023, 12:21pm

Most of the pre trained model classes have a default loss function that gets called inside the train method. I usually lookup the model source code to find out the loss method if I need to.

If you are using the trainer class you can override the loss method by following the first example on this link.

Topic		Replies	Views
Cross Entropy Loss and loss of HuggingFace T5ForConditionalGeneration does not matches 🤗Transformers	11	4446	November 29, 2023
Saving underlying language model after trained on downstream task 🤗Transformers	0	355	September 14, 2020
BART finetuning for summarization without seq2seq trainer Beginners	1	709	October 31, 2022
Which model do I use for my custom seq2seq data? Models	0	216	July 8, 2023
Fine-tune a Hugginface model with only loss function(without labels)? Beginners	0	316	October 18, 2021

Having troubel in understanding what loss is currently in use

Related Topics