Loss function used in run_mmimdb.py

zuujhyt · December 28, 2020, 10:11am

Hi,
Why BCE was used in

huggingface/transformers/blob/8e74eca7f2b3235f8d5340d66361ea656c67bac7/examples/research_projects/mm-imdb/run_mmimdb.py#L524


logger.info("Training/evaluation parameters %s", args)
# Training
if args.do_train:
    train_dataset = load_examples(args, tokenizer, evaluate=False)
    label_frequences = train_dataset.get_label_frequencies()
    label_frequences = [label_frequences[l] for l in labels]
    label_weights = (
        torch.tensor(label_frequences, device=args.device, dtype=torch.float) / len(train_dataset)
    ) ** -1
    criterion = nn.BCEWithLogitsLoss(pos_weight=label_weights)
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, criterion)
    logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
# Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
    logger.info("Saving model checkpoint to %s", args.output_dir)
    # Save a trained model, configuration and tokenizer using `save_pretrained()`.
    # They can then be reloaded using `from_pretrained()`
    model_to_save = (
        model.module if hasattr(model, "module") else model

The labels seem not binary to me.
label_frquencies looks like

Counter({'Drama': 8415, 'Comedy': 5107, 'Romance': 3226, 'Thriller': 3110, 'Crime': 2287, 'Action': 2154, 'Adventure': 1609, 'Horror': 1603, 'Mystery': 1231, 'Sci-Fi': 1212, 'Documentary': 1195, 'Fantasy': 1162, 'Family': 975, 'War': 804, 'Biography': 772, 'History': 663, 'Music': 632, 'Animation': 586, 'Musical': 503, 'Western': 423, 'Sport': 379, 'Short': 281, 'Film-Noir': 202})

What may I miss? Thanks.

Topic		Replies	Views
Mullti Label Text Classification 🤗Transformers	2	976	June 26, 2023
Explanation about VIDEOMAE loss function for multilabel fine-tuning Models	0	198	August 24, 2023
Multi-label token classification 🤗Transformers	34	5758	September 6, 2023
Distilbert-base-multilingual-cased' Beginners	2	507	June 22, 2021
Transformers replacing loss function 🤗Transformers	0	2612	March 26, 2022

Loss function used in run_mmimdb.py

Related Topics