What is the magic behind BartForConditionalGeneration？

BramVanroy · March 29, 2021, 4:13pm

In BartForConditionalGeneration a fixed bias is included as well.

huggingface/transformers/blob/06a6fea7820dc3e89d09430a49bce1c72b173647/src/transformers/models/bart/modeling_bart.py#L1209


@add_start_docstrings(
    "The BART Model with a language modeling head. Can be used for summarization.", BART_START_DOCSTRING
)
class BartForConditionalGeneration(BartPretrainedModel):
    base_model_prefix = "model"
    _keys_to_ignore_on_load_missing = [r"final_logits_bias", r"lm_head\.weight"]

    def __init__(self, config: BartConfig):
        super().__init__(config)
        self.model = BartModel(config)
        self.register_buffer("final_logits_bias", torch.zeros((1, self.model.shared.num_embeddings)))
        self.lm_head = nn.Linear(config.d_model, self.model.shared.num_embeddings, bias=False)

        self.init_weights()

    def get_encoder(self):
        return self.model.get_encoder()

    def get_decoder(self):
        return self.model.get_decoder()

github.com

huggingface/transformers/blob/06a6fea7820dc3e89d09430a49bce1c72b173647/src/transformers/models/bart/modeling_bart.py#L1293


    head_mask=head_mask,
    decoder_head_mask=decoder_head_mask,
    past_key_values=past_key_values,
    inputs_embeds=inputs_embeds,
    decoder_inputs_embeds=decoder_inputs_embeds,
    use_cache=use_cache,
    output_attentions=output_attentions,
    output_hidden_states=output_hidden_states,
    return_dict=return_dict,
)
lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias

masked_lm_loss = None
if labels is not None:
    loss_fct = CrossEntropyLoss()
    masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1))

if not return_dict:
    output = (lm_logits,) + outputs[1:]
    return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output

Topic		Replies	Views
Inheriting from BartForConditionalGeneration into a new class - weight not initializing Beginners	4	953	March 16, 2021
Generate function returns random words for BartForConditionalGeneration Beginners	1	301	February 28, 2021
Question regarding training of BartForConditionalGeneration Models	1	2046	March 2, 2021
BartForConditionalGeneration : lm_head layer dimension change 🤗Transformers	0	456	March 7, 2022
Bug in BartForConditionalGeneration's intialisation of lm_head 🤗Transformers	0	270	October 16, 2021

What is the magic behind BartForConditionalGeneration？

Related topics