Difference between transformer encoder and decoder

voidful · March 12, 2021, 3:26pm

It is because of the dropout.

[Causal Language Modeling] seems not as expected

opened 03:36PM - 06 Mar 21 UTC

closed 07:02AM - 12 Mar 21 UTC

# Problem Causal Models is only attended to the left context. Therefore causal… models should not depend on the right tokens. For example, The word embedding of "I" will be unchanged no matter what is in the right In GPT2. Since Causal Language Model are uni-directional self-attention. ``` from transformers import AutoModel,AutoTokenizer, AutoConfig import torch # gpt gpt_model = AutoModel.from_pretrained('gpt2') gpt_tokenizer = AutoTokenizer.from_pretrained('gpt2') embeddings = gpt_model.get_input_embeddings() # create ids of encoded input vectors decoder_input_ids = gpt_tokenizer("<pad> Ich will ein", return_tensors="pt", add_special_tokens=False).input_ids # pass decoder input_ids and encoded input vectors to decoder lm_logits = gpt_model(decoder_input_ids).last_hidden_state # change the decoder input slightly decoder_input_ids_perturbed = gpt_tokenizer("<pad> Ich will das", return_tensors="pt", add_special_tokens=False).input_ids lm_logits_perturbed = gpt_model(decoder_input_ids_perturbed).last_hidden_state # compare values of word embedding of "I" for input_ids and perturbed input_ids print("Is encoding for `Ich` equal to its perturbed version?: ", torch.allclose(lm_logits[0, 0], lm_logits_perturbed[0, 0], atol=1e-3)) ``` Result ``` Is encoding for `Ich` equal to its perturbed version?: True ``` However, when it comes to other models, the result is not following the assumption, the logits will be changed when changing the right side input? What is the reason? Is it a bug? I really want to know the answer, thank you! BERT ``` Is encoding for `Ich` equal to its perturbed version?: False ``` BART ``` Is encoding for `Ich` equal to its perturbed version?: False ``` Roberta ``` Is encoding for `Ich` equal to its perturbed version?: False ``` Experiment notebook [colab](https://colab.research.google.com/drive/15V37RWAL40vhrk-uBIh9m99j1gZMLjUy?usp=sharing) ## Environment info - `transformers` version: 4.3.3 - Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic - Python version: 3.7.10 - PyTorch version (GPU?): 1.7.1+cu101 (False) - Tensorflow version (GPU?): 2.4.1 (False) - Using GPU in script?: <fill in> - Using distributed or parallel set-up in script?: <fill in> ### Who can help - @patrickvonplaten - @LysandreJik - @patil-suraj  ## Information Model I am using (GPT, Bert, RoBerta, BART ForCausalLM): The problem arises when using: * [ x] the official example scripts: https://huggingface.co/blog/encoder-decoder#decoder ## To reproduce Experiment notebook [colab](https://colab.research.google.com/drive/15V37RWAL40vhrk-uBIh9m99j1gZMLjUy?usp=sharing) ## Expected behavior Causal Models should not be affected by the right context?

Topic		Replies	Views
GPT-GPT encoder decoder 🤗Transformers	0	287	May 4, 2021
Causal masks in BERT vs. GPT2 Intermediate	4	2724	December 30, 2022
BERT vs GPT architectural, conceptual and implemetational differences 🤗Transformers	0	999	November 26, 2021
Forward-looking or left-context attention mask (left-to-right) generation with BertGeneration and RobertaForCausalLM 🤗Transformers	3	1354	October 27, 2020
Difference Between Attention Mask and Causal Mask 🤗Transformers	1	6938	September 2, 2024

Difference between transformer encoder and decoder

Related topics