Hi! I’m trying to get the alignment values used for the attention weights before the softmax in the GPTLMHead model for a research project. Does anyone know if this is possible/how to do it?
I’m essentially trying to get the attention weights before this part of the model: transformers/src/transformers/models/gpt2/modeling_gpt2.py at 2a9b1f80c45cab19b542bc7cc004937d39d6f6fb · huggingface/transformers · GitHub.
I would appreciate any help!