Attention mask shape (custom attention masking)

[AnneTheBoo]

@alex-hh

Is this correct? Should this be documented? Is there anything to watch out for when doing this? (I’m interested in providing a custom attention masking pattern to the Llama architecture).

Hi,
yes I passed a custom mask in 4D and it worked (confirmed with the attention scores)

Relating to your question - did you perhaps find out how to answer this question regarding additive/binary attention mask?

Does Llama-2 use additive attention masking?

Hi, I need to build a custom causal attention mask for meta-llama/Llama-2-7b-chat-hf from LlamaForCausalML. Can someone confirm that it uses additive masking (0 or large negative number close to -inf) instead of binary masking (0 or 1) for its attention mechanism? I found that binary masking doesn’t work when I investigated the attention scores, whereas additive masking actually lead to attention scores being 0 in the right dimensions. But on the other hand the documentation says it uses bina…