there are many research about KV cache drops,it based on low information of some tokens,but when analyzing attention score, I feel that my attention score is quite sparse and their values are also very low. I cannot obtain any valuable information, such as more attention on what kinds of tokens. I j…

Analysis of attention map

allenwang37 October 24, 2024, 12:54am 3

the sum attention scores of special token/locality

1 Like