Attention heads of a Peft fine-tuned Whisper model

hashefa · August 9, 2023, 6:01am

Hi
referring to:

gist.github.com

https://gist.github.com/hollance/42e32852f24243b748ae6bc1f985b13a#gistcomment-4625910

alignment-heads.md

To allow the Hugging Face version of Whisper to predict word-level timestamps, a new property `alignment_heads` must be added to the `GenerationConfig` object. This is a list of `[layer, head]` pairs that select the cross-attention heads that are highly correlated to word-level timing. 

If your Whisper checkpoint does not have the `alignment_heads` property yet, it can be added in two possible ways.

**Method 1.** Change the `model.generation_config` property:

```python
# load the model
model = WhisperForConditionalGeneration.from_pretrained("your_checkpoint")

This file has been truncated. show original

“If you fine-tuned your own checkpoint, you may need to inspect the cross-attention weights to find the appropriate layers and attention heads”

I’m looking for a way to find the attention heads of a Peft fine-tuned Whisper model
I need it in the format of:
[[10, 12], [13, 17], [16, 11], [16, 12], [16, 13], [17, 15], [17, 16], [18, 4], [18, 11], [18, 19], [19, 11], [21, 2], [21, 3], [22, 3], [22, 9], [22, 12], [23, 5], [23, 7], [23, 13], [25, 5], [26, 1], [26, 12], [27, 15]]

Thanks