Is Flash Attention implemented in GPTBigCodeModel?

dawn17 · June 27, 2023, 8:23am

I wanted to know if the MultiQuery Attention implemented in GPTBigCodeModel is actually Flash Attention? I think it is plain MQA but the paper says that they used Flash Attention. I am a bit confused.

Code Link: transformers/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py at v4.30.0 · huggingface/transformers · GitHub

kernelpanic · July 6, 2023, 11:23pm

You are right. It is a plain MQA implementation in the transformers version.

Flash attention is implemented in the Megatron-LM version.
(Megatron-LM/megatron/model/transformer.py at multi-query-attention · bigcode-project/Megatron-LM · GitHub)

Topic		Replies	Views
Swapping GPT-2 Attention with Flash Attention 🤗Transformers	3	2708	June 4, 2023
A potential method to add emotional implicit memory and explicit memory to transformers Show and Tell	0	78	March 29, 2024
Perplexity of BlenderBot 🤗Transformers	0	390	May 13, 2021
Key-value pair from attention layer of GPT2 🤗Transformers	0	250	June 28, 2023
Non-meaningful response from finetuned GPT-2 model 🤗Transformers	0	341	June 26, 2023

Is Flash Attention implemented in GPTBigCodeModel?

Related Topics