Barkmodel not intialising with flash_attention_2

damith219 · March 1, 2024, 4:36am

initialising a barkmodel with attn_mechanism=‘flash_attention_2’ throws an unexpected keyword parameter: check_device_map error.

This seems to be because Barkmodel overrides the modelling_utils.py function called ‘check_and_enable_flash_attention’ and leaves out a parameter.

I have submitted a PR with the following fix: is this acceptable?

github.com/huggingface/transformers

Bark model Flash Attention 2 Enabling to pass on check_device_map parameter to super()

huggingface:main ← damithsenanayake:flash_attn_dev_mp_check

opened 12:33AM - 29 Feb 24 UTC

damithsenanayake

+2 -1

# What does this PR do? when using flash_attention_2 with bark model, the _chec…k_and_enable_flash_attn_2() method receives a 'check_device_map' paramter from modeling_utils.py. However it's not passed on to the super and fails execution. The fix simply grabs it from the caller and passes it on to the super. Fixes # 29332 ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). - [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section? - [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. - [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation). - [ ] Did you write any new necessary tests? ## Who can review? Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR. @sanchit-gandhi

Topic		Replies	Views
Instantiating models will not terminate Community Calls	0	674	April 6, 2023
A few questions about beginning with Huggingface Beginners	1	496	July 18, 2022
Load_in_8bit requires device_map but also does not support it 🤗Transformers	0	2701	December 19, 2022
Enabling Flash Attention 2 🤗Transformers	2	484	July 3, 2024
Great Code Cleanup Community Calls	0	1285	March 11, 2022

Barkmodel not intialising with flash_attention_2

Related Topics