Qwen 'padding_side = right' problem

Models
1

You are attempting to perform batched generation with padding_side=‘right’ this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to call tokenizer.padding_side = 'left' before tokenizing the input
this is the problem although it set to left

Temp Solution(if “tokenizer.padding_side = ‘left’” didn’t work ):
=============> find ‘transformers/models/qwen2/modeling_qwen2.py’
and change line ~622 or comment

    if self.config._attn_implementation == "flash_attention_2":
        if attention_mask is not None and past_key_values is not None:
            is_padding_right = attention_mask[:, -1].sum().item() != input_tensor.size()[0]
            if is_padding_right:
                print('I dont gaf')
                continue
                # raise ValueError(
                #     "You are attempting to perform batched generation with padding_side='right'"
                #     " this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to "
                #     " call `tokenizer.padding_side  = 'left'` before tokenizing the input. "
                # )
        if attention_mask is not None and 0.0 in attention_mask:
            return attention_mask
        return None
1 Like
2

Oh…

You can set the padding side when loading the processor as AutoProcessor.from_pretrained(model_id, padding_side="left"). We usually recommend to make sure padding side is on the correct side, before generating :slight_smile:
We might need to update the model docs, if example code for batch generation failed with the same error. Feel free to open a PR if so

3

it works on VL(Text2Image) version not regular version of Qwen

1 Like