ImportError: cannot import name '_expand_mask' from 'transformers.models.bloom.modeling_bloom'

🀗Transformers
1

If I follow the same example provided on the model card: openflamingo/OpenFlamingo-9B-vitl-mpt7b · Hugging Face

I am getting the following error:

ImportError: cannot import name '_expand_mask' from 'transformers.models.bloom.modeling_bloom' (/usr/local/lib/python3.10/dist-packages/transformers/models/bloom/modeling_bloom.py)

For reproducibility: Google Colab

1 Like
2

have you solved this problem? i have the same question too.

1 Like
3

Copy the following function to file (.\Lib\site-packages\transformers\models\clip\modeling_clip.py) can solve the problem.

def _expand_mask(
    mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None, past_key_values_length: int = 0
):
    """
    Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
    """
    bsz, src_len = mask.size()
    tgt_len = tgt_len if tgt_len is not None else src_len

    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)

    if past_key_values_length > 0:
        # concat fully attendend attention_mask to the beginning if `past_key_values` are used
        expanded_mask = torch.cat(
            [
                torch.ones(bsz, 1, tgt_len, past_key_values_length, device=expanded_mask.device, dtype=dtype),
                expanded_mask,
            ],
            dim=-1,
        )

    inverted_mask = 1.0 - expanded_mask

    return inverted_mask.masked_fill(inverted_mask.bool(), torch.finfo(dtype).min)
1 Like