How to finetune BLOOM for classification?

Models
1

I’m trying to finetune BLOOM-350m for a simple classification task. It seems right now this is impossible? Here is the error I’m getting:

Blockquote
File “/usr/local/google/home/windmaple/miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py”, line 448, in from_pretrained
f"Unrecognized configuration class {config.class} for this kind of AutoModel: {cls.name}.\n"
ValueError: Unrecognized configuration class <class ‘transformers.models.bloom.configuration_bloom.BloomConfig’> for this kind of AutoModel: TFAutoModelForSequenceClassification.
Model type should be one of AlbertConfig, BertConfig, CamembertConfig, ConvBertConfig, CTRLConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, ElectraConfig, FlaubertConfig, FunnelConfig, GPT2Config, GPTJConfig, LayoutLMConfig, LongformerConfig, MobileBertConfig, MPNetConfig, OpenAIGPTConfig, RemBertConfig, RobertaConfig, RoFormerConfig, TapasConfig, TransfoXLConfig, XLMConfig, XLMRobertaConfig, XLNetConfig.

Given that BLOOM is so new, I’m curious if this is being worked on?

2

I am also trying to fine-tune Bloom. The below works for me:

tokenizer = AutoTokenizer.from_pretrained(“bigscience/bloom-350m”)
model = AutoModel.from_pretrained(“bigscience/bloom-350m”, num_labels=1)

However, I get the below error in the forward function of my model when trying to use the transformer:

transformer_out = self.transformer(ids, mask)

667 past_key_values_length = 0
668 if past_key_values[0] is not None:
→ 669 past_key_values_length = past_key_values[0][0].shape[1]
670 current_sequence_length += past_key_values_length
671

IndexError: tuple index out of range

I am unable to figure out if I am doing something wrong, but the code works for other transformers like Deberta and Roberta-Large.

If you are able to get yours to work please post again.

3

Did you update the version to the latest? I can run inference just fine.