I was trying to instruction fine-tune StarCoder model with a custom question answer data set. However, I am not clear what AutoModel I should use for this. When I tried using AutoModelForQuestionAnswering, I am getting the error that Unrecognised configuration class. Here is the code and the error it produced.
model = AutoModelForQuestionAnswering.from_pretrained(
‘bigcode/starcoder’,
use_cache=True
)
ValueError: Unrecognized configuration class <class ‘transformers.models.gpt_bigcode.configuration_gpt_bigcode.GPTBigCodeConfig’> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BloomConfig, CamembertConfig, CanineConfig, ConvBertConfig, Data2VecTextConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, ElectraConfig, ErnieConfig, ErnieMConfig, FlaubertConfig, FNetConfig, FunnelConfig, GPT2Config, GPTNeoConfig, GPTNeoXConfig, GPTJConfig, IBertConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LongformerConfig, LukeConfig, LxmertConfig, MarkupLMConfig, MBartConfig, MegaConfig, MegatronBertConfig, MobileBertConfig, MPNetConfig, MvpConfig, NezhaConfig, NystromformerConfig, OPTConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, SplinterConfig, SqueezeBertConfig, XLMConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig
Which AutoModel should I use with StarCoder for instruction fine-tuning using a question answer dataset such as ta-prompt dataset of SQuAD dataset? Thanks.