Double Headed Bloom for Dialogue Tasks

[AutomatonBloke]

I’m trying to replicate this article using bloom as the model to fine-tune instead of a GPT model:

🦄 How to build a State-of-the-Art Conversational AI with Transfer-Learning

However, bloom doesn’t seem to have a double head framework available.

So I figured I’d need to create a custom head on top of the BloomForCausalLM, and the closest I’ve found to this is this thread and article:

How do I change the classification head of a model?

Thank you! This fixed my problem too! It was weird that this didn’t work: AutoModelForSequenceClassification.from_pretrained(“huggingface/CodeBERTa-language-id”, num_labels=15) but this did: config = AutoConfig.from_pretrained(“huggingface/CodeBERTa-language-id”) config.num_labels = 15 model = AutoModelForSequenceClassification.from_config(config) Answering to tolgayan, the point is that this gets trained, you’re fine tuning the model.

Adding Custom Layers on Top of a Hugging Face Model

I understand that blenderbot and dialoGPT are available models, but from what I’ve seen and tested they’re not the most impressive. I’d like to see what I can do on a custom data set of my own design.

I think what I’m doing should be feasible, but I’m entirely self-taught in programming → machine learning → deep learning, and my method is usually to start with a goal, then figure out how to get there. If someone could at minimum tell me if what I’m doing isn’t even possible, that would save me time continuously trying to figure it out.

Any other help is welcome from learning materials to guidance on this project.

Thanks!