Fine-Tune GPT2 for English Rap

GPT2 for English Rap

For this project, I propose to use a pretrained GPT2 model in English and fine-tune it to learn to English rap.


A GPT2 model, pretrained in English can be found here: gpt2 · Hugging Face


The model can be fine-tuned on a variety of publicly available lyrics of rap songs. The team can choose whatever lyrics they like.

Available training scripts

A training script to fine-tune a GPT2 model in Flax is available here

(Optional) Desired project outcome

The desired project output is a GPT2 model that can drop bars. Ideally the model is able to generate a whole rap song.

(Optional) Challenges

It might be difficult to find enough data and to homogenize it well.

(Optional) Links to read upon

There are some nice articles on GPT2 models that can rap:

1 Like

This is very interesting! I would like to work on it.

This project seems very interesting to me, im a big fan of rap music as is, the biggest challenge i think will be creating an appropriate dataset- currently i know of the metro songs dataset which has 300k songs (i have already used this dataset in a project where we did genre classification from lyrics). Also there exists an API on which can be used to build a dataset (the person in the 2nd link uses it to create its dataset, it’s interesting because it doesn’t only contain lyrics but also the structure of the song [Verse 1:] and [Chrous], stuff like that)

also this: Kanye West Rap Verses | Kaggle . Current lyric generators from this dataset that i’ve have only used markov chains and LSTMs, i think it would be interesting to see how gpt2 handles this dataset which is not small (364 verses). We can also try reconstructing the kanye dataset with the genius API to include song structure information. Or perhaps create other “fun” datasets, 1 maybe with all artists containing lil in their name, or maybe geographically different artists, NYC/LA/ATL
i hope im not too late in my application, i think this would be a very fun and interesting project to work on, i have no problem doing it alone. Honestly im surprised that only one other person responded to this thread, i thought it would be interesting to a lot of people :slight_smile:


Awesome 2 is enough - finalizing this project :slight_smile:

hello @Cropinky are you on discord? I would like to contact you

1 Like

I know I am late, I am interested in this project. is it possible to add my name to this project?

1 Like

adding you!

Thank You !!!

Team , please let me know the plan of action.