Format of data during pre-training

What should be the format of the data for pre-training? could it be any raw data (e.g., news articles) in my case and then after I fine-tune, then I need to define it for a specific task e.g., classification?

It comes in many forms and usually needs processing/adaptation for model input. Look into transformers examples to start.

If you are asking about the datasets project specifically - it makes it even simpler to do the above. There are many prepared datasets ready to go. Read through documentation.

1 Like