According to the hugging face documentation here It says please use the official repository for instructions on how to train from scratch:
IDEFICS modeling code in Transformers is for finetuning and inferencing the pre-trained IDEFICS models.
To train a new IDEFICS model from scratch use the m4 codebase (a link will be provided once it’s made public)
However this is the link that is provided :
https://huggingface.co/docs/transformers/main/en/model_doc/INSERT%20LINK%20TO%20GITHUB%20REPO%20HERE
I’m I just am writing to see if there are any plans on providing this code? I’m having a lot of trouble initializing one of these from scratch with custom settings or using other fine tuned models in combination with them.