Modifying architecture of the models provided by the library

Are the models provided here easily modifiable to implement custom changes to the architecture? I plan to: pretrain a GPT model on my native language → add/modify layers but keep trained parameters - > finetune the model.

I am adding/modifying intermediate layers and not adding layers after/before the model.

Is it fine to modify the code and it will not break anything?