How pretrained models are trained?


How pre-trained are obtained?
In the library there is always some pre-trained model downloaded from the server, but how is it originally trained? Is it also trained using transformers library?

In most if not all built-in cases (e.g. bert-base-cased), the original paper implementations are ported to the transformers architecture. (you can have a look at conversion scripts, e.g.,

User models (e.g. username/mymodel-uncased) may have been trained in other ways or ported to the transformers architecture manually, with custom scripts, or they are trained by using the transformers library directly.

Thanks for reply.
How can I train my own model from scratch using transformers?

You can have a look here: