I think many people initialize with an empty weight or use from_pretrained to bring in the initial values of the base model.