How to load pretrained models in Wav2Vec2PreTrainedModel

In method Wav2Vec2PreTrainedModel we can load the parameters by using
self.wav2vec2 = Wav2Vec2Model(config)
in class Wav2Vec2ForCTC,
if I want to load another pretrained model such as hubert,
I failed to use
It seems that the parameter did not loaded because the wer remains 1.