How to load pretrained models in Wav2Vec2PreTrainedModel

In method Wav2Vec2PreTrainedModel we can load the parameters by using
self.wav2vec2 = Wav2Vec2Model(config)
in class Wav2Vec2ForCTC,
if I want to load another pretrained model such as hubert,
I failed to use
HubertModel.from_pretrained(‘facebook/wav2vec2-xls-r-300m’)
It seems that the parameter did not loaded because the wer remains 1.