Load/save HF block sparse model

Research
1

Hey everyone,

I am exploring https://github.com/huggingface/pytorch_block_sparse project. One of the issues that popped up almost immediatelly is loading a saved “sparsified” model. So, let’s say you sparsified Roberta using an example provided . Now that the model has been sparsified (it’s linear layers replaced with BlockSparseLinear nn modules) how can I load previously saved model using HF ecosystem? All I can think of is that I again need to create a Roberta model with uninitialized weights, sparsify it, and the load weights with model.load_state_dict(torch.load(PATH))?

Am I overlooking something obvious?

2

No mechanism in place for loading as of now, which is ok. I sparsed the model again and loaded the weights manually via model.load_state_dict(torch.load(PATH)).