Error saving quantized model

Intermediate
1

I get the following error trying to save a quantized model. Can anyone help? Thanks.

quantize model

quantized_model = torch.quantization.quantize_dynamic(
model, {torch.nn.Linear}, dtype=torch.qint8
)
quantized_model.save_pretrained(quantized_dir)

AttributeError Traceback (most recent call last)
in ()
3 model, {torch.nn.Linear}, dtype=torch.qint8
4 )
----> 5 quantized_model.save_pretrained(quantized_dir)

1 frames
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in shard_checkpoint(state_dict, max_shard_size)
291
292 for key, weight in state_dict.items():
→ 293 weight_size = weight.numel() * dtype_byte_size(weight.dtype)
294
295 # If this weight is going to tip up over the maximal size, we split.

AttributeError: ‘torch.dtype’ object has no attribute ‘numel’

1 Like
2

I found the solution here

3

How did this issue got resolved? Am still getting the same error.

4

Im getting the same error. Im trying to save a quantized model using the following line torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.float16).

When running quantized_model.save_pretrained(path_dir), I’m running into the error AttributeError: ‘torch.dtype’ object has no attribute ‘numel’.

Anyone have a solution?

1 Like
5

I was actually able to resolve the error from the Pegasus Model Weights Compression/Pruning post above. Specifically, I was using Pytorch version 1.11.0 and HuggingFace version 4.20.1.