The error I’m getting:
Traceback (most recent call last):
File "quantize_fp8.py", line 39, in <module>
model.save_quantized(quantized_model_dir)
File "/root/AutoFP8/auto_fp8/modeling.py", line 133, in save_quantized
save_quantized_model(
File "/root/AutoFP8/auto_fp8/quantize.py", line 342, in save_quantized_model
model.save_pretrained(save_dir)
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2661, in save_pretrained
shard = {tensor: state_dict[tensor] for tensor in tensors}
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2661, in <dictcomp>
shard = {tensor: state_dict[tensor] for tensor in tensors}
NameError: free variable 'state_dict' referenced before assignment in enclosing scope
This seems to be due to 2 conflicting changes from the last two weeks, which cause the loop over shards in save_pretrained to look like this:
# Save the model
for shard_file, tensors in state_dict_split.filename_to_tensors.items():
shard = {tensor: state_dict[tensor] for tensor in tensors} # state_dict is used here, but is deleted below
# remake shard with onloaded parameters if necessary
if module_map:
...
state_dict = {name: "" for name in shard} # overwrites state_dict
...
del state_dict # deletes state_dict
Screenshot of where the conflicting changes came from is attached.
