Issue loading quantised model. I get the error KeyError: ‘inv_freq’. What is causing this? Any help is appreciated.
Quantization definition
bnb_config = BitsAndBytesConfig(
load_in_4bit = True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type=“nf4”,
bnb_4bit_compute_dtype=torch.bfloat16,
)
load the qunatized model
esm_model_name = “facebook/esm2_t30_150M_UR50D”
model = AutoModelForTokenClassification.from_pretrained(esm_model_name, quantization_config = bnb_config,trust_remote_code=True,device_map=‘cuda’)
model