"normal_kernel_cpu" not implemented for 'Char' when trying to import 8-bit model

🤗Transformers
1

Didn’t see this issue anywhere else while googling, so posting here; tried loading an 8-bit SFTT model as part of a TRL-based reinforcement learning pipeline. Returned the error as stated in the title:

RuntimeError: "normal_kernel_cpu" not implemented for 'Char'

Model config is as follows, with the loading line throwing the error:

config = PPOConfig(
    model_name="drive/MyDrive/fg_proj/lr_pipeline/sftt_models/feb7_2024_8bit",
    learning_rate=1.41e-5,
    log_with="tensorboard",#"wandb",
    project_kwargs={"logging_dir": "run_logs_107_3"}
)

model = AutoModelForCausalLMWithValueHead.from_pretrained(config.model_name)

My hunch is this related to loading in 8-bit, since the full-bit and 4-bit models I tried did not throw this error. Any ideas as to what I’d be missing for this to occur would be helpful.

2

Received similar error. when i try to load the 8 bit quantized model from local system.

I tried it for owl-vit model.

3

I receive the same error, also when loading in with 8-bit but not when loading in with 4-bit…

4

I have this problem too

5

Same problem. Someone please help

6

I am also having this problem trying to load longformer with a bitsandbytes config.