Does autogpt-q require float16?

jmhessel · August 28, 2023, 7:17pm

Hi! I’ve been excited about the recent quantization integration of auto-gptq and appreciate the work folks did to make this happen. I was trying to play around a bit, and saw this line

github.com

huggingface/transformers/blob/50573c648ae953dcc1b94d663651f07fb02268f4/src/transformers/modeling_utils.py#L2579-L2583


      
          logger.info(
              f"Overriding torch_dtype={torch_dtype} with `torch_dtype=torch.float16` due to "
              "requirements of `auto-gptq` to enable model quantization "
          )
          torch_dtype = torch.float16

which seems to imply that auto gptq only supports float16 vs. bfloat16… I uncommented this line and things seem to be working fine — do folks have thoughts about this? TheBloke seems to sometimes run quantization with bfloat16 ([BUG]CUDA OUT OF MEMORY · Issue #179 · PanQiWei/AutoGPTQ · GitHub) so I am not sure why this line is here, but could be missing something (e.g., maybe some of the CUDA kernels are float16 only)?

Topic		Replies	Views
Float16 on CPU torch support Beginners	0	337	January 16, 2024
Baffling performance issue on most NVidia GPUs with simple transformers + pytorch code Intermediate	5	3036	April 9, 2024
Issues running GPT-J-6B Beginners	1	995	January 31, 2023
Confused with setting up torch_dtype while using CPU as device 🤗Transformers	0	1581	October 12, 2022
On cpu, how to save memory when inferencing? 🤗Transformers	1	346	July 13, 2023

Does autogpt-q require float16?

Related Topics