How BitsAndBytesConfig use bnb_4bit_compute_dtype=torch.bfloat16 on gpu don't support torch.bfloat16

mohamedemam October 20, 2023, 11:05pm 1

is it convert to float16 when find doesn’t support bfloat16

Topic		Replies	Views
Changing bnb_4bit_compute_dtype Beginners	0	100	July 18, 2024
Low bf16 performance on TPU, int4 vs int8 quantizatoin 🤗Accelerate	0	328	June 1, 2024
Understanding how changing bnb_4bit_compute_dtype affects outputs 🤗Transformers	1	4333	February 10, 2024
Confused with setting up torch_dtype while using CPU as device 🤗Transformers	0	2188	October 12, 2022
Loading in Float32 vs Float16 has very different speed 🤗Transformers	1	63	February 20, 2025