How BitsAndBytesConfig use bnb_4bit_compute_dtype=torch.bfloat16 on gpu don't support torch.bfloat16

mohamedemam October 20, 2023, 11:05pm 1

is it convert to float16 when find doesn’t support bfloat16

Topic		Replies	Views
Changing bnb_4bit_compute_dtype Beginners	0	143	July 18, 2024
Low bf16 performance on TPU, int4 vs int8 quantizatoin 🤗Accelerate	0	370	June 1, 2024
Confused with setting up torch_dtype while using CPU as device 🤗Transformers	0	2320	October 12, 2022
Fp16, bf16 in TrainingArgs vs BitsAndBytesConfig Beginners	0	801	June 30, 2023
Mixed precision for bfloat16-pretrained models 🤗Transformers	2	12500	April 21, 2021