bf16=True in TrainingArgument

gptzerozero · July 2, 2023, 4:15pm

Is the following code in the official Qlora repo

artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L334-L343


      
          for name, module in model.named_modules():
              if isinstance(module, LoraLayer):
                  if args.bf16:
                      module = module.to(torch.bfloat16)
              if 'norm' in name:
                  module = module.to(torch.float32)
              if 'lm_head' in name or 'embed_tokens' in name:
                  if hasattr(module, 'weight'):
                      if args.bf16 and module.weight.dtype == torch.float32:
                          module = module.to(torch.bfloat16)

similar to passing bf16=True into transformers.TrainingArgument when using transformers.TrainingArgument with transformers.Trainer?

Topic		Replies	Views
Using Quantization with fp16/bf16 Trainer flag 🤗Transformers	0	645	February 14, 2024
How to generate using a fine-tuned qlora cast to bfloat16 Beginners	1	1176	April 6, 2024
Qlora Training with Custom Trainer Intermediate	0	67	September 19, 2024
Training Arguments to do pure bf16 training? 🤗Transformers	0	1760	December 20, 2023
The role of the bf16 arguments in SFTConfig 🤗Transformers	0	300	July 25, 2024

bf16=True in TrainingArgument

Related topics