Number of parameters reduced after loading in 4bit

wally1002 · August 9, 2023, 1:26pm

Why is there a decrease in no of parameters in 4bit?

saul95 · September 4, 2023, 4:10pm

When we load in 4 bit, the linear layers are replaced with linear 4bit layers. These layers have half the number of parameters. But still I am also not clear how number of parameters become half.

Ahatsham · November 9, 2023, 5:53pm

In the source code, when picking 4-bit, the parameter count is divided by 2.

I don’t know why, you can check the code

github.com

artidoro/qlora/blob/7f4e95a68dc076bea9b3a413d2b512eca6d004e5/qlora.py#L408C1-L423C6


      
          def print_trainable_parameters(args, model):
              """
              Prints the number of trainable parameters in the model.
              """
              trainable_params = 0
              all_param = 0
              for _, param in model.named_parameters():
                  all_param += param.numel()
                  if param.requires_grad:
                      trainable_params += param.numel()
              if args.bits == 4: trainable_params /= 2
              print(
                  f"trainable params: {trainable_params} || "
                  f"all params: {all_param} || "
                  f"trainable: {100 * trainable_params / all_param}"
              )

NPap · November 29, 2023, 7:58am

I still don’t have a clear answer for this and I would love to know, bumping for visibility.

purnasai · December 10, 2023, 5:59am

I too has the same doubt. @sgugger @sayakpaul sorry to tag you guys… I am thinking I could use your help here.

Topic		Replies	Views
Difference in Number of Parameters for load_in_4bit Beginners	0	414	August 2, 2023
Parameter Count & Shape Discrepancies in 4-bit vs. Higher bit LLM models 🤗Transformers	0	215	January 17, 2024
Get number of parameters for different parts of a model Beginners	0	4390	May 10, 2021
Does loading in 4bit override an 8bit model? 🤗Transformers	0	428	October 20, 2023
Peft model from pretrained load in 8/4 bit 🤗Transformers	6	11001	October 12, 2023

Number of parameters reduced after loading in 4bit

Related Topics