Question about RecurrentGemma intermediate_size

goodjw · May 3, 2024, 5:08am

Hello,

I have a question regarding the intermediate_size parameter in the RecurrentGemmaConfig. I noticed in the source code that intermediate_size is divided by 2.

Here’s the snippet of the code:

github.com

huggingface/transformers/blob/a0e77a1f6bdfcceccdc5618e8a01ee32ef47bfa8/src/transformers/models/recurrent_gemma/modeling_recurrent_gemma.py#L470-L473


      
          self.intermediate_size = config.intermediate_size // 2
          self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=True)
          self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=True)
          self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=True)

I observed that intermediate_size seems to correspond to mlp_expanded_width in RecurrentGemma’s official implementation. In the official 2B model, the model width is 2560 and the MLP expansion factor is 3, resulting in an mlp_expanded_width of 7680. However, in the Hugging Face implementation, the default intermediate_size is 15360.

Given that dividing by 2 yields 7680, which aligns with the official implementation, I’m curious why the division is necessary. Would it not be simpler and clearer to set intermediate_size directly to 7680, thus avoiding the division?

Thank you for your time and assistance.

Topic		Replies	Views
No option for embedding size in transformers.RobertaConfig Beginners	1	172	January 23, 2023
Making a model slightly bigger Beginners	0	63	April 27, 2024
Huggingface Transformers; Polyglot-12.8b (GPT-Neox); You may consider adding ignore_mismatched_sizes=True in the model from_pretrained Beginners	0	598	August 21, 2023
LLama2 Finetuning giving RuntimeError: mat1 and mat2 shapes cannot be multiplied (33x4096 and 1x8388608) Models	0	413	November 17, 2023
Transformers module - parameter count and size 🤗Transformers	0	462	January 12, 2024

Question about RecurrentGemma intermediate_size

Related Topics