The confusing parts of phi-3's implementation

ryan-minato · June 17, 2024, 8:39am

As far as I know, phi-3’s Phi3SuScaledRotaryEmbedding is an implementation of LongRoPE. However, several aspects of the huggingface transformers’s implementation leave me perplexed.

Firstly, there’s the determination of the scale. Here, it calculates constants from two configurations, seemingly leading to only one branch being entered.

github.com

huggingface/transformers/blob/7ae4fc271d2f0f3cc950b74cdbd9f2af810bea52/src/transformers/models/phi3/modeling_phi3.py#L153-L157


      
          scale = self.max_position_embeddings / self.original_max_position_embeddings
          if scale <= 1.0:
              scaling_factor = 1.0
          else:
              scaling_factor = math.sqrt(1 + math.log(scale) / math.log(self.original_max_position_embeddings))

If I’m not mistaken, this should implement the non-interpolated starting tokens part in LongRoPE.

Additionally, this piece of code seems to clearly disregard coding standards. The uninitialized self.inv_freq is defined, and it appears that self.inv_freq could be initialized as self.long_inv_freq and self.short_inv_freq during the initialization phase.

github.com

huggingface/transformers/blob/7ae4fc271d2f0f3cc950b74cdbd9f2af810bea52/src/transformers/models/phi3/modeling_phi3.py#L133-L140


      
          seq_len = torch.max(position_ids) + 1
          if seq_len > self.original_max_position_embeddings:
              ext_factors = torch.tensor(self.long_factor, dtype=torch.float32, device=x.device)
          else:
              ext_factors = torch.tensor(self.short_factor, dtype=torch.float32, device=x.device)
          
          inv_freq_shape = torch.arange(0, self.dim, 2, dtype=torch.int64, device=x.device).float() / self.dim
          self.inv_freq = 1.0 / (ext_factors * self.base**inv_freq_shape)

Does anyone understand the deeper purpose of these implementations, or is this simply a bug?

ryan-minato · June 17, 2024, 8:40am

Source:

As far as I know, phi-3’s Phi3SuScaledRotaryEmbedding is an implementation of LongRoPE(source).

If I’m not mistaken, this should implement the non-interpolated starting tokens part in LongRoPE.

Topic		Replies	Views
How to convert sentence-transformers/msmarco-distilbert-base-tas-b model to torchscript 🤗Transformers	0	35	October 30, 2024
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! Beginners	0	184	July 3, 2024
Huggingface classification struggling with prediction 🤗Transformers	0	826	April 5, 2022
Discrepancy in Model Inference: Local vs. Hugging Face Model Hub 🤗Transformers	1	807	December 27, 2023
Transformer/Pipelines Tutorial - where is it running? Beginners	3	1908	December 4, 2023

The confusing parts of phi-3's implementation

Related topics