RoFormer (Eng language)

John6666 · May 17, 2025, 12:08pm

Hmm… I don’t know anything about it, but it seems like there aren’t many English versions of ReFormer…

Adding support for scaling rotary position embeddings

opened 12:11AM - 26 Jun 23 UTC

closed 09:01AM - 14 Jul 23 UTC

### Feature request Hello, I would like if possible for Rotary Position Em…bedding scaling factors to be usable in the library. Currently this can only be done by monkey-patching the library. Namely, it requires modifying the: - `max_position_embeddings`: This can already be done via the model's config class or `config.json` - `position_scale`: This variable doesn't exist currently, and there is no way to incorporate this effect at the moment without monkey-patching the existing `LlamaRotaryEmbeddings` class. (I'd also like to not step over toes of a possible future XPos implementation which also uses it's own scale for different purposes) ### Motivation Recently I demonstrated it is possible to drastically reduce training compute when fine-tuning pre-trained RoPE models with an adjusted scaling factor for the purpose of extending the context length of the model. This has the effect of interpolating the position embeddings making it easier to fine-tune the model using in-distribution positions as opposed to out-of-distribution positions typically used via pure extrapolation. There is an extended write-up with motivations here https://kaiokendev.github.io/context as well as the code I used (for the 8K example) can be found here https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test/blob/main/llama_rope_scaled_monkey_patch.py Some existing discussions and benchmarks can be found here: https://github.com/ggerganov/llama.cpp/discussions/1965 Several models currently use this scaling feature, but they will not produce coherent output unless the scale is applied correctly during inference (scale is a hyperparameter): - https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test - https://huggingface.co/Peeepy/Airoboros-13b-SuperHOT-8k - https://huggingface.co/emozilla/open_llama_7b-scaled EDIT: Meta has recently written a paper about it: https://arxiv.org/abs/2306.15595 ### Your contribution I would love to help in any way possible. While the basic implementation would be easy, I'm not sure what the best way could be for adding this modification (such as if users want to used a fixed scale versus having it dynamically applied based on the input sequence length)

Topic		Replies	Views
SpanBERT, ELECTRA, MARGE from scratch? Beginners	5	1379	July 22, 2023
Suggestions for hugging face transformer models for Code and Formal Languages Intermediate	2	1754	May 3, 2022
Reformer for Sequence Classification Models	0	462	June 21, 2022
Saving underlying language model after trained on downstream task 🤗Transformers	0	420	September 14, 2020
Embedding Model Fine Tuning Spaces	2	31	May 8, 2025

RoFormer (Eng language)

Related topics