Push 4-bit converted model to hub

ckandemir · October 27, 2023, 7:28am

Following PRs do handle this issue^

github.com/TimDettmers/bitsandbytes

Save and load in NF4 / FP4 formats

TimDettmers:main ← poedator:save4

opened 08:29PM - 07 Sep 23 UTC

poedator

+353 -96

Purpose: enable saving and loading transformers models in 4bit formats. Enables… this PR in transformers: https://github.com/huggingface/transformers/pull/26037 addresses feature request #603 and other similar ones elsewhere. tested with Bloom-560 and Llama-2-7b. tested quantization, saving, loading, -- matched tensors and quant_states -- matched inference results. tested both nf4 and fp4 test added

github.com/huggingface/transformers

[bnb] Let's make serialization of 4bit models possible

huggingface:main ← poedator:save4

opened 07:37PM - 07 Sep 23 UTC

poedator

+194 -67

## What does this PR do? Purpose: enable saving and loading transformers models… in 4bit formats. tested with Bloom-560 and Llama-2-7b. Save - Load - match tensors and quant_states - match inference results. ## connection with bitsandbytes Requires this PR in bitsandbytes: https://github.com/TimDettmers/bitsandbytes/pull/753 to be able to save/load models ## testing: the functionality was tested doing this series of commands: ``` model = m4 = transformers.AutoModelForCausalLM.from_pretrained(..., [with 4bit quant config]) model.save_pretrained(SAVE_PATH, safe_serialization= [False / True] ) model2 = transformers.AutoModelForCausalLM.from_pretrained( SAVE_PATH) # then matching all params and quant_state items between the models, matching inference results ``` Specific tests will be added to this PR once the [bitsandbytes PR](https://github.com/TimDettmers/bitsandbytes/pull/753) merges. ## Open questions to the maintainers: 1. I added and tested code necessary for straightforward save/load operations with LLMs. Yet there may be other kinds of function calls that may need to be updated to handle 4bit save/load - pls suggest if/where to expand this PR. specifically: this PR covers `load_state_dict_into_meta_model()` but not `load_state_dict_into_model()` for I did not find an example that uses it. 2. Some of my edits may not fit with style / refactoring plans of the maintainers - pls give guidance if needed.

I will just wait until they are merged

Topic		Replies	Views
Pushing a quantized (4bit) model on the Hub 🤗Transformers	9	4255	January 8, 2024
Can I load a model fine-tuned with LoRA 4-bit quantization as an 8-bit model? 🤗Hub	0	291	November 27, 2023
Problem with pushing quantized model to hub 🤗Transformers	3	315	October 14, 2024
An error i ve been trying to fix for days now Intermediate	4	490	November 19, 2024
How can I load an LLM in 4-bits 🤗Transformers	0	487	August 2, 2023

Push 4-bit converted model to hub

Related topics