Thanks for taking the time out of your day to read this,
I had integrated meta-llama/Meta-Llama-3-8B-Instruct, on to my pc, which went perfect. I had decided I had wanted to integrate a streamlit ui, to make it easier to access this. However I came upon a plethora of issues, mainly regarding the quantization of such. It stated I didn’t have enough ram, I was wondering if there are any settings I can alter within my code to make this possible, or if i should just use a different model. I do understand that Llama 3 is a huge model, but does streamlit push the llama model too much where it cant even make a ui?
Here are my quantization settings using bnb config: bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type=‘nf4’,
bnb_4bit_compute_dtype=torch.bfloat16
)
and here is my text generator class:
text_generator = pipeline(
task=“text-generation”,
model=model,
tokenizer=tokenizer,
max_new_tokens=64, *note this was originally 128, but i decreased this
)
I had tried implementing loading in 8 bit fp 32, but it seems that the llama model doesn’t support that.
As I end this post, I have a 2060 super with 8 dedicated gpu memory, any tips would be helpful.
Thankful for your time!