How to get GPTJ 8bit working in a Jupiter notebook?

Beginners
1

Hi everyone.

I’m trying to get this model:
https://huggingface.co/hivemind/gpt-j-6B-8bit

working in a notebook on paper space. It seems when I try this 8bit model with the following

tokenizer = AutoTokenizer.from_pretrained("hivemind/gpt-j-6B-8bit")

the tokenizer line crashes the cell with an error saying that the tokenizer for this cannot be found.

when I try to load the model with the following. The first time I run it it downloads the model and moves on to the tokenizer line, but the second time I run the cell this line seems to crash the cell and I don’t get any output after this line.

model = AutoModelForCausalLM.from_pretrained("hivemind/gpt-j-6B-8bit")

I’m not sure what other trouble shooting steps I should try? I can load the 16bit model just fine by following the docs, but the 8 bit model does not work.

Thanks.

2

Hey !
You can now load and use 8-bit models easily on transformers! Check for example this demo: Google Colab that uses 8bit on T5