How do I load an SFTTrainer model finetuned falcon-7b-sharded-bf16 using custom dataset, and make prediction with it

Beginners
1

How do I load a saved SFTTrainer model after uploading it to HuggingFace and how do I make a prediction with the model?

The model was trained using Colab notebook to fine-tune Falcon-7B on Guanaco dataset using 4bit and PEFT

It was trained on a custom dataset similar to the Guanaco dataset.

How do I use the saved trained model that was uploaded to HuggingFace?

HUGGING_FACE_USER_NAME = "username" 
from huggingface_hub import notebook_login 
notebook_login()

After a successful login, I pushed to HuggingFace. model_name = “model_name”

trainer.model.push_to_hub(f"{HUGGING_FACE_USER_NAME}/{model_name}",use_auth_token=True)

Now I want to load and Make Inference, Can you help out with that because I am getting an error here?

from peft import LoraConfig
 model_name = "username/model_name" lora_config = LoraConfig.from_pretrained(model_name)
 '---------------------------------------------' 

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    trust_remote_code=True
)
model.config.use_cache = False

When I run the above code, I am getting an error.

OSError: "username/model_name" does not appear to have a file named config.json. Checkout 'https://huggingface.co/username/model_name/main' for available files.
1 Like
2

chage the username/model_name to a specific model path

3

Try:
model.config.to_json_file(“config.json”)

the file is saved, but then I get this other error:

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory…