I checked out BharatGPT-3B-Indic running it in following ways:
- From my CPU laptop and online model
- From my laptop model saved in my local file system.
- From collab and model called from huggingface ---- this worked abliet slowly with delay
I used a test script with Gradio UI code had to turn bits and bytes off.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import gradio as gr
import huggingface_hub
huggingface_hub.login(‘huggingface access code’')
print(huggingface_hub.whoami())
model_id = “CoRover/BharatGPT-3B-Indic”
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map=“auto”,
load_in_8bit=False, # Use bitsandbytes for memory-efficient loading
)
pipe = pipeline(
“text-generation”,
model=model,
tokenizer=tokenizer
)
def generate_response(message):
messages = [
{“role”: “system”, “content”: “You are a helpful assistant who responds in Hindi or English.”},
{“role”: “user”, “content”: message},
]
output = pipe(messages, max_new_tokens=256)
return output[0][“generated_text”]
gr.Interface(
fn=generate_response,
inputs=“text”,
outputs=“text”,
title=“Chat with BharatGPT-3B-Indic”,
description=“Runs locally using 8-bit inference”
).launch()
I got the following chat responses
input : please translate following quoted sentence in gujarati “I am english speaking”
[{‘role’: ‘system’, ‘content’: ‘You are a helpful assistant who responds in Hindi or English.’}, {‘role’: ‘user’, ‘content’: ‘please translate following quoted sentence in gujarati “I am english speaking”\n’}, {‘role’: ‘assistant’, ‘content’: ‘મારું અંગ્રેજી બોલવાનું છે.’}]
please translate following quoted sentence in gujarati “I am a gujarati”
[{‘role’: ‘system’, ‘content’: ‘You are a helpful assistant who responds in Hindi or English.’}, {‘role’: ‘user’, ‘content’: ‘please translate following quoted sentence in gujarati “I am a gujarati”\n’}, {‘role’: ‘assistant’, ‘content’: ‘મારું નામ ગુજરાતી છે.’}]
Analysis
“I am a Gujarati”
Response: મારું નામ ગુજરાતી છે.
Analysis:
- Translation: “My name is Gujarati.” — which is incorrect.
- Correct translation:
હું ગુજરાતી છું.
(This means “I am Gujarati.”)
Incorrect translation why this may ne happening:
- The model may not have been instructed explicitly enough to translate into Gujarati system message limits it to respond in Hindi or English. This can bias the output.
- BharatGPT, while trained on multiple Indian languages, may not have strong enough grounding in Gujarati, or its chat template may not fully support instruction-following for translation.
- The “quoted sentence” format I have used in the input may be adding confusion.
I am going to add Gujarati translation to system prompt and test. If anyone has tried or has a testcase code would appreciate an input.
Other aspects
I was thinking of Converting the model to GGUF Format run it with llama.cppto see if i can now run locally without GPU. And works well even on low RAM (4–6GB) with quantized GGUF without Python/transformers overhead.
Thank you
Rashmikant Dave
