Making local LLM + TTS work for a Persian voice assistant on a GTX 1650 Ti (4GB VRAM) — looking for advice

The 7B/8B direction is reasonable, but I do not think it is the cheapest first experiment on this hardware.

For example, a Q4_K_M build of Qwen2.5-7B is around 4.68 GB. A Dorna-Llama3-8B Q4_K_M build is around 4.92 GB; the Dorna GGUF repo also has smaller Q3 variants around 3.66–4.02 GB.

Useful links:

• Qwen2.5-7B-Instruct
• Qwen2.5-7B-Instruct GGUF example
• Dorna-Llama3-8B-Instruct
• Dorna-Llama3-8B-Instruct GGUF

Those file sizes do not mean “it cannot run on 4 GB”. GGUF file size is not a one-to-one VRAM number, and llama.cpp/Ollama can partially offload layers to the GPU and leave the rest in system RAM.

But starting that close to or above the VRAM capacity leaves less headroom for the KV cache and runtime buffers, and it also makes each failed experiment more expensive over your connection.

That is why I would use a smaller model as a discriminator first.

The official Gemma 4 E2B Q4_0 repo contains:

gemma-4-E2B_q4_0-it.gguf      3.35 GB
gemma-4-E2B-it-mmproj.gguf    ~987 MB

For a text-only voice assistant you do not need the multimodal projector, so the first download can stay at the 3.35 GB text GGUF.

The model page also exposes a direct llama.cpp path:

winget install llama.cpp

llama serve ^
  -hf google/gemma-4-E2B-it-qat-q4_0-gguf:Q4_0 ^
  -c 2048

See Gemma 4 E2B IT QAT Q4_0.

I would not interpret “3.35 GB” as proof that it will fully reside in 4 GB VRAM. I would interpret it only as: this is a much cheaper first experiment than starting with a ~4.7–4.9 GB 7B/8B quant.

If E2B runs well but its Persian behavior is still not good enough, then I would compare another 4B-class model before increasing the parameter count:

Gemma 4 E2B
    |
    +-- Persian good + speed good
    |       -> integrate it
    |
    +-- Persian weak, runtime fit good
    |       -> compare Gemma 3 4B / Qwen3 4B
    |
    +-- GPU placement / latency bad
    |       -> reduce context / inspect offload
    |
    +-- 4B class consistently not good enough
            -> then try 7B/8B partial offload or keep a cloud fallback

Dorna is still interesting because it is Persian-oriented. I just would not let “Persian-tuned” override the deployment cost before checking whether a modern ~4B model is already sufficient.

I did some small qualitative tests on Colab/T4. These are not GTX 1650 Ti benchmarks, and the prompt sets are too small to call any model a winner. I mainly wanted to see whether the ~4B range was worth investigating at all.

An early fixed-prompt pass looked roughly like this:

Again, those T4 memory observations should not be copied over to a GTX 1650 Ti. The interesting result was simply that ~4B models were not obviously too weak to investigate.

I also did a same-base comparison between vanilla Gemma 3 4B and a Persian SFT of Gemma 3 4B. Runtime cost was effectively tied, but the Persian SFT did not show a convincing assistant-quality advantage in that small A/B.

So I would not assume:

Persian fine-tuned = automatically better Persian assistant

It may be better for a particular task or dataset, but I would test the exact assistant behavior you need.

Later I tested the official Gemma 4 E2B/E4B Q4_0 artifacts:

These are T4/llama.cpp observations, not expected numbers for your laptop.

E4B did not clearly dominate E2B in my ten-prompt set, so E2B became the more interesting target-machine test.

One reason I would keep the evaluation application-specific is that even the apparently stronger models made ordinary mistakes. For example:

08:20 + 1:45 = 10:05

Vanilla Gemma 3 4B returned 10:05 in that test, while the Persian SFT returned 10:55, and the tested Gemma 4 E2B/E4B runs returned 09:05.

That is not evidence that Gemma 3 is globally “better”; it is evidence that ten prompts are nowhere near enough to establish general quality.

For your use case I would make a tiny smoke set around the errors that actually matter:

- normal Persian conversation
- colloquial Persian
- one noisy / ASR-like sentence
- one clarification-required request
- one date/time/number question
- one "answer only in Persian" instruction

I would watch for:

language switching
repetition / loops
instruction-following failures
date/time arithmetic failures
register instability
claims that an action was executed when no tool exists

And record only:

model / quant / context
CPU/GPU placement
VRAM
TTFT
tok/s
Persian stability
obvious task failures

That gives you much more useful information than downloading several models and asking each one a couple of generic questions.

For TTS, I ended up testing more candidates than I originally expected:

Ava-82M
Mana-Persian-Piper
Gooya + Negara G2P
Pocket-TTS Farsi v2
MOSS-TTS-Nano-Persian

The two I would keep at the front of the queue are Ava and Mana.

Ava-82M

Ava-82M is a compact Persian model (~82M class; current repository ~329 MB) and, importantly for Persian, it includes more than just the acoustic model.

Its frontend currently includes:

number/date normalization
contextual Persian G2P
pronunciation overrides
Ezafe-related repair
long-text splitting

That is attractive because Persian TTS quality is not only an acoustic-model problem. If the frontend sends the wrong pronunciation, a good acoustic model will pronounce the wrong thing cleanly.

The model card also makes an important limitation explicit: it is still a research release, and a formal MOS / Persian intelligibility benchmark has not yet been completed.

So I would call Ava a promising first local candidate, not “as good as FaridNeural” or “the best Persian TTS”.

Mana-Persian-Piper

MahtaFetrat/Mana-Persian-Piper is much smaller. The main Piper ONNX file is only about 63.5 MB.

There is also an interesting companion project:

Piper + Lightweight Context-Aware Phonemizer

It adds context-sensitive Persian phonemization, especially around ambiguous phenomena such as Ezafe.

For a first deployment I would still start with the simpler path:

Ava first
Mana raw Piper as very small fallback

and only add the LCA layer if real pronunciation errors make it worthwhile.

Automatic comparison

One complication on my side is that I do not speak Persian, so I did not pretend to do a native-listener ranking.

Instead I used:

• two different Persian ASR families as intelligibility proxies;
• disagreement between those ASRs as an uncertainty signal;
• UTMOS separately as an acoustic-quality proxy.

I deliberately did not combine those into one arbitrary “overall TTS score”.

On 5 shared utterances per model, the aggregate proxy results were:

I would interpret that as:

Ava = strongest automatic shortlist candidate
Mana = very small fallback with surprisingly strong intelligibility proxy

I would not interpret it as:

Ava sounds best to native Persian listeners
Ava == edge-tts quality
CER/UTMOS == human MOS

The ASR models have their own errors, UTMOS measures acoustic quality rather than Persian pronunciation correctness, and Ezafe is especially difficult to evaluate from standard Persian orthography because it is often spoken but not explicitly written.

So this is useful for narrowing the search, not for replacing a native listening test.

Given that you already like fa-IR-FaridNeural, I would keep your cloud TTS as a temporary quality/reference fallback while integrating Ava locally. You do not need to delete the working path on day one.

I would leave your VAD and ASR alone initially.

Shenava-Rizeh v1.0 for sherpa-onnx is already a compact ~32M Persian ASR deployment. Its model card explicitly supports offline sherpa-onnx inference and separates spoken-number output from the bundled Persian ITN/post-processing layer.

That separation is useful:

audio
  -> ASR text
  -> Persian normalization / ITN
  -> LLM

rather than baking every normalization decision into the recognizer itself.

For a 4 GB GPU, my baseline resource split would be:

CPU:
  Silero VAD
  Shenava ASR
  Persian normalization / ITN
  orchestration

GPU-first:
  local LLM

TTS:
  start CPU-side if needed to protect LLM VRAM
  try CUDA only if first-audio latency justifies it

The important point is that:

model choice != device placement

You should be able to move a component between CPU/GPU without redesigning the logical pipeline.

Streaming

For voice interaction, I would not wait for the entire LLM answer before starting TTS.

Something like:

LLM stream
  -> accumulate a semantically safe clause/sentence
  -> synthesize it
  -> continue generating the next chunk

But I would also avoid extremely tiny chunks, because TTS prosody gets worse if you synthesize fragments without enough linguistic context.

Small integrated timing experiment

I put the pipeline together once as:

Shenava-Rizeh
    ->
Gemma 4 E2B Q4_0
    ->
first speakable response chunk
    ->
Ava-82M

The cleaned Colab notebook is here:

Persian ASR → LLM → TTS timing notebook

In the latest successful T4 run, the warm means were roughly:

Median serial proxy was about 0.992 s.

Gemma 4 E2B showed about 1.75 GB of observed T4 GPU memory in that run, and Ava selected CUDA.

Those numbers have a lot of boundaries:

• Colab T4, not GTX 1650 Ti;
• synthetic fixed Persian input, not microphone speech;
• fixture-generation time excluded;
• warm models;
• simple serial addition, not a fully overlapped streaming service;
• no VAD end-of-speech wait;
• no microphone or playback-device buffering;
• no IPC/server overhead;
• first-audio timing depends on the chunking rule.

So I would not predict “~1 second on your laptop” from this.

The useful result was something else: in that pipeline, the LLM TTFT was tiny compared with first-chunk TTS. That is a good reminder to measure the complete voice timeline before spending all the optimization effort on LLM tok/s.

On your machine, I would log:

t0 = VAD says utterance ended
t1 = ASR final transcript ready
t2 = first LLM token
t3 = first complete speakable chunk
t4 = first TTS audio ready
t5 = playback actually begins

Then you immediately know which component deserves optimization.

One integration gotcha

I also hit an unrelated but useful correctness issue: an LLM can say things like “I turned it off”, “I sent it”, or “I set the reminder” even when no real action tool exists.

For a robot I would keep that boundary explicit:

speech / text
    ->
LLM proposes intent
    ->
orchestrator / real tool executes
    ->
real tool result
    ->
LLM may report success

In other words, do not let the language model’s sentence itself be evidence that a physical or software action happened.

You do not need a large agent framework for this. A small explicit dispatcher is enough.

If you later want an OpenAI-compatible local HTTP boundary around llama.cpp, its server is documented here:

llama.cpp server

With normal broadband, downloading five candidate models and comparing them is annoying.

At ~0.7 Mbps, it becomes part of the engineering problem.

I would therefore use a deliberately boring strategy:

download one discriminating artifact
        ->
measure it
        ->
make the next decision

rather than:

download E2B
download E4B
download Qwen 7B
download Dorna 8B
download several quantizations
        ->
compare later

A few things that help:

• keep the HF/Ollama/llama.cpp caches;
• do not repeatedly remove working model files;
• use exact model filenames/revisions when possible;
• do not download Gemma 4’s ~987 MB multimodal projector for a text-only assistant;
• if an already-cached model can answer the deployment question, benchmark it before replacing it;
• prefer one decisive A/B over a large model collection.

This is also why I would not make a 7B/8B download the first step.

The larger model may eventually be the right answer, but finding out that a 3–4B model is sufficient saves download time, memory pressure, and integration complexity at the same time.

If you want one compact target-machine test, I would collect only:

1. runtime + exact model/quant
2. context size
3. `ollama ps` CPU/GPU placement
4. loaded VRAM / observed split
5. LLM TTFT
6. generation tok/s
7. 5-10 application-shaped Persian prompts
8. VAD end -> ASR final -> first token -> first audio

Then I would use this decision tree:

Gemma 4 E2B @ ~2K context
|
+-- Persian quality OK, latency OK
|      -> stop model shopping and integrate
|
+-- Persian quality weak, runtime fit OK
|      -> compare Gemma 3 4B / Qwen3 4B
|
+-- model is CPU-heavy / split and latency is bad
|      -> verify GPU path, reduce context, inspect offload
|
+-- 4B class consistently not good enough
|      -> try 7B/8B with partial offload
|         or retain cloud LLM as fallback
|
+-- local LLM is good, local TTS is weak
       -> keep local LLM and temporarily retain edge-tts

That last branch is important: “fully local” does not have to be one atomic migration.

You can move:

cloud LLM + cloud TTS
        ->
local LLM + cloud TTS
        ->
local LLM + local TTS

and keep the old path available until the replacement is actually good enough.