Looking for recommendations on hardware, as well as model selection.
In Australia. Working as a physiotherapist. Looking for something like Heidi, but that doesn’t send data off anywhere at all. So essentially to - record the conversation as an audio file (or just get the text only if possible) > run the whole transcript through a model that generates a note that I approve before saving. I will be providing the model with several examples so that I can get the best formatting possible.
Best machine I am thinking of right now is a 24GB Macbook Air, as the jump to 32GB gets crazy expensive. I was thinking on this I would run:
MacWhisper Large v3 tubro and then use Qwen 27B or Gemma 26B to form the note for me.
OR, is it worth getting the extra RAM so that I am able to run Gemma 31B, as I understand it can be quite a bit more intelligent than both Qwen 27 and Gemma 26.
I am thinking as I need a laptop anyway, the Air might be the move. That’s 2500 AUD, and the jump to 32gb of RAM means I need to get a MB Pro, which is a huge jump to 3700 AUD. The other option is a 32GB Mac Mini, which is about 2000, but then doesn’t double up as a laptop.
Going a non Apple route seems absurdly difficult and expensive right now. Plus I couldn’t bring myself to run Windows, so it would need to be on Linux.
Good news: the 24GB Air should be enough, because the two models never need to be loaded at once — transcribe with Whisper first, unload it, then load the note model, so your peak memory is the larger model on its own rather than the two added together. A 26–27B model at 4-bit takes roughly 15GB, which leaves macOS and a consultation-length transcript room to breathe, and note formatting from your own examples is not a task that suddenly improves at 31B — when the note is wrong, the cause is almost always a transcript error or a weak example set, not the missing few billion parameters. So test it in that order before spending the extra ~1,200 AUD: judge Whisper on one real session’s transcript first, and judge the note model only on a clean transcript, fixing the examples and template there before considering more RAM.
Oh that’s great news. I mean with the formatting, to have the model organise the note appropriately. You’re supposed to order it sort of like, subjective info first, objective info with test scores, assessment and then plan. So it needs to follow a structure. I would have thought giving the model some examples would help it produce things better, like giving it a template, essentially.
I have used Whisper with v3 turbo on my 8gb M2 and it performs remarkably well. What comes next is the actual formation of the note. So thats where Qwen or Gemma will come in.
Which model do you think will perform best? Qwen 27 or Gemma 26. And you don’t think that bumping up an extra 8gb to afford more intelligent models like Qwen 35 or Gemma 31 will be worth it? If the smaller models can do it - perfect.
Any idea where I can test these smaller models out online with some mock info? I’ve tried Gemma 31B on duck.ai but it won’t allow such long transcripts to be put through.
Yes — and the way to make the structure hold is to split the job: first have the model sort the transcript into the four SOAP buckets, copying test scores exactly and adding nothing, then write the note from those buckets alone, in your template’s order. A fact in the wrong bucket is easy to catch; a polished sentence in the wrong section is not, and the check is mechanical — every bucket item appears once in the note, and nothing appears that was not in a bucket. Run that same two-step setup on both Qwen and Gemma over the same few mock transcripts and score section placement, dropped items, and changed numbers: that decides which is better for your notes, and a bigger model that loses there will not be rescued by more RAM, so the extra 8GB still is not worth buying first. For online testing, shrink the mocks, not the task — keep the scores, the plan, and one place where what the patient reports and what you found disagree, and cut the small talk; bucket and formatting errors show up at a third of the length, within the limits you hit.
One memory detail worth checking before you buy the 24 GB Air: on Apple Silicon the GPU can’t use all of the unified memory. By default macOS caps GPU-usable memory at roughly two-thirds of RAM on a 24 GB machine, so about 16 GB. A 27B model at Q4_K_M is a ~16–17 GB file before any context, so it sits right on that line, and a long consultation transcript adds KV cache on top. It can still work: you can pick a slightly smaller quant (Q4_K_S or IQ4_XS, around 15 GB), keep the context to what one session needs, or raise the cap with sudo sysctl iogpu.wired_limit_mb=19000 (it resets on reboot; leave a few GB for macOS). On 32 GB the default cap is about 21 GB, which fits a 27B Q4 with comfortable room for context.
Also note that the Air is fanless. Transcribing and writing one note at a time is short enough that it shouldn’t matter much, but a long batch of notes will run slower as it heats up.
For testing before you buy: you can build and tune the whole two-step SOAP pipeline on your 8 GB M2 with a small model (a 3–4B at Q4 fits), using your mock transcripts and template. Prompt and template problems show up there too. Then you only need to compare the 27B-class models on the final step, and you’ll already have the test set to do it.