How capable are 7B–14B domain-specific LLMs in real production?

1. Microsoft: a small model as an application interface

The Microsoft case study is probably the clearest public example because it describes an internal cloud-supply-chain application rather than only a benchmark.

The system handles a fixed collection of application tasks. The small model interprets a user request and generates code or API interactions for the existing application. The application’s established logic then performs the calculation or operation.

That boundary matters. The model is not independently inventing supply-chain policies or replacing the optimizer. It is learning a constrained mapping such as:

natural-language request
        ↓
supported-task / out-of-domain decision
        ↓
application code or API parameters
        ↓
existing deterministic application

The paper evaluates models including Phi-3 Mini 3.8B, Gemma 7B, Mistral 7B, and Llama 3 8B. It reports that fine-tuned small models can outperform much larger prompted models on the selected application tasks.

However, that comparison should be read carefully:

• the small models received task-specific fine-tuning data;
• the larger model was evaluated primarily through in-context prompting;
• the target was a fixed set of application interactions;
• unsupported requests could be rejected as out of domain;
• the downstream application retained deterministic control.

This is strong evidence that a small model can become an effective interface to a mature application. It is not evidence that a 3.8B model has greater general reasoning ability than GPT-4.

2. Financial transaction strings: 8B and 4B for structured extraction

The recent paper How Small Can You Go? describes a current production system using a LoRA-fine-tuned Llama 3.1 8B model to extract merchant information from noisy financial transaction strings.

The reported production baseline reaches approximately 96.95% F1. A reproduced rank-8 LoRA version reaches 96.75%, and Qwen 3.5 4B with direct JSON output reaches 96.60%.

The study also deploys 14 sub-8B variants as Databricks Model Serving endpoints. The authors report a relatively small average difference between offline and serving evaluation, although one architecture showed a materially larger drop.

This is useful evidence because it includes:

• a named production task;
• a stated model size;
• a structured output contract;
• accuracy, throughput, training, and serving considerations;
• evaluation after deployment to serving endpoints.

But the task boundary is very narrow: converting noisy transaction descriptions into a known set of fields. It does not require the model to decide whether a transaction is lawful, recommend an investment, interpret a contract, or reconcile competing financial evidence.

It also shows that “reasoning supervision” is not automatically helpful. In this extraction task, the best 4B result came from direct JSON-only output rather than an explicit chain-of-thought format. For some production tasks, making the model reason more visibly may add cost or additional failure surfaces without improving the required mapping.

This paper was released in June 2026, so I would treat it as a valuable and unusually concrete deployment report, but still a recent preprint rather than settled industry-wide evidence.

3. Natural language to a domain-specific language

The NL-to-DSL production report describes fine-tuning Mistral 7B to generate an enterprise workflow DSL.

This is a more complex task than field extraction, but it is still surrounded by strong constraints:

• the available APIs are known;
• the output language has a defined grammar;
• generated workflows can be parsed or compiled;
• execution is performed by the same runtime used for manually authored workflows;
• generation latency and workflow execution latency can be measured separately.

The authors report pre-production evaluation, load testing, production deployment, and A/B testing. They also describe important limitations, including:

• hallucinated operations;
• confusion between similar APIs;
• difficulty with unseen API combinations;
• prompt sensitivity;
• overly long workflows;
• limited reasoning and long-context behavior;
• failures that increase under load.

This is a useful middle case. It shows that a 7B model can support a genuine production workflow feature, while also showing why that does not imply reliable open-ended agency.

The compiler and execution runtime create a trust boundary:

LLM proposes a workflow
        ↓
parser / compiler validates its structure
        ↓
runtime executes permitted operations

The system is therefore safer than one in which free-form generated text is treated directly as a correct business decision.

How strong is the evidence?

It may help to distinguish several levels that are often all described as “production”:

The public cases above are stronger than ordinary leaderboard comparisons, but even these papers disclose different amounts of operational information.

Details such as false-acceptance rates, long-term regressions, human-review load, and serious tail failures are still rarely published.

“Domain-specific” can describe very different things

A “financial LLM” could mean any of the following:

• merchant-name extraction;
• transaction classification;
• document retrieval;
• table calculation;
• financial statement QA;
• regulatory-document interpretation;
• credit or fraud decisions;
• investment research;
• open-ended financial advice.

Those tasks differ radically in:

• required knowledge;
• reasoning depth;
• acceptable output;
• ease of automatic verification;
• cost of a confident error.

The public success cases are concentrated toward the top of that list: extraction, classification, structured transformation, retrieval, and bounded application control.

The broadest tasks usually have weaker automatic checks and a larger space of unfamiliar cases.

The same distinction applies to law. Extracting a termination date or classifying a clause is not equivalent to reconciling conflicting authorities, identifying a missing fact, or recommending a legal position.

In-distribution accuracy is not the whole deployment problem

A fine-tuned model may be excellent when the request resembles its training distribution and brittle when it does not.

Important boundaries include:

This can explain why one team reports excellent results from an 8B extraction model while another team finds a 27B coding or agent model inconsistent.

They may not be measuring the same capability.

Model capability and system reliability are also different

A production result is produced by a stack, not only by a checkpoint.

A more complete identity for the deployed model is something like:

checkpoint
+ adapter or merged weights
+ tokenizer
+ chat template
+ quantization
+ inference engine and version
+ reasoning parser
+ tool-call parser
+ structured-decoding backend
+ sampling and stop settings
+ retrieval system
+ tool definitions
+ agent loop

A failure in any of these layers may appear to the user as “the model reasoned incorrectly.”

For example:

• a wrong chat template may omit or corrupt tool definitions;
• a parser may fail to recognize an otherwise usable tool call;
• a stop token may terminate an agent before execution;
• constrained decoding may produce valid JSON with the wrong semantic choice;
• a quantized artifact may behave differently from the model-card evaluation;
• an agent framework may incorrectly interpret an empty or malformed call;
• a retriever may provide the wrong evidence while the generator behaves normally.

The current Qwen3.6-27B model card gives specific serving guidance, including a qwen3 reasoning parser and a qwen3_coder tool-call parser for vLLM. This does not prove that reports of unreliability are configuration mistakes. It does show that a statement such as “Qwen3.6-27B was unreliable” cannot be reduced to parameter count without knowing the exact serving and agent configuration.

The vLLM tool-calling documentation makes another important distinction: constrained function calling can guarantee a validly parseable call, but not a high-quality call.

A schema can guarantee something shaped like:

{
  "customer_id": 123,
  "action": "refund"
}

It cannot by itself guarantee that:

• 123 is the correct customer;
• a refund is authorized;
• the amount is correct;
• the tool should have been called at all;
• a clarification should have been requested first.

That semantic layer still needs model evaluation, business rules, external validation, or escalation.

Long-horizon behavior is a separate capability

A model that succeeds on isolated questions may still fail in an agent loop because each step introduces another chance to:

• select the wrong tool;
• use the wrong argument;
• misunderstand a result;
• lose state;
• repeat an operation;
• stop prematurely;
• continue when it should abstain.

Even strong frontier models remain imperfect on long-horizon agent tasks, so this is not exclusively a small-model problem. Smaller models may nevertheless reach an unacceptable failure rate sooner as the number of decisions grows.

For agent-like deployments, it therefore helps to evaluate separately:

1. tool selection;
2. argument correctness;
3. no-tool decisions;
4. clarification behavior;
5. interpretation of tool results;
6. multi-step completion;
7. repeated-run success.

A single function-calling accuracy number can hide these differences.

SFT and LoRA

SFT and LoRA can be extremely effective when the desired behavior is well represented in the training data.

They can teach or reinforce:

• a stable output format;
• domain terminology;
• known workflows;
• classification boundaries;
• extraction mappings;
• tool-selection patterns;
• organization-specific style or policy.

LoRA makes this adaptation much cheaper than updating the complete model, but it does not change the logical question being asked.

A successful LoRA result establishes that a low-rank update was sufficient for the evaluated task. It does not automatically establish that the model acquired all missing domain knowledge or broad reasoning ability.

The AdaptLLM paper is useful here. It found that continued training on raw domain corpora could increase domain knowledge while harming the model’s ability to answer questions through prompting. The authors improved this by converting domain material into reading-comprehension tasks and mixing general instructions.

The broader lesson is that these are separate properties:

• possessing domain information;
• following an instruction;
• retrieving the relevant information internally;
• reasoning with it;
• preserving general capability;
• recognizing an unfamiliar case.

A model can improve on one and regress on another.

This is also why evaluation after specialization should normally include:

• the target in-domain task;
• unseen formats within the same domain;
• out-of-domain inputs;
• ambiguous inputs;
• cases where clarification is required;
• cases where the correct action is abstention;
• a regression set for previously useful general behavior.

Distillation

Distillation can transfer a strong teacher’s behavior into a smaller model, especially when the production task is narrow enough to generate many representative examples.

It can be very effective for:

• classification;
• extraction;
• style-constrained generation;
• fixed tool patterns;
• frequently repeated question types.

But a student matching its teacher on generated training examples does not prove that it matches the teacher on:

• unseen problem structures;
• rare exceptions;
• adversarial inputs;
• uncertain or underspecified questions;
• long multi-step reasoning;
• facts absent from the teacher-generated data.

The evaluation should ideally be independent of the teacher that generated the training data. Otherwise, the student may mainly be learning the teacher’s preferred wording and error patterns, while a teacher-like judge rewards the same behavior.

RAG

RAG addresses access to external information. It does not automatically solve reasoning, evidence selection, or reliability.

A useful decomposition is:

1. Did the retriever find the right source?
2. Did the relevant passage reach the model?
3. Did the model use it correctly?
4. Did the answer remain faithful to it?
5. Did the model detect when the evidence was insufficient or conflicting?

A retrieval failure and a generator failure may produce the same wrong answer but require different fixes.

The NAACL 2025 study On the Influence of Context Size and Model Choice in RAG Systems compared multiple retrievers, context sizes, and eight base models on long-form QA. Performance improved as useful context was added up to a point, then often stagnated or declined.

The earlier Lost in the Middle study also showed that models may perform worse when the relevant information is located in the middle of a long context, even when that context fits within the advertised window.

So “the model supports 128K or 256K tokens” does not mean “the model reliably uses every relevant fact within 128K or 256K tokens.”

A small but informative RAG comparison would separate:

question only
oracle evidence only
oracle evidence plus distractors
actual retriever output
same evidence placed at different positions

This helps distinguish:

• missing model knowledge;
• retrieval failure;
• distraction sensitivity;
• long-context utilization;
• reasoning failure after successful retrieval.

For a narrow grounded-answer task, a 7B–14B model may be completely adequate. For synthesis across many partially conflicting documents, generator capability may remain the bottleneck even with a good retriever.

Compare the actual product feature, not public leaderboard scores

The strongest comparison uses:

• the same sampled production requests;
• the same retrieved documents;
• the same tools;
• the same business rules;
• the same output contract;
• the same maximum latency;
• the same escalation policy.

The SLaM study is a useful example of the methodology. It evaluated nine small-model families and 29 variants against an existing customer-facing GPT-4 feature, considering quality, consistency, latency, and cost rather than only generic benchmarks.

For that particular low-risk generation feature, some small models were competitive and were reported to reduce cost substantially. That result should not be generalized directly to legal or financial reasoning, but the evaluation method transfers well:

Start with the existing product behavior and representative product traffic, then test whether the smaller model satisfies the same feature-level requirements.

Use a risk–coverage curve

A single accuracy number does not describe selective deployment well.

Suppose the small model can either answer or escalate. For each acceptance threshold, measure:

• coverage: the proportion of requests answered by the small model;
• selective risk: the error rate among those accepted answers.

An illustrative result might look like:

The correct operating point depends on the cost of an error.

For merchant-field extraction, some mistakes may be detected downstream or corrected. For legal advice, a confident unsupported statement may be much more consequential.

Confidence should also be calibrated on held-out real traffic. A model’s verbal claim that it is “90% confident” is not by itself a reliable probability estimate.

Consider routing rather than replacement

RouteLLM provides one concrete framework for routing requests between a weaker and a stronger model. Its documentation explicitly treats the routing threshold as a quality–cost trade-off that should be calibrated for the types of queries the system receives.

Possible architectures include:

simple request ───────→ small model
complex request ──────→ large model
high-risk request ────→ human review

or:

small model proposes answer
        ↓
validator accepts ────→ return answer
        ↓ rejects
strong model retries
        ↓ still uncertain
human review

The main measurement is not whether the small model can answer everything. It is whether it can safely absorb enough traffic to justify the added routing and monitoring complexity.

One broad adapter versus a portfolio of narrow adapters

Another option is not to build one “finance model,” but several task-specific components:

transaction extraction adapter
document classification adapter
retrieval-query adapter
structured calculation adapter
policy-routing adapter

The LoRA Land / LoRAX study explored hundreds of task-specific LoRA fine-tunes and demonstrated serving many adapters over shared base-model weights.

That does not prove that every adapter is production-ready, but it shows that a portfolio of narrow specialized models can be operationally plausible.

This may preserve clearer contracts than forcing one small model to learn every task, tone, policy, tool, and exception in a broad domain.

Evaluate the exact deployed artifact

The final test should use the artifact that will actually serve traffic:

• exact checkpoint revision;
• exact adapter;
• merged versus dynamically loaded adapter;
• quantization format;
• tokenizer revision;
• chat template;
• inference runtime and version;
• tensor parallel configuration;
• context limit;
• KV-cache precision;
• structured-output backend;
• reasoning and tool parsers;
• sampling parameters;
• stop sequences.

Model-card benchmark results from BF16 weights do not automatically describe a 4-bit deployment through a different template and runtime.

Useful serving metrics include:

• time to first token;
• inter-token latency;
• total latency;
• queue time;
• throughput;
• concurrency;
• KV-cache utilization;
• error and timeout rate;
• output length;
• fallback rate under load.

Offline quality and serving quality should also be compared explicitly. The merchant-extraction report is useful partly because it tested the served endpoints rather than assuming that offline results would transfer unchanged.

For production reports, the most informative details would probably be:

Task boundary

• What exact task did the model perform?
• Was the output free-form text, JSON, SQL, DSL, or a tool call?
• Was the task extraction, classification, QA, planning, or decision support?
• How many turns and tool calls were normally required?

Traffic

• How much traffic was genuinely in distribution?
• What kinds of out-of-domain requests appeared?
• Did the evaluation use real traffic, synthetic data, public benchmarks, or a mixture?
• Was there a temporal holdout for new policies, products, or document formats?

Failure containment

• Could the output be checked independently?
• What caused automatic rejection?
• What percentage was escalated?
• What percentage required human review?
• What were the most serious failure modes?
• Could the system undo or correct a bad action?

Deployment artifact

• Which exact model revision and adapter were used?
• Was it quantized?
• Which inference runtime and chat template were used?
• Were tool or reasoning parsers involved?
• Did serving behavior differ from offline behavior?

Comparison

• Was the small model compared with a larger model on the same requests?
• Were retrieval results and tools held constant?
• Was the goal full replacement or only handling a safe subset?
• Which metric ultimately determined the model choice?
• Did review and fallback costs erase the inference savings?

A report such as:

“We use an 8B LoRA model for legal clause extraction”

is useful, but a report such as:

“It handles 72% of incoming contracts, has a 0.4% error rate among accepted extractions, escalates 28%, and all outputs are checked against deterministic date and clause rules”

would be much more transferable.