Local Models vs Decision APIs: The Real Difference May Be Domain Knowledge

This started with an internal POC at SFEIR involving Jev, TypeSafe AI’s decision-oriented SaaS model, and enterprise IT ticket classification.

The initial question was straightforward: how well can a decision API classify our tickets without being trained on our company’s historical data?

We compared several approaches on a dataset of real IT support tickets:

  • Jev, a decision-oriented SaaS model
  • Qwen3-4B, a local generative LLM
  • Laya, a compact local classification model, evaluated zero-shot, with LoRA, and with full fine-tuning

The dataset contained around 30,000 tickets. We trained on 20,000 and evaluated on 500 held-out tickets. This was an initial experiment under time constraints, not a comprehensive benchmark.

The results were particularly interesting:

Model Department accuracy Priority accuracy
Jev 41.0% 57.0%
Qwen3-4B, zero-shot 42.4% 57.2%
Laya, zero-shot 44.8% 51.8%
Laya, LoRA 57.8% 65.6%
Laya, full fine-tuning 62.0% 72.2%

The main observation wasn’t simply that a small local model could outperform a decision API.

It was the difference between zero-shot classification and learning from actual business data.

For this particular task, fine-tuning Laya improved accuracy by 17.2 percentage points for department classification and 20.4 points for priority classification compared with its zero-shot baseline.

This raises an architectural question: when we have enough labeled business data, should we keep trying to describe our business rules through prompts and criteria, or should we train a specialized model to learn the decision boundaries directly?

I’m interested in hearing from people working on enterprise AI:

  • When do you choose a decision API or frontier model over a locally fine-tuned classifier?
  • How much does domain-specific training data change the equation in your experience?
  • Do you use hybrid architectures, with a specialized classifier handling straightforward cases and a larger model handling ambiguous ones?

The full article is in French: Classification de texte en entreprise : arrêtez de demander à votre LLM de trier le courrier.

I’d love to hear your thoughts and experiences.