Bulwark: a Turkish-first scam/spam classifier + a benchmark built from a real inbox (false alarms 53% → 14%)

TL;DR: I tested a popular multilingual scam detector (Horizon-Labs/phishing-scam-detector-base) on real SMS from my own phone. It flagged 53% of genuine Turkish bank, cargo and e-Government notifications as unwanted. I fine-tuned it into Bulwark and built a small, honest Turkish benchmark around the problem. Everything is open (model: Apache-2.0, benchmark: CC BY 4.0, code: Apache-2.0).

Model: SavienWalker/bulwark-scam-detector

Results (held-out real SMS, opened once at the end)

Base model (t=0.5) Bulwark (t=0.6)
False alarm on legitimate messages 53% (54/102) 14% (14/102)
Legitimate message labelled fraud 26% 9%
Unwanted (spam+fraud) caught 100% 98%
3-class accuracy 64% 87%

On a human-reviewed hard set built from contrast pairs (a fake message and the genuine one it imitates, e.g. fake vs. real toll-violation notice), both items correct: base 9/13, Bulwark 12/13. A quick 6-language probe (de/fr/es/pt/ru/ar) shows no regression.

What mattered more than the training itself

  • A sealed test split. 434 real SMS from my inbox, PII-masked, split by sender into dev/test so one institution never appears in both halves. Model selection and threshold sweep on dev only.
  • Leakage check. My first synthetic training set had 1,303 messages too similar (token Jaccard ≥ 0.5) to evaluation messages, because I had written templates after seeing them. Rewrote 31 templates and dropped the rest.
  • Leaked personal data is a shortcut trap. Scammers now include your name, plate, card’s last 4 digits, old address. In my generated data those details appeared almost only in legitimate messages, so the model was about to learn “it knows my name, so it’s real”. I injected them into legitimate and fraud messages at equal rates.
  • Real-world patterns from public complaints. Escalating prize fees, fake car rentals that “can’t take cards”, job offers that end in money-mule transfers or screen-sharing into e-Government. No complaint text was copied; patterns were rewritten as templates.

Setup

Base: mmBERT-base (307M). Embeddings frozen (110M trained), bf16, 2 epochs, ~9k messages (~8k Turkish synthetic from 190 templates + ~1k UCI SMS Spam as English replay, spam/fraud distilled from the base model). About 6 minutes on a laptop RTX 4050 (6 GB).

Limitations

Single messages only: phone calls, long chats and fake websites are out of scope. Real non-betting fraud in one inbox is rare, so fraud recall relies on the synthetic hard sets, which were written by the same LLM-assisted pipeline as the training templates. That is stated in the cards and results are reported separately.

Next up: multi-turn chat context, and scams that arrive via fake websites & ads.

Code (collection from iPhone/Android backups with immediate PII masking, labelling tool, leakage check, training, eval): github.com/SavienWalker/bulwark
Benchmark on the Hub: SavienWalker/bulwark-bench-tr

Feedback, and real scam samples (especially non-Turkish ones), are very welcome!

One split worth adding before the multi-turn work: hold out whole template families from the synthetic side — every message generated from a held-out pattern, near-variants included, stays out of training entirely — and report fraud recall on that slice alone. Your sender split already shows the model generalizes to institutions it has not seen; this is the version that shows whether it generalizes to scam wording it has not seen, which matters because the same pipeline wrote both the training templates and the hard sets. If recall holds there, the catch rate becomes very hard to argue with.

Thanks, this is exactly the weak spot. Right now the synthetic validation split holds out individual templates, not families, so a near-variant of a held-out pattern can still sit in training, and that split was also used for checkpoint selection, so it isn’t a clean measurement either.

I’ll group the 190 templates into families (fake delivery fee, bank-link phishing, “new number” family scam, task scams, etc.), hold out whole families, retrain, and report fraud recall on that slice alone, next to the main numbers. Will post the result here, whichever way it goes.

Ran it. I held out 4 whole families (26 templates incl. near-variants: delivery/customs fees, family impersonation and “send me the 6-digit code”, job/task scams + account renting, legal threats), retrained from the same base, and measured unwanted-detection on those families only (base t=0.5, Bulwark t=0.6):

Slice Base Bulwark (saw families) Bulwark (families held out)
Synthetic messages from held-out families (927) 92% 100% 97%
↳ job/task + mule family (480) 85% 100% 94%
Hand-written hard-set items in those families (15) 15/15 14/15 13/15

So recall mostly holds on unseen scam wording. The biggest drop is in job/task scams (100 → 94%), still above the base model. The two misses are telling: a “sorry, wrong number, please forward the 6-digit code” WhatsApp takeover (missed by both Bulwark variants), and a formal, link-free debt-enforcement notice that scored 0.05 once its family was unseen. Link-free, formal-tone fraud is the weak spot.

One more honest note: the held-out run also showed more false alarms on the real-SMS dev split (15% vs 5%). I haven’t measured seed-to-seed variance yet, so I’d treat a single-run gap like that with caution. Next I’ll repeat runs over several seeds and report mean ± spread. Code: family_holdout.py in the repo. Thanks again, this made the evaluation better.

That mostly-holds result is strong, and the false-alarm jump is exactly where your seed repeats can do double duty: run the seen and held-out training sets on the same seeds, in pairs, and report the per-seed difference for both numbers. If the recall gap stays near three points pair after pair while the false-alarm gap flips or vanishes, the first is a real cost of unseen families and the second was seed noise — unpaired means and spreads can’t separate those two.

Paired, same seeds (Δ = held-out − seen, points):

seed Recall on held-out families: seen → held-out (Δ) Real-SMS dev false alarms: seen → held-out (Δ)
13 100.0 → 96.7 (−3.3) 4.4 → 15.4 (+11.0)
21 100.0 → 95.4 (−4.6) 14.3 → 9.9 (−4.4)
42 100.0 → 95.6 (−4.4) 13.2 → 12.1 (−1.1)
mean Δ −4.1 (range −4.6 … −3.3) +1.8 (range −4.4 … +11.0)

Exactly the split you predicted: the recall cost of unseen families is consistent (~4 points every pair), while the false-alarm gap flips sign, so that was seed noise. One caveat: the “seen” model’s 100% is measured on families it was trained on, so −4 is the gap to a model that saw that exact wording, an upper bound on the cost. I’ll add the seed spread (dev false alarms 4–14% for the released recipe) to the model card. Thanks, pairing was the right call.