What does a AI judge score of 0.8 actually mean? Introducing Typed Evals

I’m building Typed Evals, an open-source Python toolkit for evaluating LLM responses, RAG pipelines, and Agent runs, with human-label calibration built in.

A judge returning 0.8 doesn’t necessarily mean humans would accept 80% of similar answers. You need labeled examples to check that. Typed Evals lets you fit a calibration mapping and test it on held-out data.

It supports:

  • Response relevance, RAG faithfulness, and Agent task/tool-grounding checks
  • Custom binary, categorical, and ordered-score rubrics
  • Checks on proposed tool calls before execution
  • Per-metric calibration that you can save and reuse

For example, you can check whether an agent claiming it created a ticket is actually supported by its recorded tool results.

Jev is the default judge. There’s also native OpenAI Decisions API support and Microsoft-Decision-1 through OpenRouter’s TypeSafe-compatible endpoint.

Support for Open models on HuggingFace like clef, laya, strands, etc. is coming soon!

I benchmarked Jev on 645 held-out TRIVIA+ answers:

  • Expected calibration error: 0.0982 → 0.0313
  • Hallucination-detection F1: 0.5833 → 0.5877

That’s roughly 68% lower calibration error, with little change in detection performance. The raw and calibrated cutoffs were selected separately on validation data. Keeping a fixed 0.5 cutoff actually reduced F1 after calibration, so choosing the threshold matters too.

These results are for Jev on this dataset, not a comparison of all supported backends or a guarantee on other tasks.

GitHub · Benchmark and methodology

There’s an offline demo in the repo if you want to try the workflow without an API key.

If you’re using LLM judges, how do you choose your pass/fail thresholds? Do you check them against human labels, or mostly rely on the judge’s raw scores?

For the threshold question, I would separate a probability-calibration check from an operational acceptance rule: specify the application’s false-accept/false-reject costs, freeze the decision rule using validation data, then audit the actually accepted pool on an untouched test set.

I tried a small offline negative control against probability_diagnostics at commit fdbcb157a0085d8c9bcf5598c7077854e203ad76. Both fixtures have 90 predictions at 0.1 and 10 at 0.9:

Human labels Global ECE Coverage at 0.8 Failing answers / accepted answers
9 passes in the low-score group; none in the high-score group 0.09 10% 10/10
No passes in the low-score group; 9 in the high-score group 0.09 10% 1/10

This isn’t a bug in ECE or a reproduction of your Jev result. Your docs already expose reliability-bin counts and the benchmark uses article-level uncertainty. The extra diagnostic I’d find useful is a threshold-specific report: accepted count, coverage, failed-accept count, false-reject count, and decision loss. An empty accepted pool should return unavailable risk, not zero risk.

For a calibrated probability of human pass p, accepting has expected loss C_FA*(1-p) and rejecting has C_FR*p, giving a two-action cutoff C_FA/(C_FA+C_FR). A 100:1 cost ratio implies about 0.9901, not 0.8; this depends on deployment calibration and doesn’t cover an abstain/review action. Observing 21 passes among 22 accepted examples still gives an approximately 0.782 lower two-sided 95% Wilson bound under independent sampling. For TRIVIA+'s related answers I would use article-group resampling instead.

I prepared a runnable offline helper with these controls and no judge API calls. It could sit beside offline_demo.py as a decision-audit example if that fits the toolkit’s scope.

Disclosure: AI assisted the code and analysis; the controls were executed against the pinned upstream implementation. All labels above are synthetic, not new human annotations.

Thanks, the accepted-pool example makes the distinction clear. Reporting what happens at the chosen cutoff would be a useful addition alongside the calibration diagnostics.

Could you share your offline helper through a GitHub issue or PR? I’d like to review it and start with an example beside offline_demo.py, including counts, coverage, error rates, and decision loss. Agreed that an empty accepted pool should report unavailable risk.

The fixed 0.5 cutoff reducing F1 is a useful detail. Better calibrated scores don’t automatically mean better pass/fail decisions. Have you tried this on agent tool calls yet? I’d be curious whether the calibration holds across different tools or needs separate labeled examples.

I haven’t tried it on tool calls yet. I’d be curious about that too, especially whether a shared calibrator looks good overall but misses problems with individual tools. I’d start with labeled calls from a few different tools and compare shared vs tool-specific calibration on held-out examples.

Keep calibration, cutoff selection, and final evaluation examples separate; report reliability bins with uncertainty. A calibrated 0.8 can still drift on a new domain. I work at BOMTOBE, behind Orke.

For the tool-call test, would you label authorization separately from execution success? A ticket can really exist, so the claim is grounded, while creating it was outside the user’s request.

A small paired fixture could keep the proposed call and successful tool result identical, but change the instruction from ‘create the ticket’ to ‘draft it for my review’. That tests whether the judge notices the permission boundary instead of just rewarding a successful result. I would report those two labels separately before choosing a combined acceptance rule.

AI-assisted comment from YouBeyond, where we are exploring personal-AI concepts; this is a suggested test, not a result we have measured.