Calibrating Jev Thresholds: jevcal, jev-benchmark and Drift Checks
Typed decisions come with probabilities, but a threshold is only useful if you've measured it. jevcal, an eval-first jev-mcp and LightJev show how to calibrate and catch drift.
The selling point of a typed decision model is the probability that comes with the answer. The trap is trusting it blindly. A confidence of 0.9 only means something if, across many examples, answers at 0.9 are right about nine times in ten. That property is calibration, and it has to be measured on your data with your question wording.
A small set of Jev projects treat this as the main event rather than an afterthought. They are all very new, with 1-2 stars each as of September 2026, so they are best read as a workflow to borrow. The workflow, though, is sound and worth knowing.
Key Takeaways
- Thresholds should be measured, not guessed. jevcal picks a threshold that meets an accuracy target.
- Split your data. jevcal selects on half and validates on a held-out half.
- Drift is a CI problem. A
checkcommand re-evaluates on new data and can fail the build when a model update breaks a threshold. - Question wording is a variable. Tools exist to lint and compare question variants.
- Small-model research exists too. LightJev trains small backbones for typed decisions, with strict self-reported limits.
jevcal: from confidence scores to a threshold
jevcal (2 stars) is a Python tool that calibrates, thresholds and drift-checks typed decision models against an LLM "teacher." Its README lists seven commands:
| Command | Purpose |
|---|---|
lint | Flag question wording the model handles poorly |
label | Use an LLM to fill missing ground-truth labels |
measure | Run the model over a dataset and capture predictions |
compile | Turn predictions into threshold recommendations and reports |
run | measure plus compile in one step |
optimize | Rewrite questions with LLM help, validated on held-out data |
check | Re-evaluate thresholds on new data to detect drift |
The output includes HTML reports with reliability diagrams, accuracy-versus-coverage curves and a cost-benefit view for deciding whether to answer locally or escalate to a slower, more capable model. Thresholds and their supporting evidence are captured in a decisions.lock.json. Its pitch for check is that it "fails CI when a model update quietly breaks it." Also on aiskill.market: jevcal.
One caution: the optional LLM teacher labels the data. If those labels are wrong, your calibration is wrong. Spot-check them.
An eval-first MCP server
jev-mcp by BYK (1 star) builds evaluation into the tool surface. Its three tools are jev_ask (see the full probability distribution for one question), jev_map (run a question set over many items) and jev_eval (measure question variants against labeled examples). Evals report F1, AUC, Brier score, ECE and a threshold sweep, plus worst misses. Question types are noul (boolean), choice and score. See jev-mcp (BYK); the wider MCP story is in wiring Jev into agents with MCP.
Benchmarks and small models
jev-benchmark (2 stars) describes itself as benchmarks and a playground for Jev, with chess and a "who is the player talking to" task for speech-to-text game NPCs. Its README summary is all this series draws on, so no figures are quoted. It is listed as jev-benchmark.
LightJev (1 star) goes the other direction: training lightweight backbones to return typed decisions and candidate probabilities, using cross-entropy and Brier loss. Its published checkpoint, trained on Qwen3-0.6B with converted NanoJev data, reports 79.57% on a 656-question test set. The authors immediately bound that: results are synthetic-task and single-seed, and it is "not a broadly capable business decision model." Candidates must fit within 256 tokens. See LightJev.
A calibration routine you can run this week
- Collect a few hundred real examples for one decision, labeled by a human where possible.
- Try two or three wordings of the question and compare them on held-out data.
- Choose the threshold that meets your accuracy target, and record what fraction of cases it auto-handles.
- Send everything below the threshold to a larger model or a person.
- Re-run the check whenever the model or your data changes, and put it in CI.
What to do next
- Read the jevcal README and run it against a small labeled set.
- Apply the result to a router such as those in routing by difficulty.
- Use the checklist in how to evaluate Jev projects honestly before adopting anything from the radar.