The Jev Benchmark Landscape: Four Projects, Four Angles

Jev makes a specific pitch: typed, calibrated decisions at low latency and low cost, in place of a general chat model forced into structured output. That's a testable claim, and four independent open-source projects have tried to test it — at wildly different scales. One is a 534-decision benchmark ranking 48-plus systems on five dimensions. Two others are focused, single-domain comparisons: text classification against GLiNER, and passage reranking against dedicated rerankers and chat models. The fourth is a broader structured-output comparison still in an early, synthetic-workload stage. If you've read how to evaluate Jev projects honestly, this is the companion piece — a look at what's actually been measured so far, not just how to read a claim.

Key Takeaways

• jevbench is the most serious effort by far. 534 decisions across five dimensions (Smart, Cheap, Fast, Reliable, Open), 48-plus systems ranked, and a documented v1.4 composite score — but it's a one-person hobby project funded by donations, not an independent lab.
• Jev tops jevbench's own leaderboard, narrowly. Jev 1.13.0 scores 63.29 versus JevK5 v0.2.0's 62.04 and Hopper's 59.43 in the published v1.4 results — a close race, not a rout.
• jev-rerank-bench found near parity with Cohere, not a win. Reported nDCG@10 of 0.692 for Jev versus 0.691 for Cohere on eight English datasets — the author explicitly declines to call a winner.
• jev-benchmarks is a small pilot. 300 examples, comparing Jev against GLiNER on classification, calibration and selective automation — useful signal, not a large-sample result.
• typesafe-ai-benchmark is the least mature of the four. Its README describes limited synthetic workloads with separately captured results and says plainly that its numbers aren't a general ranking.

jevbench: the closest thing to a comprehensive scorecard

jevbench (MIT, 94 stars as of September 2026) is built around a specific claim: Jev-class decision models should be judged on five axes — Smart, Cheap, Fast, Reliable, and Open — not accuracy alone. "Smart" is chance-corrected accuracy split across difficulty tiers (hard, easy, standard, judge). "Cheap" is USD per 1,000 decisions at published pricing. "Fast" scores P50/P95 latency logarithmically. "Reliable" checks calibration — expected calibration error and fidelity to gold-label distributions, weighted toward the hard tier. "Open" tracks code license and weight availability separately, and isn't folded into the composite score.

The dataset spans 534 decisions (242 original, plus 292 added in v1.2+), and the published v1.4 leaderboard ranks 48-plus systems, from commercial APIs (Jev, Claude Opus, Gemini, DeepSeek) to open-weight models (Qwen, LLaMA, Mistral) to specialized decision models (Winnow-12B, reflex 4B, GLiNER2 variants) and rerankers (zerank-2, BAAI bge-reranker). Jev 1.13.0 leads at 63.29, ahead of JevK5 v0.2.0 at 62.04 and Hopper at 59.43 — a real result, but a close one, and the composite blends four of the five axes with an equal-weight harmonic mean plus a 20% tilt toward fresh, sealed decisions.

Caveats the author states directly: 242 decisions is pilot scale, the dataset is English-only and hand-written, the held-out split has been sent to the services under test (so it isn't contamination-proof), latency was measured from Germany at one point in time and adjusted ×2 for self-hosted systems as an assumption rather than a measurement, and the adequacy cohort is skewed (61 yes / 7 no, an 82% majority-class baseline). It's explicitly a one-person hobby project funded via donations — the most rigorous benchmark in the Jev ecosystem so far, but still a single independent effort. On aiskill.market: /skills/jevbench-fstandhartinger.

jev-rerank-bench: parity with Cohere, not a win

jev-rerank-bench (MIT, 7 stars) is narrower and more honest about its scope. It compares Jev, dedicated rerankers and chat models on identical retrieved passages, using Jev's Choice and Noul question types plus rubric scoring to rank candidates, then computing standard retrieval metrics.

The headline number is close: the author reports equal-dataset nDCG@10 of 0.692 for Jev versus 0.691 for Cohere across eight English datasets — a difference small enough that the author declines to declare a winner, and notes that query weighting changes the comparison. We reviewed the README and integration source at a fixed commit; this site has not independently run or reproduced the benchmark. At 7 stars, treat this as one careful person's pilot, not a settled comparison. On aiskill.market: /skills/jev-rerank-bench-anessbelbati.

jev-benchmarks: a 300-example classification pilot

jev-benchmarks (Apache-2.0, 17 stars) runs the same labeled text-classification tasks through both Jev and GLiNER, recording probabilities, latency and failures for each. Its focus is less "who wins" and more whether confidence scores actually support the thresholds you'd want to set on them — a question directly relevant to the calibration work covered in calibrating Jev thresholds.

The author reports results from a 300-example pilot. Hosted Jev and locally-run GLiNER timings aren't hardware-normalized against each other, which the author states outright, and this site hasn't rerun the comparison. Small sample, single domain, but a clean methodology for what it does test. On aiskill.market: /skills/jev-benchmarks-abdelstark.

typesafe-ai-benchmark: early and synthetic

typesafe-ai-benchmark (MIT, 38 stars) compares Jev against other structured-output models on shared tasks, mapping each one to Jev's Choice/Noul question format and normalizing results into one comparison table, tracking errors, latency, token counts and estimated cost.

This is the least mature of the four. Its own README describes the current workloads as limited and synthetic, with results captured separately rather than in a single controlled run, and states directly that a demo video isn't measurement data and the results shouldn't be read as a general ranking across all tasks. We haven't reproduced any of its numbers. Worth watching as it matures, not worth citing as evidence yet. On aiskill.market: /skills/typesafe-ai-benchmark-iammrduncan.

What the four together actually tell you

None of these are independent, adversarial audits — three of the four are written by people plainly invested in the Jev ecosystem, and jevbench is the only one with a large enough sample and broad enough system coverage to support a leaderboard claim. What they agree on, loosely: Jev is competitive with specialized alternatives (GLiNER, Cohere's reranker) rather than dominant over them, and the gap to purpose-built tools in adjacent tasks (reranking, entity classification) is often small. What none of them settle: how Jev performs on your specific task, your specific data distribution, or your specific latency budget — which is the whole argument for running your own eval before trusting any of these.

What to do next

1. Read jevbench's methodology section before quoting its leaderboard. The composite score is transparent about its assumptions, but the assumptions matter — especially the ×2 latency adjustment for self-hosted systems.
2. Treat the smaller benchmarks as directional, not conclusive. 300 examples and 7-17 stars are real signal from careful authors, not a substitute for your own eval.
3. Don't average across benchmarks. Each measures a different task (general decisions, reranking, classification) — a strong jevbench score says nothing about reranking performance specifically.
4. Run your own held-out set before you set thresholds in production, using the calibration approach in calibrating Jev thresholds.
5. Watch typesafe-ai-benchmark mature rather than citing its current synthetic-workload numbers, and check the Awesome Jev projects radar periodically — this category is moving fast.