jevals: An Evals Platform Built Around Typed Judges

An LLM-as-judge eval is expensive to run continuously. You either sample a small fraction of traces, run evals only in CI, or eat a real cost per production request. jevals (MIT, 65 stars as of September 2026, /skills/jevals-openlayer-ai) makes a specific bet: that a decision model returning calibrated probabilities, not free text, changes the economics enough to run evals on every trace in the loop instead of a sample after the fact.

This article looks at what jevals actually checks, the cost and latency numbers its own documentation reports, and the two local backends it offers as an alternative to calling Jev's API. For the general case for typed output over free text, see typed decisions vs free text agents; for how to weigh a claim like this one, see how to evaluate Jev projects honestly.

Key Takeaways

• One request, many questions. jevals batches typed yes/no, multiple-choice, and rubric questions into a single forward pass, so evaluating 40+ criteria costs close to what evaluating one does.
• The headline cost comparison is the author's own benchmark, not an independent one. jevals claims roughly $0.03 per 1,000 samples versus $2.60 for Ragas, on the project's own test setup.
• 37 built-in checks span three areas: agent behavior (tool choice, grounding, scope), security (injection, PII/PHI, jailbreak detection), and output quality (faithfulness, relevance, completeness).
• Two local backends remove the network dependency. Kev (Qwen3-based) runs on a Mac; Laya (ModernBERT-based) claims about 10ms on Apple Silicon for fully offline evaluation.
• The project labels itself Alpha. The README itself warns of "rough edges" in framework adapters, and says jevals won't replace a judge for work needing multi-step reasoning or a written critique.

How the typed-judge approach works

A conventional LLM judge generates a free-text critique, which you then parse or rate. jevals instead sends the agent's state alongside a set of typed questions and gets back a calibrated probability for each one, in a single request. Because the questions are bounded — not "explain why this response is good," but "is this response grounded in the retrieved context: yes or no, with what confidence" — the model doesn't need to generate prose to answer them.

The claimed effect is throughput: the project's documentation shows a full evaluation pass costing "1 request · 1,423 tokens · $0.00006 · 0.50s," with latency reported at p50 244ms and p95 371ms per request regardless of how many of the 37 built-in checks are included. That's the core value proposition — not that any individual eval is more accurate than an LLM judge would be, but that running dozens of them per trace becomes cheap enough to do continuously rather than in a nightly batch.

The cost comparison, and what it doesn't cover

jevals' README states a benchmark of roughly $0.03 per 1,000 samples against $2.60 for Ragas, and cites $0.042 per million input tokens with no output-token cost as the underlying Jev pricing. Take this as the author's own comparison, run on the author's own test harness — it isn't independently retested here, and a Ragas configuration using a smaller or cheaper judge model would close some of that gap. What the comparison does establish credibly is the mechanism: a typed-output model genuinely has no output tokens to pay for beyond the structured answer, which is a real structural advantage over a judge that writes a paragraph per question.

What the 37 built-in evals cover

The checks split into three groups. Agent-specific evals look at whether a tool call matched the task (tool choice), whether a generated answer is actually backed by retrieved context (grounding), and whether the agent stayed within its intended scope. Security evals cover prompt injection detection, PII and PHI redaction, and jailbreak attempts — the same family of typed yes/no questions used by dedicated guardrail tools like the ones covered in jev-guard, jev-mcp and Safer with Jev. Quality evals check faithfulness, relevance, and completeness against a reference or rubric.

Running all 37 on every trace is the point the project is making about cost, but it's worth being deliberate about which of them you actually need live versus in CI — more checks means more places a miscalibrated threshold can produce a false block or a false pass.

Kev and Laya: evaluation without the API call

jevals ships two open-weight alternatives to calling Jev's hosted API. Kev, built on Qwen3, runs locally via python -m kev.serve — useful if you need evaluation results to never leave your machine, at the cost of running your own inference. Laya, built on ModernBERT, is smaller and faster: the README reports about 10ms per call on Apple Silicon, small enough to run inline without noticeably slowing an agent loop, and fully offline. Neither backend's accuracy against the hosted Jev model is benchmarked in what's public — treat the choice as a latency/cost/privacy tradeoff, not a strictly better option.

What to do next

1. Start with the security and grounding evals, not all 37 — they map most directly onto failure modes worth catching before they reach a user.
2. Re-run the $0.03-vs-$2.60 comparison on your own workload before citing it. Token counts and judge model choice change the ratio.
3. Pick Kev or Laya only if you need offline evaluation. Otherwise the hosted API keeps you on the calibration TypeSafe maintains centrally.
4. Expect Alpha-quality rough edges in framework adapters specifically — the README says so directly, and it's the kind of gap that shows up as an integration bug, not a scoring one.
5. Read the evaluation checklist at how to evaluate Jev projects honestly, and see the wider set of security and eval tools at the Awesome Jev radar.