实测 Jev 能否以极低成本替代 LLM 做系统综述筛选

Subscribe now

TL;DR: I tested Jev, a cheap zero-shot decision model, for title/abstract screening in systematic reviews. On eight CLEF-TAR DTA reviews covering 26,832 records, a remarkably simple single-question Jev setup achieved mean WSS@95 = 0.661 for about US$1.46, competitive with active learning flows, WSS@95 = 0.6631 and even substantially more complicated published LLM pipelines; it was beaten only by the 2026 three-LLM ensemble approach, which reached WSS@95 = 0.680 without embedding reranking and 0.708 with it.

Surprisingly, in my test making Jev more sophisticated did not help: decomposing screening into up to five criterion-level questions produced WSS@95 of only 0.536, despite requiring roughly five times as many judgements.

So Jev looks unusually attractive in terms of performance versus cost and complexity, but these are retrospective benchmark results, not evidence that it is safe to automatically exclude papers in a live systematic review. Thresholds, calibration and performance across other review types still need prospective validation. Results are preliminary and, due to publication lag, the comparison studies use older LLMs.

Introduction

One of the most powerful and simple ways to determine relevance of a paper today is to simply take a frontier LLM, point it at the title and abstract of an article and prompt the LLM to determine if the paper is relevant, given the search query or the inclusion/exclusion criteria in an evidence synthesis task2.

Is this method perfect? Of course not. But in general, it often performs better than both conventional lexical search and semantic search. Why?

Lexical search (Boolean, BM25 and so on) uses matching terms as a proxy for relevance. If the words show up, the paper is probably relevant. Semantic search nowadays typically uses similarity between dense vector embeddings as its proxy. If the vectors are close, the paper is probably relevant. Both can work reasonably well, but notice that in each case you’re working with an approximation of relevance, not relevance itself3.

This fact has not gone unnoticed by the evidence synthesis community, and there are dozens of papers that test whether you can use frontier LLMs as screeners in systematic reviews4.

Results are generally promising with overall high sensitivity rates and reasonable specificity, although performance varies substantially between reviews. It also remains unclear exactly where LLM screening fits best: as an initial filter, as a prioritisation system, or as an additional reviewer alongside humans.

Leaving aside these issues, modern frontier LLMs are expensive to use and relatively slow. At the time of writing, Fable 5.1 is priced at US$10 per million input tokens and US$50 per million output tokens.

A typical title and abstract would consume maybe 500 tokens. Add the screening instructions (possibly discounted by prompt caching), reasoning and structured output, and the cost begins to add up.

A quick sample I ran using OpenRouter came to about US$0.05 per screened record, or around US$50 per 1,000 records. Not insane, but still not nothing.

Is there anything else?

Enter Jev

This is where TypeSafe’s Jev comes in. It describes itself as

a new class of frontier models built to make fast, structured decisions that software can use directly.

TypeSafe calls Jev a “System One” model, borrowing the term from the distinction between fast, intuitive System 1 thinking and slower, more deliberative System 2 reasoning. The analogy is that Jev is designed for fast, bounded judgements such as classifying, scoring or choosing, rather than open-ended generation and long chains of reasoning.

It is important to note that Jev is not a generative chat LLM in TypeSafe’s framing and does not autoregressively generate free form text. It acts more like a classifier: given unstructured text plus structured questions, it returns typed decisions with probabilities; some decision types also return a separate confidence estimate.

There are only three ways to use Jev

Noul - Where you ask a binary question

Choice - Where you ask the model to choose from a predefined set of options

Score - Where you ask the model to rate or give a score on a predefined scale.

Below, I consider how you would use Jev for an evidence synthesis screening task.

Noul: a binary question. For example, ‘Given these inclusion/exclusion criteria, should this record (title/abstract) be included?’ (Yes/No)

Choice: There are two ways to use this. You could ask Jev to choose between Yes/No/Maybe, or you could ask a question based on the inclusion criteria, such as ‘What is the study design?’, with the system choosing from a predefined list such as RCT, Cohort Study, Case Control Study, etc.

Or you could ask the same inclusion question with more than two options, e.g. Yes/No/Unclear.

Score: Here you ask the model to give a score or rating on a scale/rubric you give.

There are only three modes but you can also combine them in a criterion-by-criterion workflow. When using Jev for evidence synthesis screening this involves blending multiple Jev judgements of different types to make a decision.

Instead of using Noul, Choice or Score to make a single overall screening decision, you can break the eligibility criteria into separate questions. Each criterion can then be evaluated using Noul, Choice or Score, with a deterministic script combining the answers to arrive at an overall Include, Exclude or Uncertain decision.[3]

For example, Jev might separately evaluate whether the population is eligible, whether the intervention matches, and whether the study design is appropriate. A script could then specify that a study is excluded if any mandatory criterion is confidently not met, included if all criteria are met, and otherwise sent for human review.

One important limitation is that Jev does not generate free text. It therefore cannot take a set of eligibility criteria and independently formulate the screening questions in the same way a general-purpose LLM can. The questions and permitted outputs have to be specified in advance.

One possible hybrid workflow that involves a Q&A approach to screening would be to use an LLM to translate the review’s eligibility criteria into structured questions, and then use Jev to apply those questions consistently across every title and abstract.

Another distinctive feature of Jev is that its outputs are accompanied by probabilities. TypeSafe claims that these probabilities are calibrated. Calibration has a specific meaning: if a well-calibrated model assigns a probability of 0.1 to a large number of predictions, the predicted event should actually occur in roughly 10% of those cases. Similarly, among predictions assigned probability 0.9, about 90% should turn out to be correct.

This is potentially useful for evidence synthesis because the probabilities do not need to be converted immediately into hard Yes/No decisions. For example, a workflow might automatically exclude only records for which Jev assigns a very low probability of eligibility, while sending uncertain cases for human screening.

General-purpose LLMs can also provide something that looks like a probability. You can simply prompt the model to state how confident it is, or, where the API exposes token log probabilities, estimate its relative preference for outputs such as “Include” versus “Exclude”. However, neither should automatically be interpreted as a calibrated probability of correctness.

This is partly because calibration can be affected by post-training. Research has found that reinforcement learning with human feedback, or RLHF, can make model probability distributions more overconfident and degrade calibration compared with the underlying pretrained model. At the same time, other studies have found that explicitly asking RLHF-trained models to verbalise their confidence can sometimes produce better-calibrated estimates than raw token probabilities. In other words, whether an LLM saying “90% confident” really corresponds to being correct about 90% of the time is an empirical question that depends on the model, task and method used to obtain the confidence estimate.

This is what makes TypeSafe’s claim about Jev interesting: calibration is presented as an explicit design objective rather than something inferred from logits or elicited through prompting. Even so, a calibrated model in general is not necessarily calibrated for evidence-synthesis screening in a particular domain. Before using Jev probabilities to set automated exclusion thresholds, they should ideally be validated on labelled screening data from the target task.

The Jev advantage

TypeSafe claims Jev can almost match the performance of much larger LLMs, at much lower latency and cost. They also claim Jev never makes type errors; e.g., if given a list of options to select from “Yes”, “No”, “Maybe” it cannot output “Depends”.

We don’t have much detail about how it works except we know that instead of being trained with the typical Reinforcement Learning with Human Feedback (RLHF) or Reinforcement Learning with Verifiable Rewards (RLVR), TypeSafe uses Reinforcement Learning for Calibrated Decisions (RLCD) which we have little information about.

Most importantly, TypeSafe claims 40-200x speed advantage and 10-200x price advantage over LLMs depending on the size of the LLM. The cost is helped greatly by the fact that Jev does not charge anything for output tokens only input tokens.

A technical reader might object that we have had classifiers for decades and comparing LLMs to what is essentially a classifier is comparing apples with oranges.

While it is true that traditional ML focuses on classification and we have more modern classifiers or discriminative models based on transformer models like ModernBERT that can be run locally for even lower cost, these models are not generally zero-shot and you have to fine-tune them to work for your use case.

Jev is promising a flexible zero-shot classifier competitive with frontier LLMs but at much lower cost and latency.

Experiment 1: a small pilot on Donners_2021

As a first test, I picked Donners_2021 from the SYNERGY dataset collection for the ASReview platform. It’s a small dataset of only 258 records, of which 15 studies were ultimately included.

An LLM helped me design and run an experiment. See the script it came up with 5 .

In short, I chose the “Noul” method, then in

State: The title and abstract of each record

Instructions: The typical instructions you give to humans or LLM on what meets the criteria

Criteria: This defines what is “true” = retain and “false” = drop

The response, also in JSON, returns the model’s probability for “true”. For example, p = 0.87 means Jev assigns a probability of 0.87 to the “retain” outcome. Whether this probability is truly calibrated is an empirical question. In a well-calibrated model, among predictions assigned p = 0.87, the “retain” outcome should be correct roughly 87 out of 100 times in the long run.

On paper, this probably isn’t the best way to do screening with Jev. I probably should have asked multiple questions in one call, but we will address this later.

But I wanted to begin with the simplest possible baseline.

Results of Experiment 1

With the setup above, Jev does show some potential.

Of the 15 records ultimately included, the lowest Jev score was 0.36. To achieve ≥95% recall, all 15 records must be retained, since retaining 14 of 15 would yield only 93.3% recall. Using a threshold of 0.36 (where you drop records <0.36) would therefore give ≥95% or 100% recall, a raw workload reduction of 45.7%, and a WSS@95 of 40.7%6.

However, 0.36 is a post-hoc oracle threshold: the cut-off was identified after examining the gold-standard labels, i.e., it was set knowing the absolute correct answer. It therefore represents the best observed workload reduction at 100% recall for this particular scored dataset, rather than a prospectively validated screening threshold.

A more realistic round cut-off of P < 0.30 for exclusion would still retain all 15 included records, giving 100% recall, while reducing the screening workload by 39.5%. This corresponds to a WSS@95 of 34.5%. However, unless 0.30 had been specified in advance, this result should also be regarded as retrospective rather than prospectively validated.

Is WSS@95 of 40.7% good?

Compared with active-learning systems, not particularly. A thesis using the same Donners dataset reported WSS@95 of around 67.4% using ASReview with logistic regression and TF-IDF.

But this is not a fair comparison. ASReview uses an active learning technique that involves iterative human feedback, unlike Jev, which is a zero-shot tool that requires neither human-in-the-loop feedback nor task-specific training/fine-tuning.

I therefore also compared Jev with zero-shot LLM screening results reported by Shen in 2026 using the same Donners benchmark: Framework 1 — Forced Binary. But on this Donners benchmark, several zero-shot LLMs outperform this Jev configuration at matched recall, though at greater cost and latency.

Taking a very natural cutoff of 50%, we notice that Jev is dominated by Llama 3.3 70B with lower recall (86.7 vs 93.3%) and specificity (64.2% vs 74.7%) and Mistral 3.1 24B with the same recall (86.7%) but lower specificity (64.2% vs 83.5%).

What if we adjust the cutoff threshold so Jev’s results have recall levels that match those of the LLMs?

As we can see, when Jev’s threshold is set so recall is 93.3% or 86.7%, to match Llama 3.3 70B/Gemma 3 27B and Mistral 3.1 24B, it has lower specificity.

Similarly, at a 73.3% recall level so it matches a fairly modern 2026 Claude Sonnet 4.6 (Feb 2026), its specificity is still 5 points below Sonnet.

Of course, Jev is doing this at essentially no cost (my dashboard shows $0.0002 for 258 records) and this is still essentially a small dataset, so let’s try something bigger.

Experiment two: eight CLEF-TAR 2019 DTA reviews (26K records)

For the second experiment, I wanted

a larger benchmark
a study that had fairly modern LLM screening results
a gold standard that could be easily obtained

I settled on the eight Diagnostic Test Accuracy reviews from CLEF-TAR 2019 Task 2, used by Akinseloyin 2024 and Akinseloyin 2026.

I used the exact same method as before 7.

To make things comparable, I followed Akinseloyin 2024 which did not use the full 30,521 records but removed entries with no abstracts, leading to 26,832 records and 423 labelled relevant.

The overall result was a strong WSS@95 of 0.661 at the cost of less than $1.50.

Below are the individual WSS@95 for the 8 test suites.

Jev doesn’t work perfectly for every test and you can see it struggles with CD012233, CD012768 (to a lesser extent CD012669). Interestingly, Akinseloyin 2024 also reports its lowest performance on the same two.

Still, as you will see, this simple approach with Jev beats or is at least competitive with a lot of other baselines with far more complicated methods.

Jev vs Active Learning vs LLMs

An interesting comparison comes from the original CLEF-TAR 2019 results. Unlike my earlier comparison using only records with abstracts, CLEF evaluated systems on the full set of 30,521 DTA records.

It is important to note that CLEF results were from continuous active learning or relevance feedback: human screening decisions were repeatedly fed back into the system so that it could update its ranking as screening progressed.

In contrast, Jev is zero-shot: it has no labelled examples and receives no feedback. Despite that, on the full 30,521 DTA records, it achieved WSS@95 = 0.631, close to the ≈0.66 achieved by Di Nunzio’s 2019 Padua continuous-active-learning system.

Now let’s look at Akinseloyin 2024 which studies the use of LLMs but also provides a series of baselines to compare for the 8 DTA datasets from older studies.

Jev’s WSS@95 of 0.661 easily beats older zero-shot efforts such as Sheffield-baseline (2019) and Wang et al. (2023) using BERT models as classifiers.

In fact, in terms of WSS@95, Jev beats all the experiments in Table 2 of Akinseloyin 2024 even ones using LLMs with Q&A approaches (see later) + embeddings.

More surprisingly, Jev also performs extremely well compared with the much more complicated LLM pipelines tested in that paper8.

First, let’s look at GPT_QA_soft_Both_ReRank.

Here’s the process

use LLMs like GPT-3.5 to break the review’s eligibility criteria into multiple questions (up to five)
for every candidate abstract, using an LLM to answer each of those questions;
use a BART model to estimate the probability that each answer is positive to obtain s_ik
for each question, calculating the cosine similarity between the embedding of the abstract and the eligibility question to get cos(D_i, Q_k),
averaging the scores from the last two steps - the BART score (s_ik)and cosine similaritycos(D_i, Q_k) - S’^ik
averaging these re-ranked question scores across all eligibility questions to get score S_i

This is followed by an overall cosine similarity between embedding of the original overall criteria and the abstract and this score is averaged with the score from the last step S_i.

Table 4 of that paper shows that their method can generate WSS@95 > 0.6, either by switching the model that does Q&A with Claude models and/or switching the embedding used. For example:

a) GPT-3.5 to do Q&A + Gemini embedding = WSS@95 = 0.630

b) Llama2 to do Q&A + Gemini embedding = WSS@95 = 0.636

c) Claude 3 Haiku to do Q&A + Gemini embedding = WSS@95 = 0.6539

Yet none of these configurations beats my simple Jev yes/no setup, which achieves WSS@95 = 0.661!

What about the newer 2026 results?

Akinseloyin 2026 asks whether the results can be improved by using an ensemble approach of three LLMs - GPT-4o Mini, Gemini 1.5 Flash and Claude 3 Haiku.

This is done again on the same 8 DTA datasets. Again it has a complicated method.

In short, it retains the basic QA-and-BART scoring framework from the 2024 study, but its Soft-Vote method replaces a single QA model with an ensemble of the three lightweight LLMs.

For each eligibility question, the BART-derived scores from the three LLMs are simply averaged.

This ensemble achieves WSS@95 of 0.680 on DTA without any embedding reranking. This is 0.019 better than Jev’s WSS@95 of 0.661 on the same dataset.

Adding the micro- and macro-level embedding reranking used in the earlier framework increases this further to 0.708.

It’s amazing how a simple and cheap method using Jev is competitive with pretty much everything but the ensemble approach using 3 LLMs. But the cost of using Jev here? Approximately $1.46! The striking part is how little machinery this required: one Jev judgement per record, for a total cost of about US$1.46.

Support independent testing like this

This post involved testing Jev across 26,832 records, reproducing published benchmarks and comparing several screening approaches.

If this kind of exploration and experimentation saves you time or helps you understand where new AI tools actually stand, you can help fund the next experiment.

I don’t accept payment from vendors for product coverage, and everything on Musings about Librarianship remains free.

Support independent testing

Experiment 3: Surely Jev should do better if I decompose the decision in Q&A style?

Remember the System One analogy from earlier? If Jev is designed to make fast, bounded judgements, then my original screening setup suddenly looks rather crude. I was asking it one huge question: does this paper meet all the eligibility criteria?

But evidence-synthesis screening can depend on population, intervention, comparator, study design and other conditions. TypeSafe’s own philosophy suggests that a complicated judgement like this should be broken into smaller independent questions.

The strongest approaches in Akinseloyin 2024 also use this kind of Q&A decomposition, where an LLM generates multiple questions based on the criteria, and each abstract is screened based on the overall outcome of all the questions.

The question on my mind is: if we use the LLM-generated questions but have Jev judge the answers, will we get even better results10?

This approach requires generating the questions first, which Jev cannot do itself.

Instead, I used the same eight cases with 26,832 records as before but now I also used the criterion-level questions (at most five available and generated by GPT-3.5) from Appendix A in supplementary content of Akinseloyin et al. 2024 verbatim.

For every title/abstract × every question, Jev made a separate structured Choice judgement:

POSITIVE
NEUTRAL
NEGATIVE

Roughly speaking

This meant I ran around 5 × 26,832 = 134,160 Jev calls. In my setup it cost <$5, however this was because I was being inefficient. With Jev, you can send multiple questions to the same shared state; this reduces cost and latency. In terms of input tokens you don’t need to send the title and abstract five times.

I then calculated Jev-QA-Hard by assigning Positive = 1, Neutral = 0.5 and Negative = 0, and averaging the scores across the questions.

This did badly with a WSS@95 of only 0.447 vs 0.661 for the simple-minded Jev approach. But perhaps such a scoring is too simplistic.

A better approach is Jev-QA-Expected, which makes use of Jev's probability output. It is calculated as 1 × P(Positive) + 0.5 × P(Neutral).

But even that only reached a WSS@95 of 0.5358. This performance is similar to GPT_QA_soft_Both_ReRank, which uses GPT-3.5 for question generation and answering, together with embedding-based reranking (WSS@95 = 0.556, see Table 3) but still much below the simple Jev approach of just asking one question (WSS@95 = 0.661) while requiring far fewer Jev judgements.

Why did this approach fail?

One possibility is that the questions generated by GPT-3.5 are the bottleneck. If these questions are poorly crafted, better judgements from Jev may not help much.

The other possibility is that the way I set up the criteria for Jev could be improved. There are also many ways I could formulate the criterion-level decisions.

For example:

Choice with Positive / Neutral / Negative is only one.
Perhaps Noul questions would work better.
Perhaps particular questions require different answer spaces.
Perhaps criterion-level probabilities should be combined using rules reflecting mandatory versus optional criteria instead of simply averaging everything.
Evidence-synthesis screening is not necessarily an additive scoring problem. One definite exclusion criterion might matter more than four positive criteria.

More testing is needed. However, this shows that decomposing a complex judgement into more model calls does not automatically improve screening performance.

Conclusion

So where does this leave Jev? My experiments are nowhere near enough to establish Jev as a safe automated screener.

But I think they show that this class of model deserves attention. In fact, as many have predicted, leading AI labs have also launched products in this general direction. For example, OpenAI just announced its Decisions API, which uses Luna for structured classification, routing and action selection.

Important limitations

First, these are retrospective benchmark experiments. The WSS values tell us how effectively the ranking separates known relevant from non-relevant records, but we do not know a priori when to stop.

Second, although the papers I used are fairly recent, publication delays mean they don’t use the latest LLMs, so the advantage of Jev may shrink if you use more modern models. While Opus, Sol, Fable, Astra will likely squeeze a bit more performance, there is much more potential with smaller, more modern ‘flash’-class models (Gemini Flash models?) that have reasonable performance at lower cost.

If this post was useful to you, you can support the next unnecessarily deep dive on Ko-fi.

Thanks for reading Aaron Tay's Musings about Librarianship! This post is public so feel free to share it.

Jev achieved 0.661 on the filtered benchmark, while on the full CLEF set it achieved 0.631 versus about 0.663 for the active-learning result.

I will ignore cross-encoders for now. Although pretrained relevance cross-encoders can be used zero-shot, they are usually trained or fine-tuned specifically for relevance ranking, making them a somewhat different comparison.

Information retrieval gurus of course understand another more important distinction that is happening when you use an LLM to evaluate or rank relevance vs typical lexical retrieval and dense retrieval. LLMs and cross-encoders allow rich, real-time interactions between the query and every document, whereas BM25 relies on term overlap and bi-encoders encode query and document independently. In both cases, query and document never truly ‘see’ each other; queries are compared only against pre-indexed representations of documents rather than interacting with them in real time.

Results are generally promising with overall high sensitivity rates and reasonable specificity but the variance in performance can be quite large depending on the task and it is still uncertain which is the best way to use LLMs to screen.

My original attempt mistakenly put the criteria in the state field which isn’t recommended. Interestingly, the revised method had a lower WSS@95 (40.7% vs 43.1%), but this isn’t conclusive for such a small sample size.

WSS@95 subtracts 5% from the raw workload reduction because, at a 95% recall target, simply not screening the final 5% of records would already produce a 5% apparent workload saving.

Similar to note 4, I initially ran with the less correct method. In this case the WSS@95 of the correct/revised method rose from 64.8% to 66.1%.

Interestingly, in their paper, when it comes to WSS@95, a relatively simple baseline, “cosine similarity between selection criteria and abstract based on GPT embeddings” scores a decent 0.60. This is fast and cheap compared to using LLMs for Q&A approaches.

Highest among the non-reranked ensemble methods.

In case you are wondering, this script does not make the same initial error as in notes 4 and 6.

注意转载授权 · 阅读原文