我搭了一套精巧的图书馆员推荐系统，结果被一个 CSV 打败了

Why, he asked, should a search about demographics return only books and articles? Why not also return a relevant guide or the name of a librarian who could help?

Dempsey pointed to the Stanford University Libraries and the University of Michigan Libraries as early examples. Their search environments brought together collections, library webpages, guides and staff profiles.

Michigan’s presentation of relevant subject specialists in a separate results pane was especially interesting to me.

That was more than a website design choice. It represented a different idea of what the library had to offer.

The collection was something to discover. But so was the library’s expertise.

Thanks for reading Aaron Tay's Musings about Librarianship! Subscribe for free to receive new posts and support my work.

Two ways of recommending library expertise

During the 2010s, I encountered several implementations of this general idea.

One approach was contextual recommendation through browser extensions. A library could configure a message to appear when a user visited a particular website or webpage. Lean Library’s Library Assist Module Messages and later LibKey Nomad++ Messaging Module can be used in this way: when a user encounters a relevant platform or problem, the extension can surface library resources, advice or contact information of librarians.

Recommendation by Lean Library’s Library Assistant Browser extension feature as at 2019

Clarivate’s newer Nexus Extend browser extension is the latest iteration of this idea. It is currently pitched as an embedded academic assistant that identifies potentially unreliable sources by checking references against trusted academic indexes including the Primo Central Discovery Index and Web of Science using MCP. But it also has the same broader goal of bringing library content and services (e.g SpringShare content which includes Libguides and chat service) into the environments where users are already working. Recommending librarians via Springshare librarian profiles is an obvious move.

A second approach was to include librarians directly in library search results.

One of the early implementations that impressed me was the University of Michigan Library’s bento-style search. A query could produce separate boxes containing catalogue records, articles, databases, guides and relevant librarians.

Trying it out today in 2026, shows the feature is still intact.

In 2010, it was stated that Michigan’s implementation was said to infer relevant subject specialists from the call-number ranges of highly ranked books and map those ranges to areas of librarian responsibility. Where no reliable match could be made, it fell back to a link to Ask a Librarian.

It was not perfect. A search could produce false positives, particularly when the returned books crossed disciplinary boundaries. But it was much more flexible than maintaining a list of literal query triggers.

Most libraries did not build anything like Michigan’s. The cheaper version of the same idea was to make LibGuides discoverable through the discovery layer, in products such as Summon. This could surface a guide containing the relevant librarian’s profile, although there was no guarantee that it would rank prominently enough.

Products later addressed this by introducing an explicit 'best bet' mechanisms that always appeared at the top or was otherwise displayed prominently alongside search results. Summon Best Bets and later Primo Resource Recommender allow libraries to create promoted results triggered by configured tags. Primo can recommend databases, websites, guides and contacts at the top of the results page.

These systems are simple and predictable. They are also fundamentally based on keyword matching.

The limitations of keyword-based recommendations

Suppose a finance librarian’s profile is tagged with:

Bloomberg
WRDS
Compustat
financial databases

When somebody searches for one of those terms, the profile appears.

That works reasonably well for named resources. But it does not generalise very far.

Should the finance librarian appear for every query concerning finance? Should a psychology librarian be recommended whenever a result set appears to concern psychology? Being the liaison librarian for a school does not necessarily mean that every search within that discipline requires a consultation.

Even where recommending a librarian would be useful, a keyword system must anticipate the user’s language. Someone looking for help screening studies might search for “Rayyan”, “evidence synthesis software”, “systematic review screening”, “deduplicating review records” or simply misspell the name of the product.

The larger the trigger list becomes, the more maintenance it requires and the more likely it is to fire for the wrong reasons.

This is where LLMs with agentic search1 appear to improve things greatly, at least in terms of relevance.

Adding librarian recommendations to a Primo MCP server

I have recently been experimenting with a home-built Primo MCP server. It allows an LLM to search our Primo discovery system, inspect the results, reformulate unsuccessful searches and decide what to try next. As I noted in a recent post, this enables “agentic search” and was particularly effective at solving a long-standing problem: users who could not find the right databases because they used the wrong keywords.

In one example from that post, I searched for Hoover’s, a business database to which we did not subscribe. The system found no matching subscription but suggested alternatives such as Orbis.

That made me wonder whether recommending another database was always the right next step.

Sometimes the more appropriate response might be:

We do not appear to have this resource. Here is the librarian who can help you identify alternatives or investigate access.

I therefore added a librarian-recommendation function to the Primo MCP server.

Now in this example, the user searched for “Rayvan”, a misspelling of Rayyan, the systematic-review screening tool. The agent correctly interpreted the intended product and recommended me.

The interesting part was that my profile was not explicitly tagged with “Rayyan”. It included broader expertise such as systematic reviews, evidence synthesis and Covidence.

I chose this example because we are preparing to launch an institutional subscription to Covidence, although it is not yet listed on our site. Knowing that users are looking for evidence-synthesis software would therefore be helpful.

A literal keyword matcher would probably have missed the connection. The LLM could recognise that Rayyan and Covidence are related tools used for similar stages of evidence synthesis, and that someone supporting systematic reviews was therefore a plausible person to recommend.

This produced recommendations that felt considerably more natural than traditional query-triggered profiles of the past.

But the route I initially took to get there was absurdly complicated.

The recommender I overbuilt

I began with a CSV containing approximately ten librarian profiles. Each record included fields such as:

name and contact details
subjects and aliases
keywords
schools or disciplinary responsibilities
functional expertise
resources or tools supported
a short “best for” description

For this experiment, I also needed a quick way to populate the profiles so I gave an LLM examples of the librarians’ writing and LibGuides and asked it to enrich the profiles. The results needed cleaning, but they were good enough for my pilot. Also importantly, the file was tiny, basically under 10 KB.

I then built a deterministic keyword recommender.

It normalised and stemmed the query. It considered not only the query but also metadata from the Primo results (a trick borrowed from Michigan’s implementation). Matches in the query received more weight than matches in result metadata. Fields such as “subjects” and “best for” received more weight than broad school affiliations2. Generic terms were suppressed. An IDF-style calculation reduced the influence of terms appearing across many profiles. Thresholds determined whether anyone should be recommended and how many profiles could appear.

Finally, I added an optional semantic fallback using Gemini embeddings. It ran when the keyword system found nothing or produced only a weak match. The current implementation also includes explicit exclusions, confidence thresholds and evaluation tools. The figure below shows what one version looked like.

The advantage of this system is that it is deterministic and fully explainable. It can explain which terms matched and where the evidence came from. It behaves consistently. The semantic component is less so, but it is optional and is used only when the keyword evidence is weak.

But it required a great many decisions about weights, thresholds, generic terms and fallback behaviour. Those decisions would require extensive testing and continued maintenance. In my experience, libraries rarely have the capacity to do this, and it was probably not worth the effort for this particular problem.

Then I tried something embarrassingly simple.

The embarrassingly simple baseline

I created a basic Microsoft Copilot agent that was instructed to recommend the most appropriate librarian.

I uploaded the same librarian-profile CSV as the knowledge source.

The whole process took less than 10 minutes to set the agent up but my informal testing suggested that the results appeared comparable to those from my custom pipeline, and sometimes better3.

I want to be clear that this was eyeballing a handful of queries, not a full-blown evaluation test. More on that later.

My first thought was that this worked because the LLM could just load the whole tiny CSV of profiles (under 10 KB) into the context window and, like a human, simply scan it for suitable matches without the use of a search engine or retriever.

Indeed, Claude Projects on paid plans appear to use an adaptive approach.4

When the uploaded material fits within the model’s available context window, Claude can place the project knowledge directly into context and consider it as a whole.
When the project knowledge becomes too large, Claude automatically switches to retrieval-augmented generation. It searches the uploaded files and brings only the most relevant passages into context for each query.

Also notably, Anthropic’s 2024 guidance states that for knowledge bases under roughly 200,000 tokens (about 500 pages), you can pass the entire dataset directly into the prompt without building a RAG pipeline.

Copilot Studio is less clear. It was not necessarily placing the entire CSV into the model’s prompt. Microsoft says when files are added as persistant knowledge sources, the uploaded files are chunked and semantically indexed as vector embeddings, after which the most relevant chunks are retrieved for the agent.

If this is the case, there is technically a retriever.

Either way, the practical lesson is the same:

I did not need to build my own specialised retrieval and scoring system. A generic knowledge-source mechanism (whether it placed the entire source document in the model’s context or used generic semantic chunking and retrieval), combined with a capable language model and a reasonably structured directory, was enough to produce plausible recommendations.

Even where a retriever supplies, say, the top twenty candidates from embedding search, the LLM serves as the final reranker and decides what to show. In systems where the directory can be supplied directly in context, even that internal retrieval step may be unnecessary. A directory of ten, twenty or perhaps several dozen short profiles is small enough for many current models to compare in a single prompt.

The exact practical limit will depend on profile length, model context, cost and the surrounding conversation. But the general scaling argument is clear.

Recommending from ten librarians is not the same retrieval problem as searching ten million articles.

Had I needed to recommend among 10,000 experts, I would probably need proper candidate retrieval, indexing and ranking.

With ten to a hundred librarians, I may not need a retriever at all, or can just use whatever is built in.

Where does the capability sit?

This illustrates a wider question in AI search: where does the system’s capability sit?

It can sit in the model: what the language model knows about disciplines, terminology, tools and relationships between concepts.

It can sit in the retriever: the corpus, metadata, index, matching functions, rankings and filtering rules.

Or it can sit in the harness: the prompts, tools, state and agentic loop that decide when to search, how to reformulate a query, what evidence to inspect and when to stop.

These components are not interchangeable, but they can compensate for one another.

A powerful model may recognise that “Rayvan” probably means Rayyan and that Rayyan is related to systematic-review support, even when neither connection is explicitly encoded in the profiles.

A strong retriever may place the right profile before a weaker model immediately.

A sophisticated harness may inspect both the user’s query and the Primo results, notice that ordinary resource recommendations are inadequate, and “urge” the system to keep trying or continue the loop.

In my original implementation, much of the capability sat in the retriever: hand-built matching rules, weighting, IDF adjustments, thresholds and embeddings.

In the simpler implementation, more of it sat in the model and the platform’s generic knowledge-grounding machinery.

The end result looked similar with less engineering needed. This is good news for librarians like me who generally have less experience and time to set up complicated retriever pipelines and test them.

But does recommending a librarian actually help?

There is still an important unanswered question.

A system can accurately identify the librarian whose profile most closely matches a query without the recommendation being useful.

Perhaps the user merely wanted the login page for Bloomberg5 and perhaps the system could have answered the question directly. Perhaps no librarian recommendation was needed at all6. At the other extreme, the model may keep providing increasingly speculative answers when it should have escalated to a person much earlier.

A useful librarian recommender therefore needs to make two decisions:

Which librarian is most appropriate, assuming there is one?
Should a librarian be recommended at all7?

The second may be harder.

In short, the system must be able to abstain. That is far more difficult to establish without testing.

A recommender that produces a plausible librarian for every query will look impressive in demonstrations and may become annoying in real use8. This is particularly true when my testing is biased9.

The decision also raises operational questions. Profiles must be maintained. Librarians should know how they are being represented. Broadly written profiles may attract a disproportionate number of recommendations. Someone’s formal liaison assignment may not reflect their actual expertise. Availability and workload may matter.

That’s why you will need an evaluation harness based on labelled queries (which my repository has). It can measure top-one accuracy, whether the correct librarian appeared anywhere in the returned list, and whether the system correctly declined to recommend anyone.

Note: Even a high accuracy on the test suite would measure recommendation accuracy as measured by the test, which is not necessarily the same as user value depending on how the labelled queries were created10. Ultimately, we would need to know whether users followed the recommendation, contacted the librarian and received useful help.

Again we see a trade-off. Using LLMs may give objectively better performance, but predictability and interpretability take a hit when it comes to predicting or controlling when the recommender should abstain.

The lesson is bigger than librarian recommendation

This experiment made me wonder whether I had stumbled across a more general change.

For years, one common and maybe only practical way (particularly for libraries without data or machine-learning expertise) of assigning objects to categories was to write Boolean rules.

A library might classify learning objects by matching words in their titles and descriptions to learning objectives. A university might identify research relating to the Sustainable Development Goals11 or some other research schema using lengthy search strategies. A support-routing system might assign a ticket to “data management” when it detected phrases such as “storage”, “data sharing”, “retention” or “data-management plan”. A discovery system might recommend a librarian when the query contained the names of databases associated with that person.

Of course, another big area of study is using LLMs for screening in evidence synthesis12.

The underlying pattern was the same:

Convert professional judgement into lists of words and Boolean conditions.

This could work very well, especially if it only suggested candidates and let the human make the final decision. It was transparent, deterministic and inexpensive to run at scale.

But it required someone to anticipate the many ways the same concept might be expressed. And some concepts resist this entirely. How would you write a keyword rule that maps a short learning-object description to the abstract goal of “critical thinking”13? I would say nearly impossible.

Today, there is another surprisingly viable baseline:

Give an LLM the object, the possible categories and clear descriptions of what those categories mean. Then ask it to assign.

Instead of encoding “critical thinking” through every tool name and synonym we can anticipate, we describe the expertise and let the model interpret the relationship.

This does not mean that LLM classification will automatically be accurate, though in my experience it beats naive Boolean keyword matching most of the time.

Important to note again, the choice isn’t binary. Other options include traditional supervised machine-learning classifiers, fine-tuned transformer models, and classifiers built on embeddings from BERT-family or other embedding models. Some of these may ultimately be more accurate, reproducible or cheaper at scale. But they also require labelled data and considerably more technical work. For a librarian without machine-learning expertise, giving an LLM the categories and asking it to classify the objects may now be the easiest route to a remarkably strong baseline.

Another advantage that only Boolean keyword matching provides fully is predictability and interpretability: why was this object assigned this category?

It does mean that it should increasingly be tested as a baseline before we spend weeks constructing elaborate keyword rules, embeddings, indexes and scoring systems14.

How small is “small enough”?

The phrase “small enough for the model to read” hides several different scaling problems.

The first is the number and complexity of the candidate categories.

Ten librarian profiles can probably be compared directly in a single model call. The same may be true of the 17 Sustainable Development Goals, particularly when each goal is accompanied by a clear description, examples and exclusions.

A taxonomy containing hundreds or thousands of closely related categories is more difficult. The model may no longer be able to consider the entire taxonomy reliably in one pass.

At that point, we may need retrieval, hierarchical classification or a specialist classifier to narrow the choices before asking the LLM to make the final decision.

The second scaling problem is the number and frequency of classification decisions.

Classifying ten learning objects once is trivial. Classifying ten million articles is not.

But even a small category set can become expensive when the classification is repeated continuously. A system may contain only ten librarian profiles, yet compare every user query done in a discovery layer against all ten profiles. If the service receives millions of queries, repeatedly placing the profiles into the model’s context can create significant costs and latency.

The important distinction is therefore not simply between a small and large collection. It is between a small and large workload.

A third consideration is whether the categories and objects change frequently.

If the librarian profiles change only occasionally, parts of the processing can potentially be reused. The profiles might be represented as embeddings, cached in the model prompt or converted into a compact decision structure. If the categories change constantly, a direct LLM approach may remain attractive because it avoids repeatedly rebuilding rules or retraining a classifier.

This gives at least a few broad architectures.

Small category set and low classification volume

Let the model compare all categories directly.

This may suit a pilot involving ten librarian profiles, a dozen research-support services, the 17 SDGs or a modest set of learning-outcome categories.

It is easy to implement and requires relatively little specialised infrastructure.

Small category set but high classification volume

A direct LLM comparison may still work semantically, but repeatedly sending the same category descriptions can become slow or expensive.

At this scale, it may be more efficient to use:

* a smaller specialist model;

* embeddings or rules for routine cases;

* cached or precomputed category representations;

* an LLM only for uncertain or ambiguous cases.

The directory may be small, but the workload is not.

Large or overlapping category set

Use retrieval, rules or hierarchical classification to produce a shortlist, then let the model choose among the candidates.

An embedding search might reduce 500 possible categories to the most plausible ten, which the model then compares more carefully.

Very large, hierarchical or high-volume problem

Use rules, embeddings or a trained classifier for most decisions, with the LLM reserved for ambiguous cases, quality checking or explanation.

The right architecture depends not only on accuracy, but also on category size, query volume, latency, cost, consistency and the consequences of a wrong decision.

A note about costs

There is also a basic difference in how costs are distributed when designing and operating these systems.

Boolean rules can require considerable expertise and time to design, test and maintain. Someone must decide which keywords to include, how they should be combined and how false positives should be excluded. Once the rules are in place, however, applying them to another object is extremely cheap.

LLM classification often works the other way round. It is much easier to begin because the categories can be described in ordinary language rather than converted into exhaustive lists of terms and conditions. But every classification consumes computation, time and usually money.

An LLM may therefore be economical when the classification volume is low, the categories change frequently or the cost of building a specialised system would exceed the cost of simply running the model.

Boolean rules, embeddings or trained classifiers may become more economical when the same task must be performed repeatedly at very large scale.

The key trade-off is not merely small collection versus large collection. It is upfront engineering cost versus repeated inference cost.

What you give up

There is a cost to the LLM baseline that deserves more than a caveat.

My deterministic recommender can tell you why it produced a result. It can name the terms that matched, the fields they matched in and the weights applied. It behaves consistently, and it can be tested against a labelled set. The Copilot baseline cannot do any of this in an auditable way. It produces a name, and the reasoning is not reconstructable.

For a librarian recommendation, or assignment of tickets in a support system, that may not matter much. For consequential classification it matters a great deal. If an LLM assigns publications to Sustainable Development Goals and those assignments feed research assessment, or routes support tickets in ways that carry accountability, “the model decided” is not an adequate answer to the question “why was this object placed in this category”. Boolean rules and trained classifiers (to some extent)15 give you a decision you can inspect, defend and reproduce.

For consequential classification, the system must support audit, challenge and correction. We need to know which evidence informed the decision, which definition of the category was applied, how borderline cases were handled and whether the same procedure can be reproduced. A fluent explanation generated after the event is not necessarily a faithful account of why the model made its choice.

This is something that some uses require and that the LLM baseline does not currently provide.

In short, we gain the ability to represent expertise in ordinary language, and to handle concepts that resisted keyword rules with less effort. But we give up interpretability and reconstructability.

Clear category descriptions, examples and boundary cases become more important, not less, because they are the only part of the system we can inspect and control. The work moves from predicting every possible phrase towards articulating what the category actually means, which is often easier for subject experts to write and maintain than long lists of Boolean triggers.

Thanks for reading Aaron Tay's Musings about Librarianship! This post is public so feel free to share it.

Full library discovery, reinvented?

The interesting part of this experiment is not that LLMs have finally invented librarian recommendation.

They have not.

Libraries were trying to make expertise discoverable alongside collections more than a decade ago. Dempsey’s argument for full library discovery remains recognisable today: users do not necessarily separate our collections, services and expertise as neatly as our websites do.

What has changed is not the ambition, but the interface and the matching mechanism.

Earlier systems tried to insert librarians and guides into catalogue or discovery-layer results. They required libraries to maintain keyword triggers, expose LibGuides to the index or infer subjects indirectly from retrieved records.

The emerging version seems to place the discovery layer inside the AI agent.

Clarivate’s Nexus Connect, for example, is an institutional MCP gateway intended to expose library collections and services inside general-purpose agents such as ChatGPT and Claude. Its initial use cases cover academic discovery and library-account services, but Clarivate says subsequent releases will add “library expert resources”, including research guides and teaching materials, together with services from other vendors.

In other words, full library discovery might be rebuilt as an agentic integration layer.

A user may no longer begin at the library homepage, select the discovery system and encounter a box recommending a librarian beside the search results. The user may ask an AI agent a question, and the institutional connector may decide whether to retrieve an article, recommend a guide, invoke a library service or surface relevant expertise.

That creates a larger version of the question raised by my small experiment:

Where will the intelligence that connects users to library expertise sit?

Will libraries supply structured descriptions of their people, guides and services while the vendor’s model performs the matching?

Will Clarivate or another intermediary determine how these resources are indexed, ranked and presented?

Will libraries be able to inspect why one guide or service was recommended while another remained invisible?

And will locally built MCP services offer libraries more control, or merely recreate functions that vendors can provide more conveniently at scale?

My librarian recommender was a small home-built demonstration. I may have overengineered its retriever because ten profiles were few enough for a capable model to compare almost directly.

But the larger strategic question is not whether my particular recommender needs an inverted index.

It is who will construct the layer through which the AI decides what the library has, what the library knows and when a librarian should enter the conversation.

Full library discovery began as an attempt to make the whole library visible through the search box.

It may now be returning through the agentic loop.

Buy me coffee via ko-fi!

Strictly speaking, an LLM’s strong ability to classify, evaluate and rank does not require an agentic system. The agentic element lies in deciding when to apply those capabilities, whether to search again, answer directly, recommend a resource, escalate to a librarian or abstain.

In this context, “subject” was a broad category that included functional expertise e.g., research data management, bibliometrics and evidence synthesis, as well as conventional disciplinary expertise such as economics. “School” was the department to which the librarian was assigned as liaison. The former is much more specific and hence more likely to be important to recommend.

This is quite amazing because Microsoft Copilot had only the CSV file, whereas my Primo retriever could also draw signals from the top five records returned by Primo. The broader lesson may be that, once an LLM can “look” directly at the material, fewer additional workarounds are needed to supply those extra signals.

The situation is more complicated for ChatGPT. OpenAI’s documentation for ChatGPT Enterprise describes a hybrid approach: some extracted text is inserted directly into the context window, while the remaining text is stored in a private search index.

From experience, we know Bloomberg and other finance databases or terminals are the trickiest to even log in to, and have interfaces that are finicky to use.

I have argued recently that improving ranking, rather than retrieval itself, is where “AI” has the greatest potential to improve search. A separate question is how a system should decide when no relevant answer exists.

Michigan took the second decision seriously. It only surfaced a specialist when enough highly ranked items fell into a single academic discipline, and where it could not make a reliable match it offered Ask a Librarian instead.

A wrong librarian recommendation costs the user’s time, a colleague’s time, and some of the credibility of the recommendation itself. Users work out quite fast whether “here is the person who can help” is a real signal or a distraction and learn to ignore it.

My “testing” that involves eyeballing recommendations will inevitably have a bias towards wanting the system to recommend a plausible librarian.

Do they measure what a librarian thinks is valuable? Or what a user thinks?

SDG mapping illustrates how classification has evolved from manually constructed lexical rules towards machine-learning and hybrid systems. After the United Nations adopted the 17 Sustainable Development Goals and 169 targets in 2015, early institutional initiatives translated their language into keyword lists and bibliographic search strategies. These subsequently developed into elaborate searches using phrases, field restrictions, Boolean operators, proximity rules and exclusions.

Elsevier says it has produced Scopus SDG search queries since 2018. The Aurora Universities initiative, which began in 2017, constructed 17 Scopus queries composed of 169 target-level subqueries. In an evaluation against sets of publications selected by participating researchers, Aurora’s version 5 queries achieved an average precision of 70% but recall of only 14%. This illustrates the familiar problem that carefully designed lexical searches can still miss relevant work expressed using unanticipated terminology.

From around 2020, prominent SDG-mapping systems increasingly incorporated supervised machine learning. Elsevier supplemented its expanded queries with a TF-IDF-based logistic-regression classifier, intended to add highly confident publications that the queries had missed. Elsevier reported that adding the machine-learning model increased recall by approximately 10%, while the resulting 2021 mapping maintained precision above 80%.

Aurora subsequently trained multilingual BERT models for the 169 individual targets using approximately 1.4 million Scopus publications labelled through its earlier query system. OpenAlex’s published SDG-classifier implementation similarly combines Aurora’s queries with an Aurora-trained machine-learning model. These developments did not simply replace Boolean searching: expert-built queries provided much of the training data and helped operationalise what counted as relevance to each goal. Several prominent operational systems are therefore hybrid, combining manually defined queries with machine-learning classification.

More recent research has tested embeddings and general-purpose LLMs. For example, Fankhauser and colleagues supplied models with article titles and abstracts together with descriptions of the 17 goals, and evaluated zero-shot and fine-tuned approaches. This makes direct LLM classification a plausible baseline, particularly because there are only 17 top-level labels. Importantly, while many research papers published as of 2024-2025 show LLMs are not necessarily superior to embedding-based methods or even supervised ML classifiers, these papers often use GPT-4 or even GPT-3.5-era LLMs which are far behind the state of art.

One issue is that there is no universally accepted ground truth for what it means for a publication to “contribute” to an SDG, which makes comparisons tricky. Systems may measure broad topical relevance, explicit engagement with a target, or intended societal impact, and Scopus, Web of Science and OpenAlex often disagree substantially in their assignments. One comparison using publications common to all three databases found that, for five examined SDGs, fewer than 8% of classified publications were assigned to the same goal by all three systems.

That said, because this is a task that is shared globally, the argument that it is easier to use LLMs than ML classifiers is relatively weak.

Results when using LLMs as screeners are promising, but performance still varies by model, prompt, review topic and screening stage, and the technology is developing rapidly. For formal systematic reviews, fully autonomous LLM screening should not yet be treated as the default unless its performance has been validated within the review and its use is transparently reported. It’s also unclear how to deal with the non-determinism. A more defensible current approach is to use an LLM as an assistant or second screener, with humans resolving disagreements and checking that relevant studies have not been wrongly excluded.

This is made even harder if all you have for the learning object is a short text description of it.

In my experience of classification and categorisation work, the biggest barrier to a good result is that there is massive disagreement even among highly skilled humans, which points to a fundamental issue that goes beyond the technology used. See also footnote 11 on SDGs.

Boolean rules provide an inspectable and reproducible decision path: we can identify which terms and conditions triggered the classification. Conventional ML classifiers vary in transparency, but a fixed and versioned model can usually provide more stable and reproducible outputs than a general-purpose LLM. Neither, however, automatically makes the decision defensible.

注意转载授权 · 阅读原文