AI 搜索正在重新打包图书馆所知的发现系统

We talk about search as though it were one reasonably well-defined activity.

Someone types words into a box. The system finds documents. Hopefully, the right ones appear near the top.

But there is no single professional tradition of search.

Instead, there are multiple communities that look at the same search process and see quite different things. They attend different conferences, read different literatures, use different vocabularies and, most importantly, disagree about what it means for search to work well.

Some are predominantly academic research communities. Some are practitioner professions, often within librarianship. Some are infrastructure ecosystems or engineering practices rather than disciplines in the conventional sense. They are not equivalent categories, and their boundaries are not clean.

Each tradition is included because it identifies a distinct class of search failure and a distinct criterion by which success should be judged.

This has always been true. But AI search is making it much harder to ignore, because activities that were previously distributed across separate layers and professions are increasingly being rethought, remixed and rebundled at the same time.

Share Aaron Tay's Musings about Librarianship

One search box, ten different things

In 2012, Lorcan Dempsey published an essay titled Thirteen Ways of Looking at Libraries, Discovery, and the Catalog. He borrowed Wallace Stevens’s conceit because the catalogue was, as he put it, a topic about which “we don’t yet have a single story”.

Rather than force everything into one model, he looked at the catalogue from thirteen angles: web scale, the single search box, metadata sources, discovery layers, knowledge organisation, sourcing and scaling, and more.

I have been thinking about something similar for academic search, but from a different angle of expertise.

What happens when ten (or maybe eleven) different kinds of specialists whose work or jobs impact search look at the same search box?

Two caveats before I begin. First, these are traditions, not boxes, and I will return to how much they leak into one another. Second, my descriptions may be somewhat reductive and the representative names under each tradition are merely illustrative, reflecting my own limited understanding of each area as a generalist and my own blind spots. No omission should be read as a judgement.

1. The information literacy librarian sees a critical human capability

For the information literacy tradition, search is primarily something people learn to do.

Can the user turn a vague question into useful search terms? Do they understand the difference between a database and the open web? Can they evaluate authority, recognise bias and avoid accepting the first plausible result? More recently, this has expanded into fact-checking, misinformation and the critical evaluation of AI-generated answers.

This is primarily a practitioner and pedagogical tradition, although it has a substantial academic literature.

Good search: produces a more capable and critical searcher.

Where they gather: LILAC, LOEX, ECIL and the community around the ACRL Framework.

Representative people: Alison Head, Barbara Fister, Alison Hicks, Sheila Webber, Mike Caulfield, Char Booth and Margy MacMillan.

Contribution to the stack: the human layer: evaluation habits, source criticism and the insistence that searching is a learnable intellectual skill rather than merely a product feature.

Potential blind spot: what happens inside the machinery after the user presses search1.

2. The information behaviour researcher sees an evolving interaction

People rarely begin with a perfectly formed information need. They learn what they are looking for by searching, reading, reformulating and sometimes abandoning their original question.

From this perspective, search is not a single transaction. It is an interaction among a person, a task, an interface and a changing state of knowledge.

This is why a system that produces an impressive answer to the first question is not necessarily supporting the complete information-seeking process.

Unlike information literacy (#1), this is primarily an academic research tradition. Information literacy tends to ask what people should learn to do, with librarians focusing particularly on just the academic context. Information behaviour more often studies what people actually do, why they do it and how their needs change. But both are closely related.

Good search: supports exploration and helps the user make progress, even when the original query is confused or incomplete.

Where they gather: ASIS&T, Information Seeking in Context and the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR), where this tradition and information retrieval (#8) actually talk to one another.

Representative people: Marcia Bates, Carol Kuhlthau, Nicholas Belkin and, among contemporary researchers, Chirag Shah.

Contribution to the stack: models showing that information needs evolve during a search, and that sessions and tasks, rather than isolated queries, are often the real unit of analysis.

Potential blind spot: its insights into evolving needs are frequently ignored by vendors or at best reduced to interface features (e.g. citation trails, facets) rather than shaping the underlying retrieval process.

3. The evidence-synthesis specialist sees a research method

For evidence synthesis, search is not merely a convenient way to find several good papers. It is part of the research method.

The search must be sensitive, transparent and defensible. Concepts must be represented adequately. Appropriate databases must be selected. Controlled vocabulary and free-text terms must be combined. The strategy should be reported, reviewed and tested against known relevant studies.

This is mostly a professional and methodological tradition2, especially among information specialists, systematic reviewers and health librarians. But it also has a growing research community studying search methods, automation and the validity of different retrieval approaches3.

Good search: retrieves a sufficiently complete body of evidence and leaves an audit trail.

Where they gather: the Cochrane and Campbell colloquia, ESMARConf, EAHIL and the health library associations.

Representative people: Wichor Bramer, Siw Waffenschmidt, Julie Glanville, Farhad Shokraneh and Dean Giustini.

Contribution to the stack: probably the strongest culture of methodological accountability in search: PRESS peer review, PRISMA-S reporting, validated filters and the radical idea that a search strategy is a citable research object.

Potential blind spot: the field has traditionally focused on auditing search strategies far more closely than the ranking systems applied to their results. This made sense when every retrieved record was screened and ranking affected only the order in which records appeared.

Evidence-synthesis specialists have therefore been understandably cautious about relevance ranking. PubMed’s machine-learning Best Match algorithm has existed since 2017 and became the default in 2020, but using it in a systematic review remains controversial even after almost a decade4. As a result, experts from this tradition may underestimate the value of better ranking using more modern retrieval techniques.

That said, the field is becoming more comfortable with ranking for screening prioritisation and technology-assisted review, particularly where every record remains available for screening or where safeguards make the risk of missed studies explicit and testable5.

This is one of the clearest overlaps with the information retrieval community (#8): evidence synthesis imports active learning, screening prioritisation and stopping methods from IR, but subjects them to unusually demanding expectations for recall, transparency and reproducibility.

4. The metadata and knowledge-organisation specialist sees a representation problem

Search engines do not retrieve the things themselves. They retrieve representations of things.

Is this the same author or two people with similar names? Are these records different works, different versions or duplicates? How should a subject be expressed? Which relationships among authors, institutions, publications and concepts have been captured?

Embeddings do not make these problems disappear. They may simply hide them until a system confidently merges the wrong author, work, institution or version.

This is predominantly a professional tradition rooted in cataloguing, knowledge organisation and bibliographic control, although it also has a strong theoretical and research wing.

Good search: depends on entities being represented accurately and consistently.

Where they gather: DCMI, SWIB, IFLA cataloguing meetings and the linked-data-for-libraries community.

Representative people: Karen Coyle and Ashleigh Faith6.

Contribution to the stack: identity itself: authority control, work modelling and the disambiguation that stops a system merging two authors who happen to share a name.

Potential blind spot: metadata quality is still too often judged through conformance to standards rather than demonstrated effects on retrieval, discovery and user outcomes.

5. The open scholarly infrastructure specialist sees what machines are allowed to know

Search quality depends partly on what information is available for machines to process. In academic search, this boils down to questions like:

Are abstracts open? Are references available through open citation data? Do publications have DOIs? Are authors connected through ORCID and institutions through ROR? Can repository records be harvested and reused? Can a commercial system legally index and display the relevant content?

A supposedly intelligent search system cannot use signals it cannot access. Famous and well-documented resources tend to perform better partly because information about them is widely available. Content hidden behind contractual, technical and metadata barriers remains harder to discover.

This is less a single profession than an ecosystem. It includes persistent-identifier organisations, standards bodies, librarians, publishers, technologists, policy researchers and advocates for open science.

Good search: requires scholarly information that is machine-readable, interoperable, reusable and sustainably governed.

Where they gather: FORCE11, PIDfest, Crossref events and the communities around ORCID, ROR, DataCite, OpenAlex, OpenCitations and COAR.

Representative people: Geoffrey Bilder, Cameron Neylon, Jason Priem, Bianca Kramer, Martin Fenner, Kathleen Shearer, Ginny Hendricks, Maria Gould and John Chodacki.

Contribution to the stack: the identifiers, relationships and open corpora on which most academic AI search tool silently depend.

Potential blind spot: treating openness as the finish line. Getting data out is not the same as getting it found, ranked or used well.

6. The bibliometrician sees a scholarly graph

The scholarly literature is not merely a bag of documents. It is a network of authors, institutions, citations, journals, concepts and research communities.

Citation links, bibliographic coupling, co-citation and co-authorship patterns can help people navigate the literature, identify related work and locate emerging fields. They can also reproduce existing prestige, language, disciplinary and geographical biases.

This matters increasingly because academic search tools do not rely only on textual similarity. Many also exploit citation graphs, publication venues, author relationships and other scholarly signals.

Bibliometrics and scientometrics are predominantly academic research traditions, although their methods are widely applied in research evaluation, university management, funding and commercial analytics and librarians are involved in these areas. Search is not the field’s only or even necessarily its central concern, but its models of the scholarly graph increasingly shape discovery and recommendation.

Good search: uses scholarly relationships intelligently without confusing visibility with relevance or citation counts with quality.

Where they gather: ISSI , STI Conference, LIS-Bibliometrics Conference, Bibliometrics and Research Impact Community (BRIC) Conference ; Scientometrics and Quantitative Science Studies on the journal side.

Representative people: Eugene Garfield, Ludo Waltman, Cassidy Sugimoto, Nees Jan van Eck, Mike Thelwall, Paul Wouters, Loet Leydesdorff and Philipp Mayr.

Contribution to the stack: the graph signals behind citation searching, related-paper recommendation and much of what makes academic AI search feel clever.

Potential blind spot: bibliometric signals developed for analysis or navigation can be absorbed into rankings and recommendations without the cautions that bibliometricians themselves attach to them.

This group also overlaps highly with open scholarly infrastructure (#5) community when they advocate for open references and metadata e.g I4OC, I4OA, Barcelona Declaration on Open Research Information

7. The e-resource librarian sees an access and fulfilment chain

Finding the perfect article is not much use if the user cannot open it.

Behind the friendly Full Text Available label lies a chain of machinery comprising knowledge bases, holdings, link resolvers, proxies, federated identity, single sign-on, entitlement checks, open-access routing and publisher platforms.

Retrieval asks:

What should I read?

Fulfilment asks:

Can I actually read it?

Users do not care that these are managed by different systems or departments. A broken link means that, from their perspective, the search failed.

This tradition therefore sits downstream of retrieval in a technical sense but firmly inside the end-to-end discovery experience.

Good search: ends with the user obtaining an appropriate and accessible version of the resource.

Where they gather: ER&L, NASIG, UKSG, the Charleston Conference and communities around Access initiatives and services such as SeamlessAccess, GetFTR and OpenAthens.

Representative people: Herbert Van de Sompel, Todd Carpenter, Heather Flanagan, Roger Schonfeld, Lisa Janicke Hinchliffe, Jill Emery and Graham Stone.

Contribution to the stack: the switching layer: OpenURL, knowledge bases and entitlements and the machinery that turns a citation into a document that opens.

Potential blind spot: it can establish whether a link worked, but not whether the right things were found in the first place.

Many people in this group also overlap with open scholarly infrastructure (#5), including through service on persistent-identifier and standards boards.

8. The information retrieval researcher sees a ranking and evaluation problem

For the information retrieval researcher, search is something that can be modelled and tested.

How should queries and documents be represented? Should the system use BM25, dense embeddings, learned sparse retrieval, late interaction or a hybrid pipeline? Does reranking improve the results? What happens to recall, precision, Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG) on a test collection7?

Good search: performs better against defined relevance judgements under controlled evaluation.

Where they gather: SIGIR, ECIR and CIKM, with The Text Retrieval Conference (TREC) and Conference and Labs of the Evaluation Forum (CLEF) as shared evaluation infrastructures.

Representative people: Karen Spärck Jones, Stephen Robertson and Ellen Voorhees; contemporary researchers include Jimmy Lin, Omar Khattab and Nandan Thakur.

Contribution to the stack: the measurement discipline itself, together with the indexing and ranking models that run from probabilistic retrieval through dense and late-interaction systems.

Potential blind spot: benchmarks rarely reproduce the awkward reality of library discovery: licensed collections, inconsistent metadata, long research tasks and users who cannot clearly state what they need.

TREC belongs firmly to this tradition. Its classic model is almost the canonical expression of academic information retrieval: shared documents and topics, ranked runs submitted by participating systems, pooled relevance judgements and common evaluation measures.

TREC has hosted tasks far beyond traditional document ranking, but it has generally brought them into the IR evaluation paradigm rather than becoming a separate tradition of its own.

Information retrieval also overlaps significantly with information behaviour (#2). Interactive information retrieval studies what happens when the neat benchmark abstraction of one query, one ranked list and one set of relevance judgements meets an actual person with an evolving information need.

CHIIR is one place where information behaviour and information retrieval genuinely meet, although the border between studying the user and optimising the system remains surprisingly durable.

Most librarians no longer have much association with this camp. A typical librarian might remember hearing about TF-IDF once in their MLIS class. There are exceptions, of course, notably in continental Europe.

9. The relevance engineer sees a production system

This is the practitioner counterpart of information retrieval (#8).

Industry search practitioners ask a somewhat different question from academic information retrieval researchers.

They want to know why a live, usually enterprise-scale system is failing actual users.

Which queries return nothing? Which produce bad rankings? How should relevance be judged? Did the new hybrid pipeline improve results or merely add cost and latency? Can the team diagnose a failure and make the system better next week?

The IR researcher (#8) asks whether a ranking method improves nDCG on a benchmark.

The relevance engineer (#9) asks why users searching for red shoes on Tuesday are seeing nonsense and how to fix it before Friday.

The underlying concepts overlap considerably. The professional settings, constraints and feedback loops do not.

Libraries teach search. We configure search. We procure search. We sometimes survey users about search.

Today, most libraries have much less of a professional culture around relevance engineering: systematically judging, diagnosing and improving the quality of live search results8.

Relevance engineering presupposes access to query logs, relevance judgements and the ranking configuration itself. In most library discovery products, the vendor holds all three.

Libraries should borrow the engineering discipline without blindly adopting its commercial objectives. An e-commerce system may optimise for conversion. A library should not.

Good search: can be measured, diagnosed and continuously improved in production.

Where they gather: Search Solutions, Haystack, Berlin Buzzwords, and the relevance Slack channels where much of the practical craft circulates.

Representative people: Tony Russell-Rose, Martin White, Charlie Hull, Doug Turnbull, Trey Grainger, Daniel Tunkelang, Jo Kristian Bergum and Lettie Y. Conrad.

Contribution to the stack: judgement lists, offline and online evaluation and the discipline of treating a bad result as a debuggable failure rather than a fact of life.

Potential blind spot: its methods often assume query volume, instrumented systems and rapid feedback loops that library search rarely possesses.

Search UX consultants9, publishing product managers and relevance engineers do not form one completely coherent community. Search UX also overlaps with information behaviour (#2), while product managers may sit closer to library discovery (#10). The centre of gravity here is the engineering practice of evaluating and improving a live retrieval system.

10. The library discovery professional sees an integrated service

Finally, the discovery librarian or product owner who might try (whether they know it or not) to combine all these elements that are colliding within the library’s main search system.

Primo, Summon, EBSCO Discovery Service and similar products10 combine metadata from many sources, retrieve and rank records, present facets, calculate availability and send users through different access pathways.

They are the default owners of the library search stack and often the people most responsible for how users perceive library search through the one search box on the library website.

Unfortunately, they may be expected to integrate components they cannot properly inspect, evaluate or control.

Good search: delivers a coherent end-to-end service.

Where they gather: ELUNA and IGeLU, Code4Lib and vendor platforms such as the Ex Libris Idea Exchange, where enhancement requests go to join a queue.

Representative people: Ken Varnum, Christine Stohn, Lorcan Dempsey and Marshall Breeding.

Contribution to the stack: integration and translation: the tradition whose job is to make the other nine behave as one service.

Potential blind spot: it sees the system through the vendor’s administration console. What the console does not expose, the profession gradually stops asking about.

People in this group are mostly library-systems specialists and often overlap with metadata (#4), e-resource management (#7) and to a lesser extent scholarly infrastructure (#5). In my experience, however, they are less likely to be deeply embedded in information literacy (#1) or evidence synthesis (#3).

This creates gaps between librarians who are largely front-of-house and those who are largely back-of-house. The people teaching users how to search may have limited understanding of how the system retrieves and ranks11. The people configuring the system may have limited contact with users even librarians who are attempting complex searches beyond seeing such queries in a search log.

These are traditions, not boxes

There is an obvious problem with dividing search into ten traditions.

The traditions do not stay politely in their assigned boxes.

Some are intellectual descendants of others. Some pair academic research communities with practitioner counterparts. Others overlap so heavily that the placement is partly a matter of convenience.

They are also not all the same kind of entity. Information behaviour is an academic research field. E-resource management is a professional specialism. Open scholarly infrastructure (#5) is an ecosystem. Relevance engineering is an engineering practice.

Linked data is the clearest example of why the boxes leak.

It grew largely out of the metadata and knowledge-organisation tradition (#4). Authority files, identifiers and relationships among works, people, organisations and subjects existed long before anyone called them knowledge graphs.

Once those identifiers and relationships travel between catalogues, repositories, publishers, funders and research information systems, linked data becomes scholarly infrastructure (#5). ORCID is not merely a well-formed metadata field. ROR is not merely authority control for institutions. Crossref is not merely a catalogue of DOI records.

And once the resulting network is analysed or exploited for retrieval and recommendation, it overlaps with bibliometrics (#6) and information retrieval (#8).

OpenAlex is difficult to assign to any one tradition. It is simultaneously open scholarly infrastructure (#5), a scholarly knowledge graph and a source for bibliometric analysis (#6).

What unites the traditions is not institutional form. It is that each brings a distinct definition of the search problem and a distinct theory of what counts as success.

Everyone is right, and everyone is missing something

Read back through the ten definitions of good search:

None of these definitions is wrong.

None is sufficient on its own.

The problem is not that librarians and adjacent professionals know nothing about search. There is an extraordinary amount of expertise.

The problem is that this expertise is fragmented across communities that do not share a common vocabulary, evaluation framework or model of the complete system. Often they are not even aware of each other and the lessons they can teach each other.

AI search is rebundling the layers

This fragmentation was manageable when the search stack was comparatively stable and search primarily returned lists of records.

Different communities could concentrate on different layers. Practitioners worked on teaching query formulation, metadata, fulfilment and the integrated service. Researchers studied information behaviour, scholarly graphs and ranking algorithms.

The boundaries were never perfect, but the division of labour made sense because the layers were reasonably visible and the foundations were stable.

An information behaviour researcher could study how people formulated and reformulated their needs without understanding the details of BM25, a standard lexical retrieval and ranking algorithm used by many systems.

Today, that researcher may need to understand whether the system uses lexical retrieval, dense retrieval, learned sparse retrieval, late interaction or some hybrid (which affects what search queries work), and whether it generates a direct answer rather than simply returning a ranked list. A practitioner teaching information literacy faces the same complexity when trying to work out exactly what “AI-powered search” means and which component might affect learning.

AI search does not abolish these layers. Metadata remains metadata. Retrieval remains retrieval. Access remains access.
What AI search does is rebundle them into systems in which interpretation, retrieval, relevance assessment, evidence selection and synthesis repeatedly call one another in novel and different ways rather than in the same standard way.

A tsunami of change in retrieval since 2018

The family of technologies that followed the transformer architecture is changing many of these layers simultaneously12.

The major inflection point was the introduction of that architecture in 2017. From 2018 onwards, encoder models such as BERT demonstrated how effectively contextual language models could be applied to retrieval, while GPT-style decoder models showed increasingly capable generation. Retrieval-augmented generation brought the two together explicitly in 2020.

Even into the early 2020s, while academic researchers in information retrieval and NLP (#8), selected industry research teams and a handful of start-ups (#9) were excitedly experimenting with transformer-based approaches to search, the public, including most librarians, remained generally unaware of the scale of the changes coming their way13.

Changes such as Bing and Google incorporating BERT and related transformer models into parts of their query-understanding and ranking systems in 2019 happened largely under the hood and went mostly unnoticed by users, except in the SEO community, because search engines continued to present the familiar list of results.

The launch of ChatGPT on 30 November 2022 triggered a much more visible awakening. The focus was no longer merely on using transformer models to improve matching and ranking, but on using generative models to produce answers directly. That is the point at which every other community became aware that the foundations of search were shaking. More recently, we have seen the rise of “Deep Research” and “agentic search” in 2024/2025.

A language model may now interpret the user’s question, translate it into several searches, assess and rerank what comes back, extract and reconcile evidence, compose an answer, and then decide it lacks enough evidence and search again.

These are not merely improvements to the ranking algorithm. Activities that once occurred before retrieval, during retrieval and after retrieval are now orchestrated inside the same model-and-harness system.

The query is no longer necessarily a query. It may be a conversation, a research objective or an underspecified task that the system must progressively interpret.

The result is no longer necessarily a result list. It may be a direct answer, a literature map, a table of extracted findings, a generated report or a set of claims linked to supporting passages.

Ranking is no longer necessarily a single operation performed once against a fixed corpus. Documents may be retrieved, judged, discarded, searched within, compared against one another and used to generate further searches.

Relevance itself becomes more complicated. A document may be useful not because it directly answers the original question but because it supplies a missing entity, suggests a better query, contradicts an emerging conclusion or points the agent towards another source.

The old layers still exist. But they are now more tightly coupled and recursively invoked.

Everyone sees part of the problem. Nobody automatically owns the complete chain.

The emerging eleventh community: RAG and agentic search engineering

There is now an eleventh community in the room14.

It is not yet a stable professional tradition in the way that information retrieval or knowledge organisation is. It is better understood as an emerging coalition of NLP researchers, IR researchers, model providers, software engineers, framework developers and application builders.

They gather around retrieval-augmented generation, tool use, deep research and agents rather than catalogues and indexes.

They borrow the language of information retrieval, including recall, precision and nDCG. But they also introduce different definitions of success: faithfulness, groundedness, citation correctness, answer relevance and task completion.

Is this eleventh community simply information retrieval (#8) and its industry or enterprise counterpart (#9) in disguise?

Maybe. ColBERT was published at SIGIR. The TREC Deep Learning Track evaluated neural rankers, and TREC inaugurated a separate RAG Track in 2024. Researchers such as Jimmy Lin, Omar Khattab and Nandan Thakur are clearly from the IR tradition.

But at least as much of the inheritance comes from natural language processing. Dense passage retrieval, open-domain question answering and the original RAG paper emerged substantially from NLP research rather than conventional search engineering15. The original RAG work explicitly combined the parametric memory of a pretrained sequence-to-sequence model with a non-parametric dense index.

Good search: produces an answer or research output that is useful, supported by evidence and faithful to the retrieved sources.

Where they gather: arXiv, NeurIPS and ACL on the research side; framework communities such as Haystack’s Discord, LangChain meetups and vendor communities such as the OpenAI Developer Community on the industry side.

Representative people: Patrick Lewis from the original RAG paper, together with figures such as Jerry Liu from LlamaIndex as well as many of the same names from Information Retrieval (#8) and relevance engineering (#9).

Contribution to the stack: the generation and orchestration layers, and the reframing of retrieval as one component inside a larger reasoning and tool-use loop.

Potential blind spot: treating retrieval as a solved commodity and repeatedly rediscovering, at considerable cost, lessons that information retrieval researchers and search engineers have accumulated over decades16.

But what does this mean for libraries?

The emerging discovery architect or committee team

Nobody can become a deep expert in all eleven traditions17.

That would be an absurd job description18.

So let me be clear about what I am proposing. The discovery architect is not a job title to put in the next vacancy advertisement. It is a capability, and in most libraries it will be distributed across a team rather than embodied in one person.

What the capability requires is that somebody, or most likely several somebodies between them, understands that the different traditions exist, recognises how they fit together and can identify when a particular form of expertise is missing, rather than viewing academic search narrowly through one or two traditions.

Where does this sit organisationally? The obvious home is wherever discovery, systems and collections expertise already meet. In some libraries that is a discovery or systems team or community. In others it may be a research support or scholarly communication unit that has drifted into AI evaluation work. The label matters less than the mandate: someone must own the question of whether search is working, end to end, and no single existing role currently does.

It’s fairly common for academic libraries to have a discovery committee or taskforce. But, no matter what the team is called or how it is organised, most libraries will lack expertise that spans all the communities with current staff expertise e.g. Information retrieval (#8) or Relevance Engineering (#9). This is not necessarily a call to go hire such expertise but rather a call to broaden one’s understanding.

The conventional discovery professional operates and integrates the present library discovery service.

The discovery architect capability spans the broader AI-mediated scholarly search environment: discovery indexes, scholarly graphs, metadata, access systems, search engines, AI answer systems, retrieval tools and agentic workflows.

Not merely a Primo administrator.

Not merely a literature review search trainer.

Not merely a metadata librarian.

Not merely an AI enthusiast who has learnt to spell RAG.

The discovery architect team as a whole should be able to move across the scholarly search stack, translate among professional communities and ask the questions that individual systems and specialists may overlook.

When an AI search product performs badly, this team should be able to ask:

Is the problem corpus coverage?
Is it missing or poor metadata?
Is it entity disambiguation?
Is it retrieval?
Is it ranking?
Is it the search depth or stopping rule?
Is it a distorted graph signal?
Is it poor evidence selection?
Is the generated claim unsupported?
Is the source inaccessible?
Is the model relying too heavily on its own parametric knowledge?
Are we evaluating the wrong outcome?

They do not need to solve every problem themselves.

They need to recognise what kind of problem it is and which tradition’s expertise is required19.

Diagnosis without access is speculation

There is an important caveat.

Recognising the type of problem is not the same as being allowed to examine it.

Asking whether the fault lies in ranking achieves little if nobody outside the vendor can inspect the ranking.

The discovery architect team therefore carries a procurement agenda as well as a diagnostic one:

access to query logs;
the ability to conduct relevance judgements;
documentation of indexed coverage;
transparency about ranking signals;
visibility into retrieval and evidence-selection stages;
stable interfaces for evaluation;
the ability to export or audit intermediate results.

If vendors will not open these layers, the eleven traditions can meet, exchange vocabulary and produce sophisticated hypotheses.

But they can only speculate together.

Buy me coffee via ko-fi!

Coming in Part 2

There is a question underneath all of this that the eleventh community is only beginning to ask, and that the rest of us should be asking constantly.

When an agentic search system performs well, where does the capability actually live? In the model, in the retriever, or in the harness around them?

In Part 2, I will take up that question, look at what controlled benchmarks such as BrowseComp-Plus are starting to reveal about it, and ask what it means for the practical question libraries face: when a vendor shows us an impressive agentic search system, what exactly are we licensing?

Subscribe now

To be fair, I estimate that 30–50 per cent of this blog’s readership is in this community or tradition, showing eagerness to learn more.

Probably the second-largest community of people reading my blog.

There is growing interest in SWAR (Study Within A Review) methodology , including applying it for AI tools.

PubMed itself says that PubMed Best Match feature was not designed for comprehensive or systematic searching

The current Cochrane Handbook has a section that recognises a role for relevance ranking in screening prioritisation and other forms of technology-assisted review.

This brief list of people reflects my limited depth of understanding in this area.

Evidence synthesis (#3) often treats searching and screening as binary classification. Sensitivity corresponds to recall and PPV to precision; specificity and NPV measure how reliably irrelevant records are excluded. Other metrics used include F1, and Work Saved over Sampling at 95% recall (WSS@95). Evidence synthesis generally prioritises high sensitivity and safe exclusion, whereas information retrieval (#8) often evaluates ranked order using MAP, MRR (particularly useful when only the first relevant result matters) and NDCG.

To be fair, many of the systems we manage have traditionally neither allowed nor encouraged us to tweak the parameters that affect relevance ranking. Though Ex Libris’s Primo does allow this to some extent, most libraries just accept the default settings.

There is a broader library UX (LibUX) community, most notably Andy Priestner, Aaron Schmidt and Ned Potter, though this is not specific to search. Some members, such as Matt Borg and Matthew Reidsma, have also studied user search behaviour

A minority of academic libraries run open-source solutions such as VuFind and Blacklight.

From personal experience, many information literacy librarians outside my institution have asked me how Primo and Summon work.

Before transformer-based models reshaped retrieval, leading search systems in the 2010s typically combined lexical retrieval with learning-to-rank, enriched by behavioural, entity and sometimes knowledge-graph signals. Neural ranking models were already emerging, but BERT and later transformers produced a much larger shift in how queries and documents could be represented and matched.

I remember seeing such approaches emerge when researchers created search engines over the COVID-19 Open Research Dataset (CORD-19), using BERT-style models for retrieval and ranking alongside early question-answering and generative components. In the early 2020s, some librarians were becoming aware of citation-based mapping services such as Connected Papers and ResearchRabbit, which belong to a different class of tools.

Dempsey had thirteen ways. I have stopped at eleven (ten plus an emerging one), and I am confident readers can supply the rest. Some candidates I considered and left out: archival description and discovery is a genuine tradition of its own, with finding aids, EAD and a rather different idea of what a record represents. Expert search outside health, particularly patent and legal search, shares the evidence-synthesis community’s obsession with recall and defensibility but almost never talks to it. And the web SEO community, now busily reinventing itself as generative engine optimisation, looks at search entirely from the supply side. It is only a matter of time before researchers ask about the academic-search version: not how to find things, but how to be found. There are others, of course.

When I was trying to read frontier papers about “generative search”, I was often confused because the NLP and information retrieval communities seemed to be writing about many of the same technologies from different angles. This was especially clear in work on tasks such as question answering. This has been written about fairly frequently in the literature -see here for historical discussion up to 2020 and here for a more technical and up-to-date comparison of Information retrieval to “AI” (mostly NLP).

The clearest example is the rediscovery of lexical baselines. Early RAG builders reached instinctively for dense embeddings alone, only to find that hybrid pipelines including boring old BM25 were hard to beat, a result the BEIR benchmark had already demonstrated for out-of-domain retrieval.

I am a generalist, but my knowledge does not span all these communities. Early in my career, I studied and led the implementation of library discovery systems (#10). Since then, I have developed a largely self-taught interest in information literacy (#1), evidence synthesis (#3), bibliometrics (#6) and information retrieval (#8). I have also been somewhat involved in open scholarly infrastructure (#5) and, to some extent, the access and fulfilment chain (#7).

I know there are librarians with “discovery” in their titles, but they are likely to be mostly in community #10 and #4.

But interdisciplinary search governance requires more than classification. The traditions may recommend incompatible outcomes:

Evidence synthesis may privilege recall over ranking quality.
Relevance engineering may optimise measurable user behaviour.
Information literacy may prefer productive friction over immediate answers.
Open infrastructure may prioritise openness even where commercial sources currently offer better coverage.
Bibliometric signals may improve navigation while reproducing accumulated prestige.

Somebody must decide which conception of good search governs a particular task. That normative and political function is currently implicit.

注意转载授权 · 阅读原文