Retrieve-then-Disambiguate: Scalable Reviewer Discovery

   Joe Petrecca Affiliation: Michigan State University    Wolfgang E. Kerzendorf Affiliation: Michigan State University

Abstract

The growing number of scientific submissions is straining the peer review process, while a large pool of relevant researchers remains unidentified and is therefore not considered as potential reviewers. We formulate the discovery of these researchers as a retrieval problem: given a submission, we retrieve and rank researchers based on domain similarity. We present retrieve-then-disambiguate, an open-source algorithm that identifies domain relevant researchers for a query submission by retrieving similar publications from the scientific literature and resolving ambiguous author names into unique researcher identities. We evaluate the algorithm on 500 astronomy and physics publications. We use ORCID-linked first authors, coauthors, and cited researchers as proxy labels to evaluate the recovery of researchers with known connections to each query publication. When retrieving the top 500 most similar papers, the algorithm recovers roughly 68% of first authors, 60% of coauthors, and 45% of cited researchers. Our code is available at https://github.com/deepthought-initiative/Retrieve-then-Disambiguate.

1 Introduction

The exponential rise in submissions to conferences, journals, and funding agencies has placed growing pressure on the peer review system (Merrifield and Saari, 2009; Leyton-Brown et al., 2022; Hanson et al., 2023). Advances in large language models have accelerated this rise, for example, with NeurIPS submissions growing from 9467 in 2020 to 21 575 in 2025 (, 2025). In response, scientific organizations are deploying automated systems across the peer review process, for example, to assign peer reviewers (Charlin et al., 2012; Kerzendorf et al., 2020; Stelmakh et al., 2025; OpenReview, 2025). In practice, many fields across science still assign peer reviewers manually (Merrifield and Saari, 2009; Jerabkova et al., 2023). Kovanis et al. (2016) show this manual process produces a distribution problem, not a supply problem, with a minority of researchers reviewing the majority of submissions. This highlights the need for scalable methods to identify domain relevant researchers beyond those already known to editors and program committees (Merrifield and Saari, 2009; Stelmakh et al., 2021; Dumlao and Teplitskiy, 2025).

Identifying and assigning peer reviewers is a multiobjective problem, involving domain similarity, conflicts of interest, career-stage, and other considerations (Langfeldt, 2001; Nightingale and Scott, 2007; Latypova, 2023). In practice, reviewer assignment systems at large conferences or scientific organizations typically involve two stages: estimating domain relevance between submissions and potential reviewers using vector representations (Stelmakh et al., 2025; Olivo et al., 2026) and assigning reviewers using optimization algorithms based on a set of constraints (Stelmakh et al., 2019). The relative importance of these objectives and constraints depends on the priorities of editors and program committees. We focus on the specific challenge of identifying researchers with domain/topical similarity to a submission (Mimno and McCallum, 2007; Jerabkova et al., 2023; Stelmakh et al., 2025). Currently deployed reviewer assignment systems match or rank reviewers within a predefined pool of candidates (Charlin et al., 2012; OpenReview, 2025, e.g., the Toronto Paper Matching System or OpenReview). Furthermore, some journals or conferences rely on proprietary reviewer recommendation systems ¹¹ 1 https://www.elsevier.com/editor/submission-systems/find-reviewers-tips, whose data or matching methodology is not openly published.

Therefore, we present an open source algorithm for identifying relevant reviewers directly from the scientific literature. We formulate reviewer discovery as an information retrieval problem: given a query submission, retrieve and rank researchers by domain similarity. We present the retrieve-then-disambiguate algorithm: (1) retrieve domain similar papers to the submission, (2) disambiguate author identities within the retrieved pool, and (3) rank the disambiguated identities by similarity to the submission. We leverage authorship, coauthorship, and citation relationships as observable signals of domain similarity. Although these relationships do not establish reviewer suitability, they allow us to evaluate whether a retrieval system successfully identifies researchers within the domain of the query publication. In summary, we introduce an open-source algorithm for identifying topically relevant researchers directly from the scientific literature, without relying on a predefined candidate pool. Second, we present an evaluation using astronomy and physics publications from NASA ADS, with authorship, coauthorship, and citation relationships as signals of domain relevance.

2 Data

We evaluate the retrieve-then-disambiguate algorithm using publication metadata from astronomy and physics literature linked to ORCID identifiers. We cross-match all refereed article titles in the NASA Astrophysics Data System (ADS)²² 2 https://ui.adsabs.harvard.edu/ against the 2025 ORCID public data dump³³ 3 https://info.orcid.org/documentation/integration-guide/working-with-bulk-data/. This yields 2 282 214 unique publications associated with 689 003 unique ORCID-linked researchers. We randomly sample 500 query publications whose first authors have at least 20 ORCID-linked publications in the corpus. This restriction reduces cases where retrieval failures may reflect insufficient publication data rather than limitations of the algorithm. For each query publication, we retrieve candidate domain reviewers from a larger corpus of 4 942 260 astronomy and physics publications published between 1995 and 2024.

We use the ORCID-linked data to construct three proxy labels for evaluating retrieval: first author (the query paper’s first author), coauthor (any other author of the query paper), and citation (any author of a paper cited by the query). These labels provide observable links between domain researchers and query publications but do not fully capture domain expertise (Stelmakh et al., 2025; Olivo et al., 2026). We therefore use these labels to evaluate reviewer discovery rather than reviewer expertise.

3 Methods

Algorithm 1 presents the retrieve-then-disambiguate algorithm and we describe each stage below.

Stage 1: Retrieval. Given a query paper’s title and abstract, we retrieve the N most similar candidate papers. To mimic the constraints of real peer review, in which only prior work is available, we restrict candidates to papers published before the query paper. We use TF–IDF and cosine similarity to estimate similarity between candidate papers and a query paper. A single TF–IDF vectorizer is fit once over all query paper titles and abstracts (Olivo et al., 2026), using the scikit-learn implementation (Pedregosa et al., 2018) (reused in Stage 3). We set the n-gram range to capture both unigrams and bigrams, allowing the model to incorporate two word phrases (e.g., “stellar evolution,” “galaxy formation,” or “white dwarf”) that represent more domain-specific meaning than separate individual words (e.g., “white,” “dwarf”). Standard English stop words (e.g., articles, prepositions) are removed to reduce noise. We compute cosine similarity between the TF–IDF vector representations of the query and each candidate paper, and keep the top N candidates by similarity as the candidate pool for Stage 2.

Stage 2: Name-blocked Disambiguation. Every one of the N retrieved candidate papers’ full author lists is separated into individual author records, one row per author with metadata (i.e., author name string, title, abstract, and publication year). Author records are split into disjoint name blocks by first name initial and last name (e.g., ‘J.Smith’ or ‘Y.Wang’), following standard blocking practice in author name disambiguation (Subramanian et al., 2021). Within each block, we disambiguate author records following the approach of Olivo et al. (2025). Each author record is embedded by concatenating a character-level author name embedding with a SPECTER2 title/abstract embedding (Cohan et al., 2020), then passing those concatenated embeddings through a trained contrastive encoder to produce a final author record embedding. The embedding representations of each author record within a block are then clustered with a constrained variant of DBSCAN (Ester et al., 1996; Khabsa et al., 2015). Each resulting cluster is treated as one disambiguated author identity. For evaluation, we assign ORCID labels to clusters by matching publication titles. An ORCID label is considered retrieved if it appears in a cluster with no conflicting ORCID labels.

Stage 3: Expertise ranking. We construct a disambiguated author representation by concatenating the title and abstract of every distinct paper attributed to a disambiguated cluster. Using the TF–IDF vectorizer, we transform every disambiguated author identity’s representation into this shared vector space, and rank disambiguated identities by cosine similarity to the query vector representation.

4 Results

We evaluate reviewer discovery by measuring the retrieval of ORCID-linked researchers with observable connections to a query publication. A researcher is counted as retrieved if their ORCID is among the disambiguated candidates retrieved. Because the number of candidate identities increases with N, our evaluation measures candidate discovery rather than the quality of a fixed-size reviewer shortlist. Table 1 reports the hit rate for each label type as we vary the top N publications retrieved. The citation hit rate increases with N from 0.447 when retrieving 500 publications to 0.789 when retrieving 10 000 publications, while first author and coauthor hit rates increase from 0.680 to 0.844 and from 0.597 to 0.711, respectively. Meanwhile, mean name blocks per query grow with N, from 1789 at N=500 to 21 922 at N=10000, directly increasing disambiguation cost. Figure 1 visualizes the resulting vector space for one query paper, using N=500.

We diagnose failures to recover ORCID-linked researchers across the 500 query publications. For each missed labeled researcher, we determine whether they have publications in the corpus preceding the query year. We classify failures as cold start (no prior publications), retrieval miss (prior publications exist but none appear among the top N retrieved papers), or disambiguation noise (at least one prior publication was retrieved, but identity was not resolved). Across all three retrieval settings, we identify 4337 labeled retrieval failures. Of these, 98.2% were retrieval misses and 1.8% were cold starts, with no failures attributed to disambiguation noise. We profile the algorithm on 10 query publications at different retrieval depth using approximately 15 CPU cores, one GPU with 2.1 GiB peak memory usage, and 54.1 GiB of system RAM. Mean end-to-end runtime, excluding one-time model initialization or embedding costs, increases from 15.95 seconds per query at N=500 to 212.44 seconds at N=10000. At N=500, embedding takes 5.45 seconds per query, followed by retrieval (3.52 seconds) and clustering (2.47 seconds). At N=10000, embedding takes 121.86 seconds per query and clustering takes 41.06 seconds. Runtime increases with the number of retrieved publications and name blocks requiring disambiguation.

5 Discussion and Future Work

Our results show that retrieving disambiguated authors directly from the literature surfaces domain relevant reviewers for peer review. Citation-based recall improves as the candidate pool N grows, indicating that a share of domain relevant reviewers sits outside what a small, fixed candidate pool can capture (Kovanis et al., 2016).

While we evaluate the algorithm on astronomy and physics literature, the architecture is not domain-specific: TF–IDF retrieval can be swapped for alternative representation methods, such as SPECTER2 (Cohan et al., 2020) embeddings or more recent large language model embeddings (e.g., Qwen embeddings) ⁴⁴ 4 https://huggingface.co/Qwen/Qwen3-Embedding-4B, and the disambiguation stage can be replaced with alternative methods (Subramanian et al., 2021). Applying the same pipeline to other domains and environments, such as computer science conference submissions, would only require refitting ϕ and δ to that literature, not redesigning the retrieval-then-disambiguate algorithm.

Our evaluation relies on authorship, coauthorship, and citation relationships as proxy signals for domain relevance. These relationships do not establish reviewer expertise, availability, or the absence of conflicts of interest. Furthermore, our reported hit rates measure the recovery of labeled researchers within the retrieved candidate pool, rather than the quality of a fixed-size ranked reviewer shortlist. Our evaluation is also conducted entirely on a preliminary sample of 500 astronomy and physics query papers. Therefore, our future work includes evaluating the pipeline across a larger set of the available ORCID-linked data and extending evaluation to new domains and publishing environments, particularly computer science and machine learning conference publishing (e.g., using OpenReview’s public submission and review data). Retrieving reviewers directly from the literature, rather than from a predefined pool, offers a path toward surfacing the qualified researchers who remain unknown under current reviewer assignment practices and systems.

References

• Charlin et al. (2012) L. Charlin, R. S. Zemel, and C. Boutilier A Framework for Optimizing Paper Matching. arXiv. Note: arXiv:1202.3706 [cs] External Links: Link, Document Cited by: §1, §1.
• Cohan et al. (2020) A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. Weld SPECTER: Document-level Representation Learning using Citation-informed Transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 2270–2282. External Links: Link, Document Cited by: §3, §5.
• Dumlao and Teplitskiy (2025) J. M. Z. Dumlao and M. Teplitskiy Geographical diversity of peer reviewers shapes author success. Proceedings of the National Academy of Sciences 122 (33), pp. e2507394122. External Links: Document, Link, https://www.pnas.org/doi/pdf/10.1073/pnas.2507394122 Cited by: §1.
• Ester et al. (1996) M. Ester, H. Kriegel, J. Sander, and X. Xu In A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, Note: [TLDR] DBSCAN, a new clustering algorithm relying on a density-based notion of clusters which is designed to discover clusters of arbitrary shape, is presented which requires only one input parameter and supports the user in determining an appropriate value for it. External Links: Link Cited by: §3.
• Hanson et al. (2023) M. A. Hanson, P. G. Barreiro, P. Crosetto, and D. Brockington The strain on scientific publishing. Quantitative Science Studies 5, pp. 823–843. External Links: Link Cited by: §1.
• Jerabkova et al. (2023) T. Jerabkova, F. Patat, F. Primas, D. Dorigo, F. Sogni, L. Astolfi, T. Bierwirth, and M. Prümm The First Results of Distributed Peer Review at ESO Show Promising Outcomes. Text. Note: ISSN: 0722-6691 Publisher: European Southern Observatory (ESO) External Links: Link Cited by: §1, §1.
• Kerzendorf et al. (2020) W. E. Kerzendorf, F. Patat, D. Bordelon, G. van de Ven, and T. A. Pritchard Distributed peer review enhanced with natural language processing and machine learning. Nature Astronomy 4 (7), pp. 711–717 (en). Note: Publisher: Nature Publishing Group External Links: ISSN 2397-3366, Link, Document Cited by: §1.
• Khabsa et al. (2015) M. Khabsa, P. Treeratpituk, and C. L. Giles Online Person Name Disambiguation with Constraints. Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 37–46 (en). Note: Conference Name: JCDL ’15: 15th ACM/IEEE-CS Joint Conference on Digital Libraries ISBN: 9781450335942 Place: Knoxville Tennessee USA Publisher: ACM[TLDR] An extension to the density-based clustering algorithm (DBSCAN) to handle online clustering so that the disambiguation process can be done iteratively as new data points are added, and implements two types of clustering constraints to demonstrate the concept. External Links: Link, Document Cited by: §3.
• Kovanis et al. (2016) M. Kovanis, R. Porcher, P. Ravaud, and L. Trinquart The global burden of journal peer review in the biomedical literature: strong imbalance in the collective enterprise. PLoS ONE 11. External Links: Link Cited by: §1, §5.
• Langfeldt (2001) L. Langfeldt The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social studies of science 31 (6), pp. 820–841. Cited by: §1.
• Latypova (2023) V. Latypova Decision-making support in optimal multicriteria peer reviewer selection in scientific conference organization. In 2023 5th International Conference on Control Systems, Mathematical Modeling, Automation and Energy Efficiency (SUMMA), pp. 383–386. Cited by: §1.
• Leyton-Brown et al. (2022) K. Leyton-Brown, Mausam, Y. Nandwani, H. Zarkoob, C. Cameron, N. Newman, and D. Raghu Matching Papers and Reviewers at Large Conferences. arXiv. Note: arXiv:2202.12273 [cs] External Links: Link, Document Cited by: §1.
• Merrifield and Saari (2009) M. R. Merrifield and D. G. Saari Telescope Time Without Tears: A Distributed Approach to Peer Review. Astronomy & Geophysics 50 (4), pp. 4.16–4.20. Note: arXiv:0906.1943 [astro-ph]Comment: 9 pages, accepted for publication in Astronomy & Geophysics External Links: ISSN 13668781, 14684004, Link, Document Cited by: §1.
• Mimno and McCallum (2007) D. Mimno and A. McCallum Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’07, New York, NY, USA, pp. 500–509. External Links: ISBN 978-1-59593-609-7, Link, Document Cited by: §1.
• Nightingale and Scott (2007) P. Nightingale and A. Scott Peer review and the relevance gap: ten suggestions for policy-makers. Science and Public Policy 34 (8), pp. 543–553. Cited by: §1.
• Olivo et al. (2026) V. A. Olivo, T. Jerabkova, J. Klencki, J. Carpenter, M. Malički, F. Patat, L. Strolger, and W. Kerzendorf Traditional statistical representations outperform generative ai in identifying expert peer reviewers. External Links: 2605.18752, Link Cited by: §1, §2, §3.
• Olivo et al. (2025) V. A. Olivo, W. Kerzendorf, B. Lu, J. V. Shields, A. Flörs, and N. Chen Practical author name disambiguation under metadata constraints: a contrastive learning approach for astronomy literature. External Links: 2511.10722, Link Cited by: §3.
• OpenReview (2025) OpenReview Openreview/openreview-expertise. OpenReview. Note: original-date: 2018-08-02T18:12:33Z External Links: Link Cited by: §1, §1.
• Pedregosa et al. (2018) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, A. Müller, J. Nothman, G. Louppe, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay Scikit-learn: Machine Learning in Python. arXiv. Note: arXiv:1201.0490 [cs]Comment: Update authors list and URLs External Links: Link, Document Cited by: §3.
• [20] (2025) Reflections on the 2025 review process from the program committee chairs. Note: NeurIPS Blog External Links: Link Cited by: §1.
• Stelmakh et al. (2021) I. Stelmakh, N. B. Shah, A. Singh, and H. Daumé III A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, pp. 4785–4793. Cited by: §1.
• Stelmakh et al. (2019) I. Stelmakh, N. B. Shah, and A. Singh PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review. arXiv. Note: arXiv:1806.06237 [stat] External Links: Link, Document Cited by: §1.
• Stelmakh et al. (2025) I. Stelmakh, J. Wieting, S. Xi, G. Neubig, and N. B. Shah A Gold Standard Dataset for the Reviewer Assignment Problem. arXiv. Note: arXiv:2303.16750 [cs] External Links: Link, Document Cited by: §1, §1, §2.
• Subramanian et al. (2021) S. Subramanian, D. King, D. Downey, and S. Feldman S2AND: A Benchmark and Evaluation System for Author Name Disambiguation. 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 170–179. Note: Conference Name: 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) ISBN: 9781665417709 Place: Champaign, IL, USA Publisher: IEEE[TLDR] This work presents S2AND, a unified benchmark dataset for AND on scholarly papers, as well as an open-source reference model implementation, and releases the unified dataset, model code, trained models, and evaluation suite to the research community. External Links: Link, Document Cited by: §3, §5.

注意转载授权 · 阅读原文