TY - GEN
T1 - A dataset for expert reviewer recommendation with large language models as zero-shot rankers
AU - Karan, Vanja M.
AU - McQuistin, Stephen
AU - Yanagida, Ryo
AU - Perkins, Colin
AU - Tyson, Gareth
AU - Castro, Ignacio
AU - Healey, Patrick
AU - Purver, Matthew
N1 - Funding: This work was partially supported by the UK EP-SRC via the projects Sodestream (EP/S033564/1, EP/S036075/1), AP4L (EP/W032473/1), ARCIDUCA (EP/W001632/1) and AdSoLve (Responsible AI UK, EP/Y009800/1, project KP0016); and by the Slovenian Research Agency via research core funding for the programme Knowledge Technologies (P2-0103) and the project EMMA (L2-50070).
PY - 2025/1/1
Y1 - 2025/1/1
N2 - The task of reviewer recommendation is increasingly important, with main techniques utilizing general models of text relevance. However, state of the art (SotA) systems still have relatively high error rates. Two possible reasons for this are: a lack of large datasets and the fact that large language models (LLMs) have not yet been applied. To fill these gaps, we first create a substantial new dataset, in the domain of Internet specification documents; then we introduce the use of LLMs and evaluate their performance. We find that LLMs with prompting can improve on SotA in some cases, but that they are not a cure-all: this task provides a challenging setting for prompt-based methods.
AB - The task of reviewer recommendation is increasingly important, with main techniques utilizing general models of text relevance. However, state of the art (SotA) systems still have relatively high error rates. Two possible reasons for this are: a lack of large datasets and the fact that large language models (LLMs) have not yet been applied. To fill these gaps, we first create a substantial new dataset, in the domain of Internet specification documents; then we introduce the use of LLMs and evaluate their performance. We find that LLMs with prompting can improve on SotA in some cases, but that they are not a cure-all: this task provides a challenging setting for prompt-based methods.
M3 - Conference contribution
AN - SCOPUS:85218490502
T3 - Proceedings - international conference on computational linguistics, COLING
SP - 11422
EP - 11427
BT - Proceedings of the 31st international conference on computational linguistics
A2 - Rambow, Owen
A2 - Wanner, Leo
A2 - Apidianaki, Marianna
A2 - Al-Khalifa, Hend
A2 - Di Eugenio, Barbara
A2 - Schockaert, Steven
PB - Association for Computational Linguistics (ACL)
CY - Stroudsburg, PA
T2 - 31st International Conference on Computational Linguistics, COLING 2025
Y2 - 19 January 2025 through 24 January 2025
ER -