Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy

Zihao  lI; Rosa Filgueira

doi:10.1109/e-Science58273.2023.10254873

Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy

Zihao lI, Rosa Filgueira^*

^*Corresponding author for this work

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The rapid growth of scientific software development has led to the emergence of large and complex codebases, making it challenging to search, find, and compare software repositories within the scientific research community. In this paper, we propose a solution by leveraging deep learning techniques to learn embeddings that capture semantic similarities among repositories. Our approach focuses on identifying repositories with similar semantics, even when their code fragments and documentation exhibit different syntax. To address this challenge, we introduce two complementary open-source tools: RepoSim and RepoSnipy. RepoSim is a command-line toolbox designed to represent repositories at both the source code and documentation levels. It utilizes the UniXcoder pre-trained language model, which has demonstrated remarkable performance in code-related understanding tasks. RepoSnipy is a web-based neural semantic search engine that utilizes the powerful capabilities of RepoSim and offers a user-friendly search interface, allowing researchers and practitioners to query public repositories hosted on GitHub and discover semantically similar repositories. RepoSim and RepoSnipy empower researchers, developers, and practitioners by facilitating the comparison and analysis of software repositories. They not only enable efficient collaboration and code reuse but also accelerate the development of scientific software.

Original language	English
Title of host publication	Proceedings
Subtitle of host publication	2023 IEEE 19th international conference on e-science (e-science)
Editors	George Angelos Papadopoulos, Rosa Filgueira, Rafael Ferreira Da Silva
Place of Publication	Piscataway, NJ
Publisher	IEEE
Number of pages	10
ISBN (Electronic)	9798350322231
ISBN (Print)	9798350322248
DOIs	https://doi.org/10.1109/e-Science58273.2023.10254873
Publication status	Published - 25 Sept 2023
Event	19th IEEE International Conference on eScience - Limassol, Cyprus, Limassol, Cyprus Duration: 9 Oct 2023 → 13 Oct 2023 Conference number: 19 https://www.escience-conference.org/2023/

Publication series

Name	IEEE international conference on e-science
ISSN (Print)	2325-372X
ISSN (Electronic)	2325-3703

Conference

Conference	19th IEEE International Conference on eScience
Abbreviated title	eScience
Country/Territory	Cyprus
City	Limassol
Period	9/10/23 → 13/10/23
Internet address	https://www.escience-conference.org/2023/

Keywords

Semantic similarity
Code search
Code understanding
Embeddings, pre-trained language models
GitHub

Access to Document

10.1109/e-Science58273.2023.10254873

RepoSim-submitted
Copyright © 2023 IEEE. This work has been made available online in accordance with publisher policies or with permission. Permission for further reuse of this content should be sought from the publisher or the rights holder. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1109/e-Science58273.2023.10254873.
Accepted author manuscript, 2.58 MB

Cite this

lI, Z., & Filgueira, R. (2023). Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy. In G. A. Papadopoulos, R. Filgueira, & R. F. Da Silva (Eds.), Proceedings: 2023 IEEE 19th international conference on e-science (e-science) Article 10254873 (IEEE international conference on e-science). IEEE. https://doi.org/10.1109/e-Science58273.2023.10254873

@inproceedings{ebf12b01018341d08d5dd3432d5739e5,

title = "Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy",

abstract = "The rapid growth of scientific software development has led to the emergence of large and complex codebases, making it challenging to search, find, and compare software repositories within the scientific research community. In this paper, we propose a solution by leveraging deep learning techniques to learn embeddings that capture semantic similarities among repositories. Our approach focuses on identifying repositories with similar semantics, even when their code fragments and documentation exhibit different syntax. To address this challenge, we introduce two complementary open-source tools: RepoSim and RepoSnipy. RepoSim is a command-line toolbox designed to represent repositories at both the source code and documentation levels. It utilizes the UniXcoder pre-trained language model, which has demonstrated remarkable performance in code-related understanding tasks. RepoSnipy is a web-based neural semantic search engine that utilizes the powerful capabilities of RepoSim and offers a user-friendly search interface, allowing researchers and practitioners to query public repositories hosted on GitHub and discover semantically similar repositories. RepoSim and RepoSnipy empower researchers, developers, and practitioners by facilitating the comparison and analysis of software repositories. They not only enable efficient collaboration and code reuse but also accelerate the development of scientific software.",

keywords = "Semantic similarity, Code search, Code understanding, Embeddings, pre-trained language models, GitHub",

author = "Zihao lI and Rosa Filgueira",

year = "2023",

month = sep,

day = "25",

doi = "10.1109/e-Science58273.2023.10254873",

language = "English",

isbn = "9798350322248",

series = "IEEE international conference on e-science",

publisher = "IEEE",

editor = "Papadopoulos, {George Angelos} and Rosa Filgueira and {Da Silva}, {Rafael Ferreira}",

booktitle = "Proceedings",

note = "19th IEEE International Conference on eScience, eScience ; Conference date: 09-10-2023 Through 13-10-2023",

url = "https://www.escience-conference.org/2023/",

}

lI, Z & Filgueira, R 2023, Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy. in GA Papadopoulos, R Filgueira & RF Da Silva (eds), Proceedings: 2023 IEEE 19th international conference on e-science (e-science)., 10254873, IEEE international conference on e-science, IEEE, Piscataway, NJ, 19th IEEE International Conference on eScience, Limassol, Cyprus, 9/10/23. https://doi.org/10.1109/e-Science58273.2023.10254873

Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy. / lI, Zihao ; Filgueira, Rosa.
Proceedings: 2023 IEEE 19th international conference on e-science (e-science). ed. / George Angelos Papadopoulos; Rosa Filgueira; Rafael Ferreira Da Silva. Piscataway, NJ: IEEE, 2023. 10254873 (IEEE international conference on e-science).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Mapping the repository landscape

T2 - 19th IEEE International Conference on eScience

AU - lI, Zihao

AU - Filgueira, Rosa

N1 - Conference code: 19

PY - 2023/9/25

Y1 - 2023/9/25

N2 - The rapid growth of scientific software development has led to the emergence of large and complex codebases, making it challenging to search, find, and compare software repositories within the scientific research community. In this paper, we propose a solution by leveraging deep learning techniques to learn embeddings that capture semantic similarities among repositories. Our approach focuses on identifying repositories with similar semantics, even when their code fragments and documentation exhibit different syntax. To address this challenge, we introduce two complementary open-source tools: RepoSim and RepoSnipy. RepoSim is a command-line toolbox designed to represent repositories at both the source code and documentation levels. It utilizes the UniXcoder pre-trained language model, which has demonstrated remarkable performance in code-related understanding tasks. RepoSnipy is a web-based neural semantic search engine that utilizes the powerful capabilities of RepoSim and offers a user-friendly search interface, allowing researchers and practitioners to query public repositories hosted on GitHub and discover semantically similar repositories. RepoSim and RepoSnipy empower researchers, developers, and practitioners by facilitating the comparison and analysis of software repositories. They not only enable efficient collaboration and code reuse but also accelerate the development of scientific software.

AB - The rapid growth of scientific software development has led to the emergence of large and complex codebases, making it challenging to search, find, and compare software repositories within the scientific research community. In this paper, we propose a solution by leveraging deep learning techniques to learn embeddings that capture semantic similarities among repositories. Our approach focuses on identifying repositories with similar semantics, even when their code fragments and documentation exhibit different syntax. To address this challenge, we introduce two complementary open-source tools: RepoSim and RepoSnipy. RepoSim is a command-line toolbox designed to represent repositories at both the source code and documentation levels. It utilizes the UniXcoder pre-trained language model, which has demonstrated remarkable performance in code-related understanding tasks. RepoSnipy is a web-based neural semantic search engine that utilizes the powerful capabilities of RepoSim and offers a user-friendly search interface, allowing researchers and practitioners to query public repositories hosted on GitHub and discover semantically similar repositories. RepoSim and RepoSnipy empower researchers, developers, and practitioners by facilitating the comparison and analysis of software repositories. They not only enable efficient collaboration and code reuse but also accelerate the development of scientific software.

KW - Semantic similarity

KW - Code search

KW - Code understanding

KW - Embeddings, pre-trained language models

KW - GitHub

UR - https://doi.org/10.1109/e-Science58273.2023

U2 - 10.1109/e-Science58273.2023.10254873

DO - 10.1109/e-Science58273.2023.10254873

M3 - Conference contribution

SN - 9798350322248

T3 - IEEE international conference on e-science

BT - Proceedings

A2 - Papadopoulos, George Angelos

A2 - Filgueira, Rosa

A2 - Da Silva, Rafael Ferreira

PB - IEEE

CY - Piscataway, NJ

Y2 - 9 October 2023 through 13 October 2023

ER -

Mapping the repository landscape: harnessing similarity with RepoSim and RepoSnipy

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this