Re-ranking via local embeddings: a use case with permutation-based indexing and the nSimplex projection

Lucia Vadicamo*, Claudio Gennaro, Fabrizio Falchi, Edgar Chávez, Richard Connor, Giuseppe Amato

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)

Abstract

Approximate Nearest Neighbor (ANN) search is a prevalent paradigm for searching intrinsically high dimensional objects in large-scale data sets. Recently, the permutation-based approach for ANN has attracted a lot of interest due to its versatility in being used in the more general class of metric spaces. In this approach, the entire database is ranked by a permutation distance to the query. Typically, permutations allow the efficient selection of a candidate set of results, but typically to achieve high recall or precision this set has to be reviewed using the original metric and data. This can lead to a sizeable percentage of the database being recalled, along with many expensive distance calculations.

To reduce the number of metric computations and the number of database elements accessed, we propose here a re-ranking based on a local embedding using the nSimplex projection. The nSimplex projection produces Euclidean vectors from objects in metric spaces which possess the n-point property. The mapping is obtained from the distances to a set of reference objects, and the original metric can be lower bounded and upper bounded by the Euclidean distance of objects sharing the same set of references.

Our approach is particularly advantageous for extensive databases or expensive metric function. We reuse the distances computed in the permutations in the first stage, and hence the memory footprint of the index is not increased.

An extensive experimental evaluation of our approach is presented, demonstrating excellent results even on a set of hundreds of millions of objects.

Original languageEnglish
Article number101506
Number of pages18
JournalInformation Systems
Volume95
Early online date13 Feb 2020
DOIs
Publication statusPublished - Jan 2021

Keywords

  • Distance bounds
  • Metric local embeddings
  • Metric search
  • n-point property
  • nSimplex projection
  • Permutation-based indexing

Fingerprint

Dive into the research topics of 'Re-ranking via local embeddings: a use case with permutation-based indexing and the nSimplex projection'. Together they form a unique fingerprint.

Cite this