Non-deterministic Behavior of Ranking-Based Metrics When Evaluating Embeddings

Nicolaou A, Dey S, Christlein V, Maier A, Karatzas D (2019)

Publication Language: English

Publication Type: Conference contribution

Publication year: 2019

Journal

Lecture Notes in Computer Science Springer Verlag

Publisher: Springer Verlag

Book Volume: 11455 LNCS

Pages Range: 71-82

Conference Proceedings Title: RRPR 2018: Reproducible Research in Pattern Recognition

Event location: Beijing CN

ISBN: 9783030239862

URI: https://arxiv.org/pdf/1806.07171.pdf

DOI: 10.1007/978-3-030-23987-9_5

Abstract

Embedding data into vector spaces is a very popular strategy of pattern recognition methods. When distances between embeddings are quantized, performance metrics become ambiguous. In this paper, we present an analysis of the ambiguity quantized distances introduce and provide bounds on the effect. We demonstrate that it can have a measurable effect in empirical data in state-of-the-art systems. We also approach the phenomenon from a computer security perspective and demonstrate how someone being evaluated by a third party can exploit this ambiguity and greatly outperform a random predictor without even access to the input data. We also suggest a simple solution making the performance metrics, which rely on ranking, totally deterministic and impervious to such exploits.

Authors with CRIS profile

Vincent Christlein Lehrstuhl für Informatik 5 (Mustererkennung) Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung)

Involved external institutions

Autonomous University of Barcelona (UAB) / Universitat Autònoma de Barcelona ES Spain (ES)

How to cite

APA:

Nicolaou, A., Dey, S., Christlein, V., Maier, A., & Karatzas, D. (2019). Non-deterministic Behavior of Ranking-Based Metrics When Evaluating Embeddings. In Hugues Talbot, Pascal Monasse, Bertrand Kerautret, Miguel Colom, Daniel Lopresti (Eds.), RRPR 2018: Reproducible Research in Pattern Recognition (pp. 71-82). Beijing, CN: Springer Verlag.

MLA:

Nicolaou, Anguelos, et al. "Non-deterministic Behavior of Ranking-Based Metrics When Evaluating Embeddings." Proceedings of the 2nd International Workshop on Reproducible Research in Pattern Recognition, RRPR 2018, Beijing Ed. Hugues Talbot, Pascal Monasse, Bertrand Kerautret, Miguel Colom, Daniel Lopresti, Springer Verlag, 2019. 71-82.

BibTeX: Download