Correlations of cross-entropy loss in machine learning

Richard Connor*, Al Dearle, Ben Claydon, Lucia Vadicamo

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Cross-entropy loss is crucial in training many deep neural networks. In this context, we show a number of novel and strong correlations among various related divergence functions. In particular, we demonstrate that, in some circumstances, (a) cross-entropy is almost perfectly correlated with the little-known triangular divergence, and (b) cross-entropy is strongly correlated with the Euclidean distance over the logits from which the softmax is derived. The consequences of these observations are as follows. First, triangular divergence may be used as a cheaper alternative to cross-entropy. Second, logits can be used as features in a Euclidean space which is strongly synergistic with the classification process. This justifies the use of Euclidean distance over logits as a measure of similarity, in cases where the network is trained using softmax and cross-entropy. We establish these correlations via empirical observation, supported by a mathematical explanation encompassing a number of strongly related divergence functions.
Original languageEnglish
Article number491
Number of pages16
JournalEntropy
Volume26
Issue number6
DOIs
Publication statusPublished - 30 May 2024

Keywords

  • Softmax
  • Cross-entropy
  • f-divergence
  • Kullback-Liebler divergence
  • Jensen-Shannon 12 divergence
  • Triangular divergence

Fingerprint

Dive into the research topics of 'Correlations of cross-entropy loss in machine learning'. Together they form a unique fingerprint.

Cite this