Bayesian bin distribution inference and mutual information

Dominik Maria Endres, Peter Foldiak

Research output: Contribution to journalArticlepeer-review

31 Citations (Scopus)
1 Downloads (Pure)


We present an exact Bayesian treatment of a simple, yet sufficiently general probability distribution model. We consider piecewise-constant distributions' P(X) with uniform (second-order) prior over location of discontinuity points and assigned chances. The predictive distribution and the model complexity can be determined completely from the data in a computational time that is linear in the number of degrees of freedom and quadratic in the number of possible values of X. Furthermore, exact values of the expectations of entropies and their variances can be computed with polynomial effort. The expectation of the mutual information becomes thus available, too, and a strict upper bound on its variance. The resulting algorithm is particularly useful in experimental research areas where the number of available samples is severely limited (e.g., neurophysiology). Estimates on a simulated data set provide more accurate results than using a previously proposed method.

Original languageEnglish
Pages (from-to)3766-3779
Number of pages14
JournalIEEE Transactions on Information Theory
Issue number11
Publication statusPublished - Nov 2005


  • Bayesian inference
  • Entropy
  • Model selection
  • Mutual information


Dive into the research topics of 'Bayesian bin distribution inference and mutual information'. Together they form a unique fingerprint.

Cite this