Information geometry

• General overview of information geometry:
  • An elementary introduction to information geometry

    → A primer on the fundamental theorem of information geometry

  • Introduction to Bregman manifolds (aka dually flat spaces, Hessian structures)

    → Dually flat spaces of exponential families and mixture families, smooth strictly convex functions, and the dual generalized Pythagorean theorems

  • Cramer-Rao lower bound and information geometry

    → A first read introducing the Fisher information matrix, the CRLB, and the Fisher-Rao differential-geometrization of statistics (aka 0-geometry)

• Some contributions to information geometry:
  • Bregman Voronoi diagrams, Discrete and Computational Geometry, 2010. → Bregman Voronoi diagrams can be built by clipping power diagrams, or by calculating the intersection of halfspaces, space of Bregman spheres, etc.
  • The Burbea-Rao and Bhattacharyya centroids, IEEE Transactions on Information Theory, 2011. → Statistical Bhattacharyya distances between exponential families yield Burbea-Rao parameter divergences. Centroid calculated using the Convex-ConCave Procedure (CCCP)
  • An information-geometric characterization of Chernoff information, IEEE Signal Processing Letters, 2013. → On the exponential family manifold, we characterize the best error exponent by calculating the intersection of a geodesic with a dual bisector. The Chernoff information (alpha-divergence for the optimal error exponent) amounts to a Bregman divergence.
  • On the Geometry of Mixtures of Prescribed Distributions, IEEE ICASSP 2018 → The Kullback-Leibler divergence between two mixture densities with prescribed components (w-mixtures) amount to a Bregman divergence for the negentropy Bregman generator. Report applications of this fact.
• Some applications of information geometry:
  • arxiv GLSR-VAE: Geodesic latent space regularization for variational autoencoder architectures, IEEE Symposium Series on Computational Intelligence 2017
  • JMLR proc Relative Fisher Information and Natural Gradient for Learning Large Modular Models, ICML 2017.
    arxiv Relative Natural Gradient for Learning Large Complex Models, 2016
    Web page
  • Lightlike Neuromanifolds, Occam's Razor and Deep Learning
• Some applications of dissimilarities:
  • On the chi square and higher-order chi distances for approximating f-divergences, IEEE Signal Processing Letters, 2013.
  • Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means, Pattern Recognition Letters, 2014.
  • Generalizing Skew Jensen Divergences and Bregman Divergences With Comparative Convexity, IEEE SPL 2017.
  • On a Generalization of the Jensen-Shannon Divergence and the Jensen-Shannon Centroid. Entropy 22(2): 221 (2020)

    → Skew the Jensen-Shannon divergence with a skewing vector (instead of an ordinary scalar α) Calculate the Jensen-Shannon centroid of a set of normalized histograms using the Convex-ConCave Procedure (CCCP)

• Books:
  • Geometric Structures of Information, Springer 2019
  • Computational Information Geometry For Image and Signal Processing, Springer 2017
  • Differential Geometrical Theory of Statistics, MDPI Entropy, Special issue, 2017
  • Geometric Theory of Information, Springer 2014
  • Matrix Information Geometry, Springer 2013
  • ETVC 2008 GSI'13 GSI'15 GSI'17 GSI'19
• Applications of Natural Gradient (coming soon!:
  • Natural Evolution Strategies (NES) [Wikipedia]