Information geometry [2] defines, studies, and applies core dualistic structures on smooth manifolds: Namely, pairs of dual affine connections (∇,∇^*) coupled with Riemannian metrics g. In particular, those (g,∇,∇^*) structures can be built from statistical models [2] or induced by divergences [3] (contrast functions on product manifolds) or convex functions [19] on open convex domains (e.g., logarithmic characteristic functions of symmetric cones [21, 18]). In the latter case, manifolds are said dually flat [2] or Hessian [19] since the Riemannian metrics can be expressed locally either as g(θ) = ∇²F(θ) in the ∇-affine coordinate system θ or equivalently as g(η) = ∇²F^*(η) in the ∇^*-affine coordinate system η. The Legendre-Fenchel duality F^*(η) = sup_θ∈Θ⟨θ,η⟩- F(θ) allows to convert between primal to dual coordinates: η(θ) = ∇F(θ) and θ(η) = ∇F^*(η). Dually flat spaces have been further generalized to handle singularities in [10].

To get a taste of computational information geometry (CIG), let us mention the following two problems when implementing information-geometric structures and algorithms:

• In practice, we can fully implement geometric algorithms on dually flat spaces when both the primal potential function F(θ) and the dual potential function F^*(η) are known in closed-form and computationally tractable [14]. See also the Python library pyBregMan [16]. To overcome computationally intractable potential functions, we may either consider Monte Carlo information geometry [14] or discretizing continuous distributions into a finite number of bins [6, 13] (amounts to consider standard simplex models).

• The Chernoff information [5] between two absolutely continuous distributions P and Q with densities p(x) and q(x) with respect to some dominating measure μ is defined by

  

  where α^* is called the optimal exponent. Chernoff information is used in statistics and for information fusion tasks [7] among others. In general, the Chernoff information between two continuous distributions is not available in closed form (e.g., not known in closed-form between multivariate Gaussian distributions [12]). However, for densities p and q of an exponential family, the optimal exponent α^* can be characterized exactly geometrically as the unique intersection of the e-geodesic γ_pq with a dual m-bisector [11]. This geometric characterization yields an efficient approximation algorithm.

Thus computational information geometry aims at implementing robustly the information-geometric structures and the geometric algorithms on those structures for various applications. To give two examples of CIG, consider

• computing the minimum enclosing ball (MEB) of a finite set of m-dimensional points on a dually flat space: The MEB is always unique and can be calculated (in theory) using a LP-type randomized linear-time solver [15] (linear programming-type) relying on oracles which exactly compute the enclosing balls passing through exactly k points for k ∈ {2,…,m}. However, these oracles are in general computationally intractable so that guaranteed approximation algorithms have been considered [17].

• Learning a deep neural networks using natural gradient [1, 4]: In practice, the number of parameters of a DNN is very large so that it is impractical to learn the weights of a DNN with natural gradient descent which require to handle large (potentially inverse) Fisher information matrices. Many practical approaches closely related to natural gradient have been thus considered in machine learning [9, 20, 8].

References