1 Origins

Let (,,μ) be a measure space, and (w₁,P₁),…,(w_n,P_n) be n weighted probability measures dominated by a measure μ (with w_i > 0 and ∑ w_i = 1). Denote by := {(w₁,p₁),…,(w_n,p_n)} the set of their weighted Radon-Nikodym densities p_i = with respect to μ.

A statistical divergence D[p : q] is a measure of dissimilarity between two densities p and q (i.e., a 2-point distance) such that D[p : q] ≥ 0 with equality if and only if p(x) = q(x) μ-almost everywhere. A statistical diversity index D() is a measure of variation of the weighted densities in related to a measure of centrality, i.e., a n-point distance which generalizes the notion of 2-point distance when ₂(p,q) := {(,p₁),(,p₂)}:

The fundamental measure of dissimilarity in information theory is the I-divergence (also called the Kullback-Leibler divergence, KLD, see Equation (2.5) page 5 of [5]):

The KLD is asymmetric (hence the delimiter notation “:” instead of ‘,’) but can be symmetrized by defining the Jeffreys J-divergence (Jeffreys divergence, denoted by I₂ in Equation (1) in 1946’s paper [4]):

Although symmetric, any positive power of Jeffreys divergence fails to satisfy the triangle inequality: That is, D_J^α is never a metric distance for any α > 0, and furthermore D_J^α cannot be upper bounded.

In 1991, Lin proposed the asymmetric K-divergence (Equation (3.2) in [7]):

and defined the L-divergence by analogy to Jeffreys’s symmetrization of the KLD (Equation (3.4) in [7]):

By noticing that

where h denotes Shannon entropy (Equation (3.14) in [7]), Lin coined the (skewed) Jensen-Shannon divergence between two weighted densities (1 - α,p) and (α,q) for α ∈ (0,1) as follows (Equation (4.1) in [7]):

(1)

Finally, Lin defined the generalized Jensen-Shannon divergence (Equation (5.1) in [7]) for a finite weighted set of densities:

This generalized Jensen-Shannon divergence is nowadays called the Jensen-Shannon diversity index.

To contrast with the Jeffreys’ divergence, the Jensen-Shannon divergence (JSD) D_JS := D_JS, is upper bounded by log 2 (does not require the densities to have the same support), and is a metric distance [2, 3]. Lin cited precursor work [17, 8] yielding definition of the Jensen-Shannon divergence: The Jensen-Shannon divergence of Eq. 1 is the so-called “increments of entropy” defined in (19) and (20) of [17].

The Jensen-Shannon diversity index was also obtained very differently by Sibson in 1969 when he defined the information radius [16] of order α using Rényi α-means and Rényi α-entropies [15]. In particular, the information radius IR₁ of order 1 of a weighted set of densities is a diversity index obtained by solving the following variational optimization problem:

(2)

Sibson solved a more general optimization problem, and obtained the following expression (term K₁ in Corollary 2.3 [16]):

Thus Eq. 2 is a variational definition of the Jensen-Shannon divergence.

2 Some extensions

• Skewing the JSD.

  The K-divergence of Lin can be skewed with a scalar parameter α ∈ (0,1) to give

  (3)

  Skewing parameter α was first studied in [6] (2001, see Table 2 of [6]). We proposed to unify the Jeffreys divergence with the Jensen-Shannon divergence as follows (Equation 19 in [9]):

  (4)

  When α = , we have D_K,^J = D_JS, and when α = 1, we get D_K,1^J = D_J.

  Notice that

  

  amounts to calculate

  

  where

  

  denotes the cross-entropy. By choosing α = β, we have h^×[(1-β)p+βq : (1-α)p+αq] = h[(1-α)p+αq], and thus recover the skewed Jensen-Shannon divergence of Eq. 1.

  In [11] (2020), we considered a positive skewing vector α ∈ [0,1]^k and a unit positive weight w belonging to the standard simplex Δ_k, and defined the following vector-skewed Jensen-Shannon divergence:

  

  where α = ∑ _i=1^kw_iα_i. The divergence D_JS^α,w generalizes the (scalar) skew Jensen-Shannon divergence when k = 1, and is a Ali-Silvey-Csiszár f-divergence upper bounded by log  [11].
• A priori mid-density. The JSD can be interpreted as the total divergence of the densities to the mid-density p = ∑ _i=1ⁿw_ip_i, a statistical mixture:
  

  Unfortunately, the JSD between two Gaussian densities is not known in closed form because of the definite integral of a log-sum term (i.e., K-divergence between a density and a mixture density p). For the special case of the Cauchy family, a closed-form formula [14] for the JSD between two Cauchy densities was obtained. Thus we may choose a geometric mixture distribution [10] instead of the ordinary arithmetic mixture p. More generally, we can choose any weighted mean M_α (say, the geometric mean, or the harmonic mean, or any other power mean) and define a generalization of the K-divergence of Equation 3:

  (7)

  where

  

  is a statistical M-mixture with Z_Mα(p,q) denoting the normalizing coefficient:

  

  so that ∫ (pq)_Mα(x)dμ(x) = 1. These M-mixtures are well-defined provided the convergence of the definite integrals.

  Then we define a generalization of the JSD [10] termed (M_α,N_β)-Jensen-Shannon divergence as follows:

  (8)

  where N_β is yet another weighted mean to average the two M_α-K-divergences. We have D_JS = D_JS^A,A where A(a,b) = is the arithmetic mean. The geometric JSD yields a closed-form formula between two multivariate Gaussians, and has been used in deep learning [1]. More generally, we may consider the Jensen-Shannon symmetrization of an arbitrary distance D as

  (9)

• A posteriori mid-density. We consider a generalization of Sibson’s information radius [16]. Let S_w(a₁,…,a_n) denote a generic weighted mean of n positive scalars a₁,…,a_n, with weight vector w ∈ Δ_n. Then we define the S-variational Jensen-Shannon diversity index [12] as

  (10)

  When S_w = A_w (with A_w(a₁,…,a_n) = ∑ _i=1ⁿw_ia_i the arithmetic weighted mean), we recover the ordinary Jensen-Shannon diversity index. More generally, we define the S-Jensen-Shannon index of an arbitrary distance D as

  (11)

  When n = 2, this yields a Jensen-Shannon-symmetrization of distance D.

  The variational optimization defining the JSD can also be constrained to a (parametric) family of densities , thus defining the (S,)-relative Jensen-Shannon diversity index:

  (12)

  The relative Jensen-Shannon divergences are useful for clustering applications: Let p_θ1 and p_θ2 be two densities of an exponential family with cumulant function F(θ). Then the -relative Jensen-Shannon divergence is the Bregman information of ₂(p,q) for the conjugate function F^*(η) = -h[p_θ] (with η = ∇F(θ)). The -relative JSD amounts to a Jensen divergence for F^*:

  

  since η^* := (a right-sided Bregman centroid [13]).

References