Divergence (statistics)

Short description: Function that measures dissimilarity between two probability distributions

In information geometry, a divergence is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold.

The simplest divergence is squared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence is relative entropy (also called Kullback–Leibler divergence), which is central to information theory. There are numerous other specific divergences and classes of divergences, notably f-divergences and Bregman divergences (see § Examples).

## Contents

* 1 Definition
* 1.1 Difference from other similar concepts
* 2 Notation
* 3 Geometrical properties
* 4 Examples
* 4.1 f-divergences
* 4.2 Bregman divergences
* 5 History
* 6 See also
* 7 Notes
* 8 References
* 8.1 Bibliography

## Definition

Given a differentiable manifold[lower-alpha 1] M of dimension n, a divergence on M is a C2-function D:M×M->[0,∞) satisfying:[1][2]

1. D(p,q)≥0 for all p,q∈M (non-negativity),
2. D(p,q)=0 if and only if p=q (positivity),
3. At every point p∈M, D(p,p+dp) is a positive-definite quadratic form for infinitesimal displacements dp from p.

In applications to statistics, the manifold M is typically the space of parameters of a parametric family of probability distributions.

Condition 3 means that D defines an inner product on the tangent space TpM for every p∈M. Since D is C2 on M, this defines a Riemannian metric g on M.

Locally at p∈M, we may construct a local coordinate chart with coordinates x, then the divergence is D(x(p),x(p)+dx)=12dxTgp(x)dx+O(|dx|3)where gp(x) is a matrix of size n×n. It is the Riemannian metric at point p expressed in coordinates x.

Dimensional analysis of condition 3 shows that divergence has the dimension of squared distance.[3]

The dual divergence D* is defined as

D*(p,q)=D(q,p).

When we wish to contrast D against D*, we refer to D as primal divergence.

Given any divergence D, its symmetrized version is obtained by averaging it with its dual divergence:[3]

DS(p,q)=12(D(p,q)+D(q,p)).

### Difference from other similar concepts

Unlike metrics, divergences are not required to be symmetric, and the asymmetry is important in applications.[3] Accordingly, one often refers asymmetrically to the divergence "of q from p" or "from p to q", rather than "between p and q". Secondly, divergences generalize squared distance, not linear distance, and thus do not satisfy the triangle inequality, but some divergences (such as the Bregman divergence) do satisfy generalizations of the Pythagorean theorem.

In general statistics and probability, "divergence" generally refers to any kind of function D(p,q), where p,q are probability distributions or other objects under consideration, such that conditions 1, 2 are satisfied. Condition 3 is required for "divergence" as used in information geometry.

As an example, the total variation distance, a commonly used statistical divergence, does not satisfy condition 3.

## Notation

Notation for divergences varies significantly between fields, though there are some conventions.

Divergences are generally notated with an uppercase 'D', as in D(x,y), to distinguish them from metric distances, which are notated with a lowercase 'd'. When multiple divergences are in use, they are commonly distinguished with subscripts, as in DKL for Kullback–Leibler divergence (KL divergence).

Often a different separator between parameters is used, particularly to emphasize the asymmetry. In information theory, a double bar is commonly used: D(p∥q); this is similar to, but distinct from, the notation for conditional probability, P(A|B), and emphasizes interpreting the divergence as a relative measurement, as in relative entropy; this notation is common for the KL divergence. A colon may be used instead,[lower-alpha 2] as D(p:q); this emphasizes the relative information supporting the two distributions.

The notation for parameters varies as well. Uppercase P,Q interprets the parameters as probability distributions, while lowercase p,q or x,y interprets them geometrically as points in a space, and μ1,μ2 or m1,m2 interprets them as measures.

## Geometrical properties

Many properties of divergences can be derived if we restrict S to be a statistical manifold, meaning that it can be parametrized with a finite-dimensional coordinate system θ, so that for a distribution p ∈ S we can write p = p(θ).

For a pair of points p, q ∈ S with coordinates θp and θq, denote the partial derivatives of D(p, q) as

D((∂i)p,q) =def ∂∂θpiD(p,q),D((∂i∂j)p,(∂k)q) =def ∂∂θpi∂∂θpj∂∂θqkD(p,q), etc.

Now we restrict these functions to a diagonal p = q, and denote [4]

D[∂i,⋅] : p↦D((∂i)p,p),D[∂i,∂j] : p↦D((∂i)p,(∂j)p), etc.

By definition, the function D(p, q) is minimized at p = q, and therefore

D[∂i,⋅]=D[⋅,∂i]=0,D[∂i∂j,⋅]=D[⋅,∂i∂j]=−D[∂i,∂j] ≡ gij(D),

where matrix g(D) is positive semi-definite and defines a unique Riemannian metric on the manifold S.

Divergence D(·, ·) also defines a unique torsion-free affine connection ∇(D) with coefficients

Γij,k(D)=−D[∂i∂j,∂k],

and the dual to this connection ∇* is generated by the dual divergence D*.

Thus, a divergence D(·, ·) generates on a statistical manifold a unique dualistic structure (g(D), ∇(D), ∇(D*)). The converse is also true: every torsion-free dualistic structure on a statistical manifold is induced from some globally defined divergence function (which however need not be unique).[5]

For example, when D is an f-divergence[6] for some function ƒ(·), then it generates the metric g(Df) = c·g and the connection ∇(Df) = ∇(α), where g is the canonical Fisher information metric, ∇(α) is the α-connection, c = ƒ′′(1), and α = 3 + 2ƒ′′′(1)/ƒ′′(1).

## Examples

The two most important divergences are the relative entropy (Kullback–Leibler divergence, KL divergence), which is central to information theory and statistics, and the squared Euclidean distance (SED). Minimizing these two divergences is the main way that linear inverse problems are solved, via the principle of maximum entropy and least squares, notably in logistic regression and linear regression.[7]

The two most important classes of divergences are the f-divergences and Bregman divergences; however, other types of divergence functions are also encountered in the literature. The only divergence for probabilities over a finite alphabet that is both an f-divergence and a Bregman divergence is the Kullback–Leibler divergence.[8] The squared Euclidean divergence is a Bregman divergence (corresponding to the function x2) but not an f-divergence.

### f-divergences

Given a convex function f:[0,+∞)->(−∞,+∞] such that f(0)=limt->0+f(t),f(1)=0, the f-divergence generated by f is defined as

Df(p,q)=∫p(x)f(q(x)p(x))dx.
Kullback–Leibler divergence:  | DKL(p,q)=∫p(x)ln⁡(p(x)q(x))dx

|

squared Hellinger distance:  | H2(p,q)=2∫(p(x)−q(x))2dx
Jensen–Shannon divergence:  | DJS(p,q)=12∫(p(x)−q(x))(ln⁡p(x)−ln⁡q(x))dx
α-divergence  | D(α)(p,q)=41−α2(1−∫p(x)1−α2q(x)1+α2dx)
chi-squared divergence:  | Dχ2(p,q)=∫(p(x)−q(x))2p(x)dx
(α,β)-product divergence[citation needed]:  | Dα,β(p,q)=2(1−α)(1−β)∫(1−(q(x)p(x))1−α2)(1−(q(x)p(x))1−β2)p(x)dx

### Bregman divergences

Main article: Bregman divergence

Bregman divergences correspond to convex functions on convex sets. Given a strictly convex, continuously differentiable function F on a convex set, known as the Bregman generator, the Bregman divergence measures the convexity of: the error of the linear approximation of F from q as an approximation of the value at p:

DF(p,q)=F(p)−F(q)−⟨∇F(q),p−q⟩.

The dual divergence to a Bregman divergence is the divergence generated by the convex conjugate F* of the Bregman generator of the original divergence. For example, for the squared Euclidean distance, the generator is x2, while for the relative entropy the generator is the negative entropy xlog⁡x.

## History

The use of the term "divergence" – both what functions it refers to, and what various statistical distances are called – has varied significantly over time, but by c. 2000 had settled on the current usage within information geometry, notably in the textbook (Amari Nagaoka).[1]

The term "divergence" for a statistical distance was used informally in various contexts from c. 1910 to c. 1940. Its formal use dates at least to (Bhattacharyya 1943), entitled "On a measure of divergence between two statistical populations defined by their probability distributions", which defined the Bhattacharyya distance, and (Bhattacharyya 1946), entitled "On a Measure of Divergence between Two Multinomial Populations", which defined the Bhattacharyya angle. The term was popularized by its use for the Kullback–Leibler divergence in (Kullback Leibler) and its use in the textbook (Kullback 1959). The term "divergence" was used generally by (Ali Silvey) for statistically distances. Numerous references to earlier uses of statistical distances are given in (Adhikari Joshi) and (Kullback 1959).

(Kullback Leibler) actually used "divergence" to refer to the symmetrized divergence (this function had already been defined and used by Harold Jeffreys in 1948[9]), referring to the asymmetric function as "the mean information for discrimination ... per observation",[10] while (Kullback 1959) referred to the asymmetric function as the "directed divergence".[11] (Ali Silvey) referred generally to such a function as a "coefficient of divergence", and showed that many existing functions could be expressed as f-divergences, referring to Jeffreys' function as "Jeffreys' measure of divergence" (today "Jeffreys divergence"), and Kullback–Leibler's asymmetric function (in each direction) as "Kullback's and Leibler's measures of discriminatory information" (today "Kullback–Leibler divergence").[12]

The information geometry definition of divergence (the subject of this article) was initially referred to by alternative terms, including "quasi-distance" (Amari 1982) and "contrast function" (Eguchi 1985), though "divergence" was used in (Amari 1985) for the α-divergence, and has become standard for the general class.[1][2]

The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.[13] For example, the term "Bregman distance" is still found, but "Bregman divergence" is now preferred.

Notationally, (Kullback Leibler) denoted their asymmetric function as I(1:2), while (Ali Silvey) denote their functions with a lowercase 'd' as d(P1,P2).

## See also

* Statistical distance

## Notes

1. ↑ Throughout, we only require differentiability class C2 (continuous with continuous first and second derivatives), since only second derivatives are required. In practice, commonly used statistical manifolds and divergences are infinitely differentiable ("smooth").
2. ↑ A colon is used in (Kullback Leibler), where the KL divergence between measure μ1 and μ2 is written as I(1:2).

## References

1. ↑ 1.0 1.1 1.2 Amari & Nagaoka 2000, chapter 3.2. sfn error: no target: CITEREFAmariNagaoka2000 (help)
2. ↑ 2.0 2.1 Amari 2016, p. 10, Definition 1.1. sfn error: no target: CITEREFAmari2016 (help)
3. ↑ 3.0 3.1 3.2 Amari 2016, p. 10. sfn error: no target: CITEREFAmari2016 (help)
4. ↑ (Eguchi 1992)
5. ↑ (Matumoto 1993)
6. ↑ Nielsen, F.; Nock, R. (2013). "On the Chi square and higher-order Chi distances for approximating f-divergences". IEEE Signal Processing Letters 21: 10–13. doi:10.1109/LSP.2013.2288355.
7. ↑ Csiszar 1991.
8. ↑ Jiao, Jiantao; Courtade, Thomas; No, Albert; Venkat, Kartik; Weissman, Tsachy (December 2014). "Information Measures: the Curious Case of the Binary Alphabet". IEEE Transactions on Information Theory 60 (12): 7616–7626. doi:10.1109/TIT.2014.2360184. ISSN 0018-9448.
9. ↑ Jeffreys 1948, p. 158.
10. ↑ Kullback & Leibler 1951, p. 80. sfn error: no target: CITEREFKullbackLeibler1951 (help)
11. ↑ Kullback 1959, p. 7. sfn error: no target: CITEREFKullback1959 (help)
12. ↑ Ali & Silvey 1966, p. 139.
13. ↑ Kullback 1959, p. 6. sfn error: no target: CITEREFKullback1959 (help)

### Bibliography

* Adhikari, B. P.; Joshi, D. D. (1956). "Distance, discrimination et résumé exhaustif". Pub. Inst. Stat. Univ. Paris 5: 57–74.
* Amari, Shun-Ichi (1982). "Differential Geometry of Curved Exponential Families-Curvatures and Information Loss". The Annals of Statistics 10 (2): 357–385. doi:10.1214/aos/1176345779. ISSN 0090-5364.
* Amari, Shun-Ichi (1985). Differential-Geometrical Methods in Statistics. Lecture Notes in Statistics. 28. Springer-Verlag.
* Methods of information geometry. Oxford University Press. 2000. ISBN 0-8218-0531-2.
* Information Geometry and Its Applications. Applied Mathematical Sciences. 194. Springer Japan. 2016. pp. XIII, 374. doi:10.1007/978-4-431-55978-8. ISBN 978-4-431-55977-1.
* Bhattacharyya, A. (1946). "On a Measure of Divergence between Two Multinomial Populations". Sankhyā: The Indian Journal of Statistics (1933-1960) 7 (4): 401–406. ISSN 0036-4452.
* Bhattacharyya, A. (1943). "On a measure of divergence between two statistical populations defined by their probability distributions". Bull. Calcutta Math. Soc. 35: 99–109.
* Csiszar, Imre (1 December 1991). "Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems". The Annals of Statistics 19 (4). doi:10.1214/aos/1176348385.
* Eguchi, Shinto (1985). "A differential geometric approach to statistical inference on the basis of contrast functionals". Hiroshima Mathematical Journal 15 (2): 341–391. doi:10.32917/hmj/1206130775.
* Eguchi, Shinto (1992). "Geometry of minimum contrast". Hiroshima Mathematical Journal 22 (3): 631–647. doi:10.32917/hmj/1206128508.
* Ali, S. M.; Silvey, S. D. (1966). "A General Class of Coefficients of Divergence of One Distribution from Another". Journal of the Royal Statistical Society. Series B (Methodological) 28 (1): 131–142. doi:10.1111/j.2517-6161.1966.tb00626.x. ISSN 0035-9246.
* Jeffreys, Harold (1948). Theory of Probability (Second ed.). Oxford University Press.
* "On information and sufficiency". Annals of Mathematical Statistics 22 (1): 79–86. 1951. doi:10.1214/aoms/1177729694.
* Information Theory and Statistics, John Wiley & Sons, 1959 . Republished by Dover Publications in 1968; reprinted in 1978: ISBN 0-8446-5625-9
* Matumoto, Takao (1993). "Any statistical manifold has a contrast function — on the C³-functions taking the minimum at the diagonal of the product manifold". Hiroshima Mathematical Journal 23 (2): 327–332. doi:10.32917/hmj/1206128255.

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Divergence (statistics). Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Divergence_(statistics)&oldid=222405"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template