Statistical distance

Distance between two statistical objects

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages)

This article includes a list of general references, but it lacks sufficient corresponding inline citations. Please help to improve this article by introducing more precise citations. (February 2012) (Learn how and when to remove this message)

|

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.
Find sources: "Statistical distance" – news · newspapers · books · scholar · JSTOR (December 2020) (Learn how and when to remove this message)

|

(Learn how and when to remove this message)

In statistics, probability theory, and information theory, a statistical distance quantifies the distance between two statistical objects, which can be two random variables, or two probability distributions or samples, or the distance can be between an individual sample point and a population or a wider sample of points.

A distance between populations can be interpreted as measuring the distance between two probability distributions and hence they are essentially measures of distances between probability measures. Where statistical distance measures relate to the differences between random variables, these may have statistical dependence,[1] and hence these distances are not directly related to measures of distances between probability measures. Again, a measure of distance between random variables may relate to the extent of dependence between them, rather than to their individual values.

Many statistical distance measures are not metrics, and some are not symmetric. Some types of distance measures, which generalize squared distance, are referred to as (statistical) divergences.

## Terminology

[edit]

Many terms are used to refer to various notions of distance; these are often confusingly similar, and may be used inconsistently between authors and over time, either loosely or with precise technical meaning. In addition to "distance", similar terms include deviance, deviation, discrepancy, discrimination, and divergence, as well as others such as contrast function and metric. Terms from information theory include cross entropy, relative entropy, discrimination information, and information gain.

## Distances as metrics

[edit]

### Metrics

[edit]

A metric on a set X is a function (called the distance function or simply distance) d : X × X → R+ (where R+ is the set of non-negative real numbers). For all x, y, z in X, this function is required to satisfy the following conditions:

1. d(x, y) ≥ 0  (non-negativity)
2. d(x, y) = 0  if and only if  x = y (identity of indiscernibles. Note that condition 1 and 2 together produce positive definiteness)
3. d(x, y) = d(y, x)  (symmetry)
4. d(x, z) ≤ d(x, y) + d(y, z)  (subadditivity / triangle inequality).

### Generalized metrics

[edit]

Many statistical distances are not metrics, because they lack one or more properties of proper metrics. For example, pseudometrics violate property (2), identity of indiscernibles; quasimetrics violate property (3), symmetry; and semimetrics violate property (4), the triangle inequality. Statistical distances that satisfy (1) and (2) are referred to as divergences.

## Statistically close

[edit]

The  total variation distance of two distributions  X {\displaystyle X} and  Y {\displaystyle Y} over a finite domain  D {\displaystyle D} , (often referred to as statistical difference[2] or statistical distance[3] in cryptography) is defined as

Δ ( X , Y ) = 1 2 ∑ α ∈ D | Pr [ X = α ] − Pr [ Y = α ] | {\displaystyle \Delta (X,Y)={\frac {1}{2}}\sum _{\alpha \in D}|\Pr[X=\alpha ]-\Pr[Y=\alpha ]|} .

We say that two probability ensembles { X k } k ∈ N {\displaystyle \\{X_{k}\\}_{k\in \mathbb {N} }} and  { Y k } k ∈ N {\displaystyle \\{Y_{k}\\}_{k\in \mathbb {N} }} are statistically close if  Δ ( X k , Y k ) {\displaystyle \Delta (X_{k},Y_{k})} is a negligible function in  k {\displaystyle k} .

## Examples

[edit]

### Metrics

[edit]

* Total variation distance (sometimes just called "the" statistical distance)
* Hellinger distance
* Lévy–Prokhorov metric
* Wasserstein metric: also known as the Kantorovich metric, or earth mover's distance
* Mahalanobis distance
* Amari distance
* Integral probability metrics generalize several metrics or pseudometrics on distributions

### Divergences

[edit]

* Kullback–Leibler divergence
* Rényi divergence
* Jensen–Shannon divergence
* Bhattacharyya distance (despite its name it is not a distance, as it violates the triangle inequality)
* f-divergence: generalizes several distances and divergences
* Discriminability index, specifically the Bayes discriminability index, is a positive-definite symmetric measure of the overlap of two distributions.

## See also

[edit]

* Probabilistic metric space
* Randomness extractor
* Similarity measure
* Zero-knowledge proof

## Notes

[edit]

1. ^ Dodge, Y. (2003)--entry for distance
2. ^ Goldreich, Oded (2001). Foundations of Cryptography: Basic Tools (1st ed.). Berlin: Cambridge University Press. p. 106\. ISBN 0-521-79172-3.
3. ^ Reyzin, Leo. (Lecture Notes) Extractors and the Leftover Hash Lemma

## External links

[edit]

* Distance and Similarity Measures (Wolfram Alpha)

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* Arithmetic
* Arithmetic-Geometric
* Contraharmonic
* Cubic
* Generalized/power
* Geometric
* Harmonic
* Heronian
* Heinz
* Lehmer
* Median
* Mode

|

Dispersion|

* Average absolute deviation
* Coefficient of variation
* Interquartile range
* Percentile
* Range
* Standard deviation
* Variance

Shape|

* Central limit theorem
* Moments
* Kurtosis
* L-moments
* Skewness

Count data|

* Index of dispersion

Summary tables|

* Contingency table
* Frequency distribution
* Grouped data

Dependence|

* Partial correlation
* Pearson product-moment correlation
* Rank correlation
* Kendall's τ
* Spearman's ρ
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Radar chart
* Run chart
* Scatter plot
* Stem-and-leaf display
* Violin plot

Data collection

| Study design|

* Effect size
* Missing data
* Optimal design
* Population
* Replication
* Sample size determination
* Statistic
* Statistical power

|

Survey methodology|

* Sampling
* Cluster
* Stratified
* Opinion poll
* Questionnaire
* Standard error

Controlled experiments|

* Blocking
* Factorial experiment
* Interaction
* Random assignment
* Randomized controlled trial
* Randomized experiment
* Scientific control

Adaptive designs|

* Adaptive clinical trial
* Stochastic approximation
* Up-and-down designs

Observational studies|

* Cohort study
* Cross-sectional study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)
* Van der Waerden test

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test
* Cochran–Mantel–Haenszel statistics

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Least-squares spectral analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* Mathematics portal
* Commons
* WikiProject

## References

[edit]

* Dodge, Y. (2003) Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9

*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template