Deviance (statistics)

Short description: Measure of goodness of fit for a statistical model

In statistics, deviance is a goodness-of-fit statistic for a statistical model; it is often used for statistical hypothesis testing. It is a generalization of the idea of using the sum of squares of residuals (SSR) in ordinary least squares to cases where model-fitting is achieved by maximum likelihood. It plays an important role in exponential dispersion models and generalized linear models.

Deviance can be related to Kullback-Leibler divergence.[1]

## Contents

* 1 Definition
* 2 Examples
* 3 See also
* 4 Notes
* 5 References
* 6 External links

## Definition

The unit deviance[2][3] d(y,μ) is a bivariate function that satisfies the following conditions:

* d(y,y)=0
* d(y,μ)>0∀y≠μ

The total deviance D(𝐲,μ^) of a model with predictions μ^ of the observation 𝐲 is the sum of its unit deviances: D(𝐲,μ^)=∑id(yi,μ^i).

The (total) deviance for a model M0 with estimates μ^=E[Y|θ^0], based on a dataset y, may be constructed by its likelihood as:[4][5] D(y,μ^)=2(log⁡[p(y∣θ^s)]−log⁡[p(y∣θ^0)]).

Here θ^0 denotes the fitted values of the parameters in the model M0, while θ^s denotes the fitted parameters for the saturated model: both sets of fitted values are implicitly functions of the observations y. Here, the saturated model is a model with a parameter for every observation so that the data are fitted exactly. This expression is simply 2 times the log-likelihood ratio of the full model compared to the reduced model. The deviance is used to compare two models - in particular in the case of generalized linear models (GLM) where it has a similar role to residual sum of squares from ANOVA in linear models (RSS).

Suppose in the framework of the GLM, we have two nested models, M1 and M2. In particular, suppose that M1 contains the parameters in M2, and k additional parameters. Then, under the null hypothesis that M2 is the true model, the difference between the deviances for the two models follows, based on Wilks' theorem, an approximate chi-squared distribution with k-degrees of freedom.[5] This can be used for hypothesis testing on the deviance.

Some usage of the term "deviance" can be confusing. According to Collett:[6]

"the quantity −2log⁡[p(y∣θ^0)] is sometimes referred to as a deviance. This is [...] inappropriate, since unlike the deviance used in the context of generalized linear modelling, −2log⁡[p(y∣θ^0)] does not measure deviation from a model that is a perfect fit to the data."

However, since the principal use is in the form of the difference of the deviances of two models, this confusion in definition is unimportant.

## Examples

The unit deviance for the Poisson distribution is d(y,μ)=2(ylog⁡yμ−y+μ), the unit deviance for the Normal distribution is given by d(y,μ)=(y−μ)2.

## See also

* Akaike information criterion
* Deviance information criterion
* Hosmer–Lemeshow test, a quality of fit statistic that can be used for binary data
* Pearson's chi-squared test, an alternative quality of fit statistic for generalized linear models for count data
* Peirce's criterion

## Notes

1. ↑ Hastie, Trevor. "A closer look at the deviance." The American Statistician 41.1 (1987): 16-20.
2. ↑ Jørgensen, B. (1997). The Theory of Dispersion Models. Chapman & Hall.
3. ↑ Song, Peter X. -K. (2007). Correlated Data Analysis: Modeling, Analytics, and Applications. Springer Series in Statistics. Springer Series in Statistics. doi:10.1007/978-0-387-71393-9. ISBN 978-0-387-71392-2.
4. ↑ Nelder, J.A.; Wedderburn, R.W.M. (1972). "Generalized Linear Models". Journal of the Royal Statistical Society. Series A (General) 135 (3): 370–384. doi:10.2307/2344614.
5. ↑ 5.0 5.1 McCullagh and Nelder (1989): page 17
6. ↑ Collett (2003): page 76

## References

* McCullagh, Peter; Nelder, John (1989). Generalized Linear Models, Second Edition. Chapman & Hall/CRC. ISBN 0-412-31760-5.

* Collett, David (2003). Modelling Survival Data in Medical Research, Second Edition. Chapman & Hall/CRC. ISBN 1-58488-325-1.

## External links

* Generalized Linear Models \- Edward F. Connor
* Lectures notes on Deviance

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

* v
* t
* e

Least squares and regression analysis

Computational statistics|

* Least squares
* Linear least squares
* Non-linear least squares
* Iteratively reweighted least squares

Correlation and dependence|

* Pearson product-moment correlation
* Rank correlation (Spearman's rho
* Kendall's tau)
* Partial correlation
* Confounding variable

Regression analysis|

* Ordinary least squares
* Partial least squares
* Total least squares
* Ridge regression

Regression as a
statistical model|

| Linear regression|

* Simple linear regression
* Ordinary least squares
* Generalized least squares
* Weighted least squares
* General linear model

|

Predictor structure|

* Polynomial regression
* Growth curve (statistics)
* Segmented regression
* Local regression

Non-standard|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Robust
* Quantile
* Isotonic

Non-normal errors|

* Generalized linear model
* Binomial
* Poisson
* Logistic

Decomposition of variance|

* Analysis of variance
* Analysis of covariance
* Multivariate AOV

Model exploration|

* Stepwise regression
* Model selection
* Mallows's Cp
* AIC
* BIC
* Model specification
* Regression validation

Background|

* Mean and predicted response
* Gauss–Markov theorem
* Errors and residuals
* Goodness of fit
* Studentized residual
* Minimum mean-square error

Design of experiments|

* Response surface methodology
* Optimal design
* Bayesian design

Numerical approximation|

* Numerical analysis
* Approximation theory
* Numerical integration
* Gaussian quadrature
* Orthogonal polynomials
* Chebyshev polynomials
* Chebyshev nodes

Applications|

* Curve fitting
* Calibration curve
* Numerical smoothing and differentiation
* System identification
* Moving least squares

* Regression analysis category
* Statistics category
*   * Statistics outline
* Statistics topics

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Deviance (statistics). Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Deviance_(statistics)&oldid=175540"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template