Parametric statistics

Short description: Branch of statistics

Parametric statistics is a branch of statistics that is concerned with the analysis of and inference from data assuming that the underlying distribution, from which the observed data was drawn, can be described by a finite set of (unknown) parameters.[1][2] In contrast, nonparametric statistics does not assume explicit (finite-parametric) mathematical forms for distributions when modeling data. However, it may make some assumptions about that distribution, such as continuity or symmetry, or even an explicit mathematical shape but have a model for a distributional parameter that is not itself finite-parametric.

Most well-known statistical methods are parametric.[3] Regarding nonparametric (and semiparametric) models, Sir David Cox has said, "These typically involve fewer assumptions of structure and distributional form but usually contain strong assumptions about independencies".[4]

## Contents

* 1 Overview
* 2 Parameter estimation
* 2.1 Frequentist approaches
* 2.2 Bayesian approaches
* 2.3 Theoretical guarantees
* 2.3.1 Uniformly minimum-variance unbiased estimators
* 2.3.2 Information inequality and exponential families
* 3 Parametric models
* 3.1 Density estimation
* 3.2 Regression
* 3.3 Classification
* 4 Example
* 5 History
* 6 See also
* 7 Notes
* 8 References

## Overview

The main goal of statistical inference is to provide methods to systematically analyse data and infer properties of the probability distribution from which the data was drawn. The fundamental assumption underlying parametric statistics is that the data distribution belongs to a more general family of distributions that can be parameterized by a finite number of parameters θ1,…,θp, which are unknown. Such a family of distributions is called a parametric model. Typical questions in this setting are:[2]

* Parameter estimation: Which choice of parameters best explains the observed data or leads to best predictions?
* Confidence intervals: What is a suitable range of parameters in which the true parameter lies with high probability?
* Hypothesis testing: Is it plausible that the true parameter lies within a certain range?
* Model (mis-)specification: What is an appropriate family of distributions that can explain the observed data?

## Parameter estimation

Main article: Point estimation

In the typical setting of parameter estimation, the chosen parametric family of distributions has either a density function or a density mass function Lθ, called the likelihood depending on the parameters θ=(θ1,…,θp), and a finite sample of data X1,…,Xn is given.

### Frequentist approaches

In a frequentist approach, the data is assumed to be distributed according to Lθ* for some "true" parameter θ*. The goal is to recover θ* from the observed data.[5] The most common methods are the following.

* Maximum Likelihood estimation (MLE): The model parameters are chosen such that the probability (or probability density) of the given observation is maximal.
* Method of moments (MoM): If the model parameters can be expressed as functions g1,…,gp of the moments of the distribution, then the moment estimates of the parameters are g1(∑i=1nXi,…,∑i=1nXip),…,gp(∑i=1nXi,…,∑i=1nXip).
* Least square estimation (LSE): This method applies to a regression setting, where the data arises in pairs (X1,Y1),…,(Xn,Yn) and a regression function f is to be determined. The model parameters are chosen such that the sum of squared differences ∑i=1n(Yi−fθ(Xi))2 between the observed data and the model prediction is minimal. In fact, LSE is a special case of MLE, where the conditional distribution of Y given X is normally distributed.[6]

### Bayesian approaches

In a Bayesian approach, the data is not assumed to be generated by a distribution Lθ* for some true θ*. Instead, the set of all possible (or plausible) model parameters is initially weighted with an à-priori distribution π reflecting the statistician's prior belief. Given the observed data, the parameter distribution is updated via Bayes' rule, yielding the à-posteriori distribution pθ that is proportional to the likelihood Lθ times the prior π. Bayesian estimators therefore rather give a best estimate given the statisticians beliefs.[5]

* Posterior mean estimation: The estimator takes the average of the posterior distribution.
* Posterior median estimation: The estimator takes the median of the posterior distribution.
* Maximum à-posteriori estimation (MAP): The estimator takes the parameter that maximizes the posterior distribution. If the prior is the uniform distribution, then MAP estimation is equivalent to maximum likelihood estimation.

### Theoretical guarantees

#### Uniformly minimum-variance unbiased estimators

Main article: Minimum-variance unbiased estimator

Uniformly minimum-variance unbiased estimators (UMVUE), sometimes called best unbiased estimators as well[7], are estimators that have minimum variance among all unbiased estimators. Due to the bias-variance decomposition, they are optimal in the sense that they minimise the mean squared error among all unbiased estimators.

Let's assume a function of the model parameters q(θ) needs to be estimated (e.g., mean, variance, median, or θ itself). Most generally, if an UMVU estimator of q(θ) with finite variance exists, then it must be unique due to the Rao-Blackwell theorem[8]. If there exists a complete sufficient statistic T in the chosen statistical model, which is often easy to determine using the Neyman-Fisher theorem, then it holds the Lehmann-Scheffé theorem: An estimators of the form f(T) that is unbiased, is automatically UMVU. If such an estimator has finite variance for all θ, then it is also the unique UMVUE.

#### Information inequality and exponential families

In regular statistical models, it can be shown that the variance of unbiased estimators cannot be arbitrarily small: Any unbiased estimator T of the quantity q(θ) is bounded from below by the universal bound[9]Varθ[T(X)]≥∇q(θ)I(θ)−1∇q(θ)T,where I(θ) is the Fisher-information matrix of the statistical model. The right hand-side is the so-called Cramér-Rao bound.

Estimators for which the variance equals the Cramér-Rao bound are called efficient and are also UMVU by definition. However, not every UMVUE is efficient. In fact, an estimator T is efficient if and only if (i) the statistical model is an exponential family and (ii) T is the natural sufficient statistic.[10]

## Parametric models

Main article: Parametric model

The choice of the model, that is, the probability distribution from which the data is assumed to be drawn in density estimation problems or an assumed functional depence between pairs of data X and Y in regression/classification problems, lies at the core of parametric procedures. Here is a list of common models used in practice.

### Density estimation

* Exponential families (e.g. normal distribution, exponential distribution, log-normal distribution, Gamma distribution, Chi-squared distribution, Erlang distribution, Beta distribution, Gumbel distribution, Pareto distribution, (Negative-)Binomial distribution, Poisson distribution, geometric distribution)
* Laplace distribution
* Uniform distribution
* Weibull distribution

### Regression

* linear model (special cases thereof are ANOVA and ANCOVA)
* generalized linear model (GLM)
* neural networks

### Classification

* logistic regression
* linear discriminant analysis (LDA)
* quadratic discriminant analysis (QDA)
* neural networks

## Example

The normal family of distributions all have the same general shape and are parameterized by mean and standard deviation. That means that if the mean and standard deviation are known and if the distribution is normal, the probability of any future observation lying in a given range is known.

Suppose that we have a sample of 99 test scores with a mean of 100 and a standard deviation of 1. If we assume all 99 test scores are random observations from a normal distribution, then we predict there is a 1% chance that the 100th test score will be higher than 102.33 (that is, the mean plus 2.33 standard deviations), assuming that the 100th test score comes from the same distribution as the others. Parametric statistical methods are used to compute the 2.33 value above, given 99 independent observations from the same normal distribution.

A non-parametric estimate of the same thing is the maximum of the first 99 scores. We don't need to assume anything about the distribution of test scores to reason that before we gave the test it was equally likely that the highest score would be any of the first 100. Thus there is a 1% chance that the 100th score is higher than any of the 99 that preceded it.

## History

Parametric statistics was mentioned by R. A. Fisher in his work Statistical Methods for Research Workers in 1925, which created the foundation for modern statistics.

## See also

* Aggregated distribution
* All models are wrong
* Inverse problem
* Parametric model
* Sufficient statistic

## Notes

1. ↑ Geisser, S. (2006), Modes of Parametric Statistical Inference, John Wiley & Sons
2. ↑ 2.0 2.1 Lehmann & Casella (1998), p. 1
3. ↑ Cox, D. R. (2006), Principles of Statistical Inference, Cambridge University Press
4. ↑ Cox (2006), p. 2
5. ↑ 5.0 5.1 Casella & Berger (2002), p. 324
6. ↑ Casella & Berger (2002), p. 549
7. ↑ Casella & Berger (2002), p. 334
8. ↑ Casella & Berger (2002), p. 343
9. ↑ Lehmann & Casella (1998), p. 120
10. ↑ Lehmann & Casella (1998), p. 121

## References

* Theory of Point Estimation (2nd ed.), New York: Springer, 1998, ISBN 0-387-98502-6
* Statistical Inference (2nd ed.), Duxbury, 2002
* Modes of Parametric Statistical Inference, John Wily & Sons, 2006
* Principles of Statistical Inference, Cambridge University Press, 2006, ISBN 978-0-511-34950-8

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Parametric statistics. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Parametric_statistics&oldid=4413392"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template