Maximum a posteriori estimation

Short description: Method of estimating the parameters of a statistical model

Part of a series on

Bayesian statistics
Theory

* Admissible decision rule
* Bayesian efficiency
* Bayesian probability
* Probability interpretations
* Bayes' theorem
* Bayes factor
* Bayesian inference
* Bayesian network
* Prior
* Posterior
* Likelihood
* Conjugate prior
* Posterior predictive
* Hyperparameter
* Hyperprior
* Principle of indifference
* Principle of maximum entropy
* Empirical Bayes method
* Cromwell's rule
* Bernstein–von Mises theorem
* Schwarz criterion
* Credible interval
* Maximum a posteriori estimation
* Radical probabilism

Techniques

* Bayesian linear regression
* Bayesian estimator
* Approximate Bayesian computation
* Markov chain Monte Carlo

* v
* t
* e

In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is an estimate of an unknown quantity, that equals the mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to the method of maximum likelihood (ML) estimation, but employs an augmented optimization objective which incorporates a prior distribution (that quantifies the additional information available through prior knowledge of a related event) over the quantity one wants to estimate. MAP estimation can therefore be seen as a regularization of maximum likelihood estimation.

## Contents

* 1 Description
* 2 Computation
* 3 Limitations
* 4 Example
* 5 References

## Description

Assume that we want to estimate an unobserved population parameter θ on the basis of observations x. Let f be the sampling distribution of x, so that f(x∣θ) is the probability of x when the underlying population parameter is θ. Then the function:

θ↦f(x∣θ)

is known as the likelihood function and the estimate:

θ^MLE(x)=argmaxθ f(x∣θ)

is the maximum likelihood estimate of θ.

Now assume that a prior distribution g over θ exists. This allows us to treat θ as a random variable as in Bayesian statistics. We can calculate the posterior distribution of θ using Bayes' theorem:

θ↦f(θ∣x)=f(x∣θ)g(θ)∫Θf(x∣ϑ)g(ϑ)dϑ

where g is density function of θ, Θ is the domain of g.

The method of maximum a posteriori estimation then estimates θ as the mode of the posterior distribution of this random variable:

θ^MAP(x)=argmaxθ f(θ∣x)=argmaxθ f(x∣θ)g(θ)∫Θf(x∣ϑ)g(ϑ)dϑ=argmaxθ f(x∣θ)g(θ).

The denominator of the posterior distribution (so-called marginal likelihood) is always positive and does not depend on θ and therefore plays no role in the optimization. Observe that the MAP estimate of θ coincides with the ML estimate when the prior g is uniform (i.e., g is a constant function).

When the loss function is of the form

L(θ,a)={0,if |a−θ|<c,1,otherwise,

as c goes to 0, the Bayes estimator approaches the MAP estimator, provided that the distribution of θ is quasi-concave.[1] But generally a MAP estimator is not a Bayes estimator unless θ is discrete.

## Computation

MAP estimates can be computed in several ways:

1. Analytically, when the mode(s) of the posterior distribution can be given in closed form. This is the case when conjugate priors are used.
2. Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
3. Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
4. Via a Monte Carlo method using simulated annealing

## Limitations

While only mild conditions are required for MAP estimation to be a limiting case of Bayes estimation (under the 0–1 loss function),[1] it is not very representative of Bayesian methods in general. This is because MAP estimates are point estimates, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively—which are more representative of typical loss functions—and for a continuous posterior distribution there is no loss function which suggests the MAP is the optimal point estimator. In addition, the posterior distribution may often not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find its mode(s) may be difficult or impossible.[citation needed]

An example of a density of a bimodal distribution in which the highest mode is uncharacteristic of the majority of the distribution

In many types of models, such as mixture models, the posterior may be multi-modal. In such a case, the usual recommendation is that one should choose the highest mode: this is not always feasible (global optimization is a difficult problem), nor in some cases even possible (such as when identifiability issues arise). Furthermore, the highest mode may be uncharacteristic of the majority of the posterior.

Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum.[2]

As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs x as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification h1, h2 and h3 with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, x, h1 classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier h1, x is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify x as negative.

## Example

Suppose that we are given a sequence (x1,…,xn) of IID N(μ,σv2) random variables and a prior distribution of μ is given by N(μ0,σm2). We wish to find the MAP estimate of μ. Note that the normal distribution is its own conjugate prior, so we will be able to find a closed-form solution analytically.

The function to be maximized is then given by

f(μ)f(x∣μ)=π(μ)L(μ)=12πσmexp⁡(−12(μ−μ0σm)2)∏j=1n12πσvexp⁡(−12(xj−μσv)2),

which is equivalent to minimizing the following function of μ:

∑j=1n(xj−μσv)2+(μ−μ0σm)2.

Thus, we see that the MAP estimator for μ is given by

μ^MAP=σm2nσm2n+σv2(1n∑j=1nxj)+σv2σm2n+σv2μ0=σm2(∑j=1nxj)+σv2μ0σm2n+σv2.

which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.

The case of σm->∞ is called a non-informative prior and leads to an ill-defined a priori probability distribution; in this case μ^MAP->μ^MLE.

## References

1. ↑ 1.0 1.1 Bassett, Robert; Deride, Julio (2018-01-30). "Maximum a posteriori estimators as a limit of Bayes estimators" (in en). Mathematical Programming: 1–16. doi:10.1007/s10107-018-1241-0. ISSN 0025-5610.
2. ↑ Murphy, Kevin P. (2012). Machine learning : a probabilistic perspective. Cambridge, Massachusetts: MIT Press. pp. 151–152. ISBN 978-0-262-01802-9.

* DeGroot, M. (1970). Optimal Statistical Decisions. McGraw-Hill. ISBN 0-07-016242-5.
* Sorenson, Harold W. (1980). Parameter Estimation: Principles and Problems. Marcel Dekker. ISBN 0-8247-6987-2.
* Hald, Anders (2007). "Gauss's Derivation of the Normal Distribution and the Method of Least Squares, 1809". A History of Parametric Statistical Inference from Bernoulli to Fisher, 1713–1935. New York: Springer. pp. 55–61. ISBN 978-0-387-46409-1.

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Maximum a posteriori estimation. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Maximum_a_posteriori_estimation&oldid=211223"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template