Minimax estimator

In statistical decision theory, where we are faced with the problem of estimating a deterministic parameter (vector) [math]\displaystyle{ \theta \in \Theta }[/math] from observations [math]\displaystyle{ x \in \mathcal{X}, }[/math] an estimator (estimation rule) [math]\displaystyle{ \delta^M \,\\! }[/math] is called minimax if its maximal risk is minimal among all estimators of [math]\displaystyle{ \theta \,\\! }[/math]. In a sense this means that [math]\displaystyle{ \delta^M \,\\! }[/math] is an estimator which performs best in the worst possible case allowed in the problem.

## Contents

* 1 Problem setup
* 2 Definition
* 3 Least favorable distribution
* 4 Some examples
* 5 Asymptotic minimax estimator
* 6 Randomised minimax estimator
* 7 Relationship to robust optimization
* 8 References

## Problem setup

Consider the problem of estimating a deterministic (not Bayesian) parameter [math]\displaystyle{ \theta \in \Theta }[/math] from noisy or corrupt data [math]\displaystyle{ x \in \mathcal{X} }[/math] related through the conditional probability distribution [math]\displaystyle{ P(x\mid\theta)\,\\! }[/math]. Our goal is to find a "good" estimator [math]\displaystyle{ \delta(x) \,\\! }[/math] for estimating the parameter [math]\displaystyle{ \theta \,\\! }[/math], which minimizes some given risk function [math]\displaystyle{ R(\theta,\delta) \,\\! }[/math]. Here the risk function (technically a Functional or Operator since [math]\displaystyle{ R }[/math] is a function of a function, NOT function composition) is the expectation of some loss function [math]\displaystyle{ L(\theta,\delta) \,\\! }[/math] with respect to [math]\displaystyle{ P(x\mid\theta)\,\\! }[/math]. A popular example for a loss function[1] is the squared error loss [math]\displaystyle{ L(\theta,\delta)= \|\theta-\delta\|^2 \,\\! }[/math], and the risk function for this loss is the mean squared error (MSE).

Unfortunately, in general, the risk cannot be minimized since it depends on the unknown parameter [math]\displaystyle{ \theta \,\\! }[/math] itself (If we knew what was the actual value of [math]\displaystyle{ \theta \,\\! }[/math], we wouldn't need to estimate it). Therefore additional criteria for finding an optimal estimator in some sense are required. One such criterion is the minimax criterion.

## Definition

Definition : An estimator [math]\displaystyle{ \delta^M:\mathcal{X} \rightarrow \Theta \,\\! }[/math] is called minimax with respect to a risk function [math]\displaystyle{ R(\theta,\delta) \,\\! }[/math] if it achieves the smallest maximum risk among all estimators, meaning it satisfies

[math]\displaystyle{ \sup_{\theta \in \Theta} R(\theta,\delta^M) = \inf_\delta \sup_{\theta \in \Theta} R(\theta,\delta). \, }[/math]

## Least favorable distribution

Logically, an estimator is minimax when it is the best in the worst case. Continuing this logic, a minimax estimator should be a Bayes estimator with respect to a least favorable prior distribution of [math]\displaystyle{ \theta \,\\! }[/math]. To demonstrate this notion denote the average risk of the Bayes estimator [math]\displaystyle{ \delta_\pi \,\\! }[/math] with respect to a prior distribution [math]\displaystyle{ \pi \,\\! }[/math] as

[math]\displaystyle{ r_\pi = \int R(\theta,\delta_\pi) \, d\pi(\theta) \, }[/math]

Definition: A prior distribution [math]\displaystyle{ \pi \,\\! }[/math] is called least favorable if for every other distribution [math]\displaystyle{ \pi ' \,\\! }[/math] the average risk satisfies [math]\displaystyle{ r_\pi \geq r_{\pi '} \, }[/math].

Theorem 1: If [math]\displaystyle{ r_\pi = \sup_\theta R(\theta,\delta_\pi), \, }[/math] then:

1. [math]\displaystyle{ \delta_\pi\,\\! }[/math] is minimax.
2. If [math]\displaystyle{ \delta_\pi\,\\! }[/math] is a unique Bayes estimator, it is also the unique minimax estimator.
3. [math]\displaystyle{ \pi\,\\! }[/math] is least favorable.

Corollary: If a Bayes estimator has constant risk, it is minimax. Note that this is not a necessary condition.

Example 1: Unfair coin[2][3]: Consider the problem of estimating the "success" rate of a binomial variable, [math]\displaystyle{ x \sim B(n,\theta)\,\\! }[/math]. This may be viewed as estimating the rate at which an unfair coin falls on "heads" or "tails". In this case the Bayes estimator with respect to a Beta-distributed prior, [math]\displaystyle{ \theta \sim \text{Beta}(\sqrt{n}/2,\sqrt{n}/2) \, }[/math] is

[math]\displaystyle{ \delta^M=\frac{x+0.5\sqrt{n}}{n+\sqrt{n}}, \, }[/math]

with constant Bayes risk

[math]\displaystyle{ r=\frac{1}{4(1+\sqrt{n})^2} \, }[/math]

and, according to the Corollary, is minimax.

Definition: A sequence of prior distributions [math]\displaystyle{ \pi_n\,\\! }[/math] is called least favorable if for any other distribution [math]\displaystyle{ \pi '\,\\! }[/math],

[math]\displaystyle{ \lim_{n \rightarrow \infty} r_{\pi_n} \geq r_{\pi '}. \, }[/math]

Theorem 2: If there are a sequence of priors [math]\displaystyle{ \pi_n\,\\! }[/math] and an estimator [math]\displaystyle{ \delta\,\\! }[/math] such that [math]\displaystyle{ \sup_\theta R(\theta,\delta)=\lim_{n \rightarrow \infty} r_{\pi_n} \,\\! }[/math], then :

1. [math]\displaystyle{ \delta\,\\! }[/math] is minimax.
2. The sequence [math]\displaystyle{ \pi_n\,\\! }[/math] is least favorable.

Notice that no uniqueness is guaranteed here. For example, the ML estimator from the previous example may be attained as the limit of Bayes estimators with respect to a uniform prior, [math]\displaystyle{ \pi_n \sim U[-n,n]\,\\! }[/math] with increasing support and also with respect to a zero-mean normal prior [math]\displaystyle{ \pi_n \sim N(0,n \sigma^2) \,\\! }[/math] with increasing variance. So neither the resulting ML estimator is unique minimax nor the least favorable prior is unique.

Example 2: Consider the problem of estimating the mean of [math]\displaystyle{ p\,\\! }[/math] dimensional Gaussian random vector, [math]\displaystyle{ x \sim N(\theta,I_p \sigma^2)\,\\! }[/math]. The maximum likelihood (ML) estimator for [math]\displaystyle{ \theta\,\\! }[/math] in this case is simply [math]\displaystyle{ \delta_\text{ML}=x\,\\! }[/math], and its risk is

[math]\displaystyle{ R(\theta,\delta_\text{ML})=E{\|\delta_{ML}-\theta\|^2}=\sum_{i=1}^p E(x_i-\theta_i)^2=p \sigma^2. \, }[/math]

MSE of maximum likelihood estimator versus James–Stein estimator

The risk is constant, but the ML estimator is actually not a Bayes estimator, so the Corollary of Theorem 1 does not apply. However, the ML estimator is the limit of the Bayes estimators with respect to the prior sequence [math]\displaystyle{ \pi_n \sim N(0,n \sigma^2) \,\\! }[/math], and, hence, indeed minimax according to Theorem 2. Nonetheless, minimaxity does not always imply admissibility. In fact in this example, the ML estimator is known to be inadmissible (not admissible) whenever [math]\displaystyle{ p \gt 2\,\\! }[/math]. The famous James–Stein estimator dominates the ML whenever [math]\displaystyle{ p \gt 2\,\\! }[/math]. Though both estimators have the same risk [math]\displaystyle{ p \sigma^2\,\\! }[/math] when [math]\displaystyle{ \|\theta\| \rightarrow \infty\,\\! }[/math], and they are both minimax, the James–Stein estimator has smaller risk for any finite [math]\displaystyle{ \|\theta\|\,\\! }[/math]. This fact is illustrated in the following figure.

## Some examples

In general, it is difficult, often even impossible to determine the minimax estimator. Nonetheless, in many cases, a minimax estimator has been determined.

Example 3: Bounded normal mean: When estimating the mean of a normal vector [math]\displaystyle{ x \sim N(\theta,I_n \sigma^2)\,\\! }[/math], where it is known that [math]\displaystyle{ \|\theta\|^2 \leq M\,\\! }[/math]. The Bayes estimator with respect to a prior which is uniformly distributed on the edge of the bounding sphere is known to be minimax whenever [math]\displaystyle{ M \leq n\,\\! }[/math]. The analytical expression for this estimator is

[math]\displaystyle{ \delta^M(x)=\frac{MJ_{n+1}(M\|x\|)}{\|x\|J_{n}(M\|x\|)}x, \, }[/math]

where [math]\displaystyle{ J_n(t)\,\\! }[/math], is the modified Bessel function of the first kind of order n.

## Asymptotic minimax estimator

The difficulty of determining the exact minimax estimator has motivated the study of estimators of asymptotic minimax – an estimator [math]\displaystyle{ \delta' }[/math] is called [math]\displaystyle{ c }[/math]-asymptotic (or approximate) minimax if

[math]\displaystyle{ \sup_{\theta\in\Theta} R(\theta,\delta')\leq c \inf_\delta \sup_{\theta \in \Theta} R(\theta,\delta). }[/math]

For many estimation problems, especially in the non-parametric estimation setting, various approximate minimax estimators have been established. The design of the approximate minimax estimator is intimately related to the geometry, such as the metric entropy number, of [math]\displaystyle{ \Theta }[/math].

## Randomised minimax estimator

Sometimes, a minimax estimator may take the form of a randomised decision rule. An example is shown on the left. The parameter space has just two elements and each point on the graph corresponds to the risk of a decision rule: the x-coordinate is the risk when the parameter is [math]\displaystyle{ \theta_1 }[/math] and the y-coordinate is the risk when the parameter is [math]\displaystyle{ \theta_2 }[/math]. In this decision problem, the minimax estimator lies on a line segment connecting two deterministic estimators. Choosing [math]\displaystyle{ \delta_1 }[/math] with probability [math]\displaystyle{ 1 - p }[/math] and [math]\displaystyle{ \delta_2 }[/math] with probability [math]\displaystyle{ p }[/math] minimises the supremum risk.

## Relationship to robust optimization

Robust optimization is an approach to solve optimization problems under uncertainty in the knowledge of underlying parameters,.[4][5] For instance, the MMSE Bayesian estimation of a parameter requires the knowledge of parameter correlation function. If the knowledge of this correlation function is not perfectly available, a popular minimax robust optimization approach[6] is to define a set characterizing the uncertainty about the correlation function, and then pursuing a minimax optimization over the uncertainty set and the estimator respectively. Similar minimax optimizations can be pursued to make estimators robust to certain imprecisely known parameters. For instance, a recent study dealing with such techniques in the area of signal processing can be found in.[7]

In R. Fandom Noubiap and W. Seidel (2001) an algorithm for calculating a Gamma-minimax decision rule has been developed, when Gamma is given by a finite number of generalized moment conditions. Such a decision rule minimizes the maximum of the integrals of the risk function with respect to all distributions in Gamma. Gamma-minimax decision rules are of interest in robustness studies in Bayesian statistics.

## References

* E. L. Lehmann and G. Casella (1998), Theory of Point Estimation, 2nd ed. New York: Springer-Verlag.
* F. Perron and E. Marchand (2002), "On the minimax estimator of a bounded normal mean," Statistics and Probability Letters 58: 327–333.
* R. Fandom Noubiap and W. Seidel (2001), "An Algorithm for Calculating Gamma-Minimax Decision Rules under Generalized Moment Conditions," Annals of Statistics, August, 2001, vol. 29, no. 4, pp. 1094–1116
* "Estimation of the mean of a multivariate normal distribution". Annals of Statistics 9 (6): 1135–1151. 1981. doi:10.1214/aos/1176345632.

1. ↑ Statistical Decision Theory and Bayesian Analysis (2 ed.). New York: Springer-Verlag. 1985. pp. xv+425. ISBN 0-387-96098-8.
2. ↑ Hodges, Jr., J.L.; Lehmann, E.L. (1950). "Some problems in minimax point estimation". Ann. Math. Statist. 21 (2): 182–197. doi:10.1214/aoms/1177729838.
3. ↑ Steinhaus, Hugon (1957). "The problem of estimation". Ann. Math. Statist. 28 (3): 633–648. doi:10.1214/aoms/1177706876.
4. ↑ S. A. Kassam and H. V. Poor (1985), "Robust Techniques for Signal Processing: A Survey," Proceedings of the IEEE, vol. 73, pp. 433–481, March 1985.
5. ↑ A. Ben-Tal, L. El Ghaoui, and A. Nemirovski (2009), "Robust Optimization", Princeton University Press, 2009.
6. ↑ S. Verdu and H. V. Poor (1984), "On Minimax Robustness: A general approach and applications," IEEE Transactions on Information Theory, vol. 30, pp. 328–340, March 1984.
7. ↑ M. Danish Nisar. Minimax Robustness in Signal Processing for Communications, Shaker Verlag, ISBN:978-3-8440-0332-1, August 2011.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Minimax estimator. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Minimax_estimator&oldid=64394"