Maximum spacing estimation

Short description: Method of estimating a statistical model's parameters

The maximum spacing method tries to find a distribution function such that the spacings, D(i), are all approximately of the same length. This is done by maximizing their geometric mean.

In statistics, maximum spacing estimation (MSE or MSP), or maximum product of spacing estimation (MPS), is a method for estimating the parameters of a univariate statistical model.[1] The method requires maximization of the geometric mean of spacings in the data, which are the differences between the values of the cumulative distribution function at neighbouring data points.

The concept underlying the method is based on the probability integral transform, in that a set of independent random samples derived from any random variable should on average be uniformly distributed with respect to the cumulative distribution function of the random variable. The MPS method chooses the parameter values that make the observed data as uniform as possible, according to a specific quantitative measure of uniformity.

One of the most common methods for estimating the parameters of a distribution from data, the method of maximum likelihood (MLE), can break down in various cases, such as involving certain mixtures of continuous distributions.[2] In these cases the method of maximum spacing estimation may be successful.

Apart from its use in pure mathematics and statistics, the trial applications of the method have been reported using data from fields such as hydrology,[3] econometrics,[4] magnetic resonance imaging,[5] and others.[6]

## Contents

* 1 History and usage
* 2 Definition
* 3 Examples
* 3.1 Example 1
* 3.2 Example 2
* 4 Properties
* 4.1 Consistency and efficiency
* 4.2 Sensitivity
* 5 Moran test
* 6 Generalized maximum spacing
* 6.1 Alternate measures and spacings
* 6.2 Multivariate distributions
* 7 See also
* 8 Notes
* 9 References
* 9.1 Citations
* 9.2 Works cited

## History and usage

The MSE method was derived independently by Russel Cheng and Nik Amin at the University of Wales Institute of Science and Technology, and Bo Ranneby at the Swedish University of Agricultural Sciences.[2] The authors explained that due to the probability integral transform at the true parameter, the “spacing” between each observation should be uniformly distributed. This would imply that the difference between the values of the cumulative distribution function at consecutive observations should be equal. This is the case that maximizes the geometric mean of such spacings, so solving for the parameters that maximize the geometric mean would achieve the “best” fit as defined this way. (Ranneby 1984) justified the method by demonstrating that it is an estimator of the Kullback–Leibler divergence, similar to maximum likelihood estimation, but with more robust properties for some classes of problems.

There are certain distributions, especially those with three or more parameters, whose likelihoods may become infinite along certain paths in the parameter space. Using maximum likelihood to estimate these parameters often breaks down, with one parameter tending to the specific value that causes the likelihood to be infinite, rendering the other parameters inconsistent. The method of maximum spacings, however, being dependent on the difference between points on the cumulative distribution function and not individual likelihood points, does not have this issue, and will return valid results over a much wider array of distributions.[1]

The distributions that tend to have likelihood issues are often those used to model physical phenomena. (Hall al.) seek to analyze flood alleviation methods, which requires accurate models of river flood effects. The distributions that better model these effects are all three-parameter models, which suffer from the infinite likelihood issue described above, leading to Hall's investigation of the maximum spacing procedure. (Wong Li), when comparing the method to maximum likelihood, use various data sets ranging from a set on the oldest ages at death in Sweden between 1905 and 1958 to a set containing annual maximum wind speeds.

## Definition

Given an iid random sample {x1, ..., xn} of size n from a univariate distribution with continuous cumulative distribution function F(x;θ0), where θ0 ∈ Θ is an unknown parameter to be estimated, let {x(1), ..., x(n)} be the corresponding ordered sample, that is the result of sorting of all observations from smallest to largest. For convenience also denote x(0) = −∞ and x(n+1) = +∞.

Define the spacings as the “gaps” between the values of the distribution function at adjacent ordered points:[7] Di(θ)=F(x(i);θ)−F(x(i−1);θ),i=1,…,n+1.

Then the maximum spacing estimator of θ0 is defined as a value that maximizes the logarithm of the geometric mean of sample spacings: θ^=argmaxθ∈ΘSn(θ),where Sn(θ)=ln⁡D1D2⋯Dn+1n+1=1n+1∑i=1n+1ln⁡Di(θ).

By the inequality of arithmetic and geometric means, function Sn(θ) is bounded from above by −ln(n+1), and thus the maximum has to exist at least in the supremum sense.

Note that some authors define the function Sn(θ) somewhat differently. In particular, (Ranneby 1984) multiplies each Di by a factor of (n+1), whereas (Cheng Stephens) omit the ​1⁄n+1 factor in front of the sum and add the “−” sign in order to turn the maximization into minimization. As these are constants with respect to θ, the modifications do not alter the location of the maximum of the function Sn.

## Examples

This section presents two examples of calculating the maximum spacing estimator.

### Example 1

Plots of the log value of λ for the simplistic example under both likelihood and spacing estimation. The values for which both likelihood and spacing are maximized, the maximum likelihood and maximum spacing estimates, are identified.

Suppose two values x(1) = 2, x(2) = 4 were sampled from the exponential distribution F(x;λ) = 1 − e−xλ, x ≥ 0 with unknown parameter λ > 0\. In order to construct the MSE we have to first find the spacings:

i | F(x(i)) | F(x(i−1)) | Di = F(x(i)) − F(x(i−1))

|
|
|

1 | 1 − e−2λ | 0 | 1 − e−2λ
2 | 1 − e−4λ | 1 − e−2λ | e−2λ − e−4λ
3 | 1 | 1 − e−4λ | e−4λ

The process continues by finding the λ that maximizes the geometric mean of the “difference” column. Using the convention that ignores taking the (n+1)st root, this turns into the maximization of the following product: (1 − e−2λ) · (e−2λ − e−4λ) · (e−4λ). Letting μ = e−2λ, the problem becomes finding the maximum of μ5−2μ4+μ3. Differentiating, the μ has to satisfy 5μ4−8μ3+3μ2 = 0. This equation has roots 0, 0.6, and 1. As μ is actually e−2λ, it has to be greater than zero but less than one. Therefore, the only acceptable solution is μ=0.6⇒λMSE=ln⁡0.6−2≈0.255, which corresponds to an exponential distribution with a mean of ​1⁄λ ≈ 3.915. For comparison, the maximum likelihood estimate of λ is the inverse of the sample mean, 3, so λMLE = ⅓ ≈ 0.333.

### Example 2

Suppose {x(1), ..., x(n)} is the ordered sample from a uniform distribution U(a,b) with unknown endpoints a and b. The cumulative distribution function is F(x;a,b) = (x−a)/(b−a) when x∈[a,b]. Therefore, individual spacings are given by D1=x(1)−ab−a, Di=x(i)−x(i−1)b−a for i=2,…,n, Dn+1=b−x(n)b−a

Calculating the geometric mean and then taking the logarithm, statistic Sn will be equal to Sn(a,b)=ln⁡(x(1)−a)n+1+∑i=2nln⁡(x(i)−x(i−1))n+1+ln⁡(b−x(n))n+1−ln⁡(b−a) Here only three terms depend on the parameters a and b. Differentiating with respect to those parameters and solving the resulting linear system, the maximum spacing estimates will be

a^=nx(1)−x(n)n−1, b^=nx(n)−x(1)n−1.

These are known to be the uniformly minimum variance unbiased (UMVU) estimators for the continuous uniform distribution.[1] In comparison, the maximum likelihood estimates for this problem a^=x(1) and b^=x(n) are biased and have higher mean-squared error.

## Properties

### Consistency and efficiency

Density

Distribution

Plot of a “J-shaped” density function and its corresponding distribution. A shifted Weibull with a scale parameter of 15, a shape parameter of 0.5, and a location parameter of 10. The density asymptotically approaches infinity as x approaches 10, rendering the estimates of the other parameters inconsistent. Note that there is no inflection point in the graph of the distribution.

The maximum spacing estimator is a consistent estimator in that it converges in probability to the true value of the parameter, θ0, as the sample size increases to infinity.[2] The consistency of maximum spacing estimation holds under much more general conditions than for maximum likelihood estimators. In particular, in cases where the underlying distribution is J-shaped, maximum likelihood will fail where MSE succeeds.[1] An example of a J-shaped density is the Weibull distribution, specifically a shifted Weibull, with a shape parameter less than 1. The density will tend to infinity as x approaches the location parameter rendering estimates of the other parameters inconsistent.

Maximum spacing estimators are also at least as asymptotically efficient as maximum likelihood estimators, where the latter exist. However, MSEs may exist in cases where MLEs do not.[1]

### Sensitivity

Maximum spacing estimators are sensitive to closely spaced observations, and especially ties.[8] Given Xi+k=Xi+k−1=⋯=Xi, we get Di+k(θ)=Di+k−1(θ)=⋯=Di+1(θ)=0.

When the ties are due to multiple observations, the repeated spacings (those that would otherwise be zero) should be replaced by the corresponding likelihood.[1] That is, one should substitute fi(θ) for Di(θ), as limxi->xi−1∫xi−1xif(t;θ)dtxi−xi−1=f(xi−1,θ)=f(xi,θ), since xi=xi−1.

When ties are due to rounding error, (Cheng Stephens) suggest another method to remove the effects.[note 1] Given r tied observations from xi to xi+r−1, let δ represent the round-off error. All of the true values should then fall in the range x±δ. The corresponding points on the distribution should now fall between yL=F(x−δ,θ^) and yU=F(x+δ,θ^). Cheng and Stephens suggest assuming that the rounded values are uniformly spaced in this interval, by defining Dj=yU−yLr−1(j=i+1,…,i+r−1).

The MSE method is also sensitive to secondary clustering.[8] One example of this phenomenon is when a set of observations is thought to come from a single normal distribution, but in fact comes from a mixture normals with different means. A second example is when the data is thought to come from an exponential distribution, but actually comes from a gamma distribution. In the latter case, smaller spacings may occur in the lower tail. A high value of M(θ) would indicate this secondary clustering effect, and suggesting a closer look at the data is required.[8]

## Moran test

The statistic Sn(θ) is also a form of Moran or Moran-Darling statistic, M(θ), which can be used to test goodness of fit.[note 2] It has been shown that the statistic, when defined as Sn(θ)=Mn(θ)=−∑j=1n+1ln⁡Dj(θ), is asymptotically normal, and that a chi-squared approximation exists for small samples.[8] In the case where we know the true parameter θ0, (Cheng Stephens) show that the statistic Mn(θ) has a normal distribution with μM≈(n+1)(ln⁡(n+1)+γ)−12−112(n+1),σM2≈(n+1)(π26−1)−12−16(n+1), where γ is the Euler–Mascheroni constant which is approximately 0.57722.[note 3]

The distribution can also be approximated by that of A, where A=C1+C2χn2, in which C1=μM−σM2n2,C2=σM22n, and where χn2 follows a chi-squared distribution with n degrees of freedom. Therefore, to test the hypothesis H0 that a random sample of n values comes from the distribution F(x,θ), the statistic T(θ)=M(θ)−C1C2 can be calculated. Then H0 should be rejected with significance α if the value is greater than the critical value of the appropriate chi-squared distribution.[8]

Where θ0 is being estimated by θ^, (Cheng Stephens) showed that Sn(θ^)=Mn(θ^) has the same asymptotic mean and variance as in the known case. However, the test statistic to be used requires the addition of a bias correction term and is: T(θ^)=M(θ^)+k2−C1C2, where k is the number of parameters in the estimate.

## Generalized maximum spacing

### Alternate measures and spacings

(Ranneby Ekström) generalized the MSE method to approximate other measures besides the Kullback–Leibler measure. (Ekström 1997) further expanded the method to investigate properties of estimators using higher order spacings, where an m-order spacing would be defined as F(Xj+m)−F(Xj).

### Multivariate distributions

(Ranneby al.) discuss extended maximum spacing methods to the multivariate case. As there is no natural order for ℝk(k>1), they discuss two alternative approaches: a geometric approach based on Dirichlet cells and a probabilistic approach based on a “nearest neighbor ball” metric.

## See also

* Kullback–Leibler divergence
* Maximum likelihood
* Probability distribution

## Notes

1. ↑ There appear to be some minor typographical errors in the paper. For example, in section 4.2, equation (4.1), the rounding replacement for Dj, should not have the log term. In section 1, equation (1.2), Dj is defined to be the spacing itself, and M(θ) is the negative sum of the logs of Dj. If Dj is logged at this step, the result is always ≤ 0, as the difference between two adjacent points on a cumulative distribution is always ≤ 1, and strictly < 1 unless there are only two points at the bookends. Also, in section 4.3, on page 392, calculation shows that it is the variance σ2~ which has MPS estimate of 6.87, not the standard deviation σ~. – Editor
2. ↑ The literature refers to related statistics as Moran or Moran-Darling statistics. For example, (Cheng Stephens) analyze the form M(θ)=−∑j=1n+1log⁡Di(θ) where Di(θ) is defined as above. (Wong Li) use the same form as well. However, (Beirlant al.) uses the form Mn=−∑j=0nln⁡((n+1)(Xn,i+1−Xn,i)), with the additional factor of (n+1) inside the logged summation. The extra factors will make a difference in terms of the expected mean and variance of the statistic. For consistency, this article will continue to use the Cheng & Amin/Wong & Li form. -- Editor
3. ↑ (Wong Li) leave out the Euler–Mascheroni constant from their description. -- Editor

## References

### Citations

1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 (Cheng Amin)
2. ↑ 2.0 2.1 2.2 (Ranneby 1984)
3. ↑ (Hall al.)
4. ↑ (Anatolyev Kosenok)
5. ↑ (Pieciak 2014)
6. ↑ (Wong Li)
7. ↑ (Pyke 1965)
8. ↑ 8.0 8.1 8.2 8.3 8.4 (Cheng Stephens)

### Works cited

* Anatolyev, Stanislav; Kosenok, Grigory (2005). "An alternative to maximum likelihood based on spacings". Econometric Theory 21 (2): 472–476. doi:10.1017/S0266466605050255. http://fir.nes.ru/~gkosenok/MPS.pdf. Retrieved 2009-01-21.
* Beirlant, J.; Dudewicz, E.J.; Györfi, L.; van der Meulen, E.C. (1997). "Nonparametric entropy estimation: an overview". International Journal of Mathematical and Statistical Sciences 6 (1): 17–40. ISSN 1055-7490. http://www.menem.com/ilya/digital_library/entropy/beirlant_etal_97.pdf. Retrieved 2008-12-31. Note: linked paper is an updated 2001 version.
* Cheng, R.C.H.; Amin, N.A.K. (1983). "Estimating parameters in continuous univariate distributions with a shifted origin". Journal of the Royal Statistical Society, Series B 45 (3): 394–403. doi:10.1111/j.2517-6161.1983.tb01268.x. ISSN 0035-9246.
* Cheng, R.C.H; Stephens, M. A. (1989). "A goodness-of-fit test using Moran's statistic with estimated parameters". Biometrika 76 (2): 386–392. doi:10.1093/biomet/76.2.385.
* Ekström, Magnus (1997). "Generalized maximum spacing estimates". University of Umeå, Department of Mathematics 6. ISSN 0345-3928. http://www.matstat.umu.se/varia/reports/rep9706.ps.gz. Retrieved 2008-12-30.
* Hall, M.J.; van den Boogaard, H.F.P.; Fernando, R.C.; Mynett, A.E. (2004). "The construction of confidence intervals for frequency analysis using resampling techniques". Hydrology and Earth System Sciences 8 (2): 235–246. doi:10.5194/hess-8-235-2004. ISSN 1027-5606. https://hal.archives-ouvertes.fr/hal-00304907/document.
* Pieciak, Tomasz (2014). "The maximum spacing noise estimation in single-coil background MRI data". IEEE International Conference on Image Processing. Paris. pp. 1743–1747. http://home.agh.edu.pl/pieciak/publikacje_pieciak/2014_ICIP_Pieciak.pdf. Retrieved 2015-07-07.
* Pyke, Ronald (1965). "Spacings". Journal of the Royal Statistical Society, Series B 27 (3): 395–449. doi:10.1111/j.2517-6161.1965.tb00602.x. ISSN 0035-9246.
* Ranneby, Bo (1984). "The maximum spacing method. An estimation method related to the maximum likelihood method". Scandinavian Journal of Statistics 11 (2): 93–112. ISSN 0303-6898.
* Ranneby, Bo; Ekström, Magnus (1997). "Maximum spacing estimates based on different metrics". University of Umeå, Department of Mathematics 5. ISSN 0345-3928. http://www.matstat.umu.se/varia/reports/rep9705.ps.gz. Retrieved 2008-12-30.
* Ranneby, Bo; Jammalamadakab, S. Rao; Teterukovskiy, Alex (2005). "The maximum spacing estimation for multivariate observations". Journal of Statistical Planning and Inference 129 (1–2): 427–446. doi:10.1016/j.jspi.2004.06.059. http://www.pstat.ucsb.edu/faculty/jammalam/html/research publication_files/MSP2.pdf. Retrieved 2008-12-31.
* Wong, T.S.T; Li, W.K. (2006). "A note on the estimation of extreme value distributions using maximum product of spacings". Time series and related topics: in memory of Ching-Zong Wei. Institute of Mathematical Statistics Lecture Notes – Monograph Series. Beachwood, Ohio: Institute of Mathematical Statistic. pp. 272–283. doi:10.1214/074921706000001102. ISBN 978-0-940600-68-3.

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Maximum spacing estimation. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Maximum_spacing_estimation&oldid=222610"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template