Multiple comparisons problem

Short description: Statistical interpretation with many tests

An example of coincidence produced by data dredging (uncorrected multiple comparisons) showing a correlation between the number of letters in a spelling bee's winning word and the number of people in the United States killed by venomous spiders. Given a large enough pool of variables for the same time period, it is possible to find a pair of graphs that show a spurious correlation.

Multiple comparisons, multiplicity or multiple testing problem occurs when many statistical tests are performed on the same dataset. Each test has its own chance of a Type I error (false positive), so the overall probability of making at least one false positive increases as the number of tests grows. In statistics, this occurs when one simultaneously considers a set of statistical inferences[1] or estimates a subset of selected parameters based on observed values.[2]

The probability of false positives is measured through the family-wise error rate (FWER). The larger the number of inferences made in a series of tests, the more likely erroneous inferences become. Several statistical techniques have been developed to compensate for the number of inferences being made—for example, by requiring a stricter significance threshold for individual comparisons.

## Contents

* 1 History
* 2 Definition
* 2.1 Classification of multiple hypothesis tests
* 3 Controlling procedures
* 3.1 Multiple testing correction
* 4 Large-scale multiple testing
* 4.1 Assessing whether any alternative hypotheses are true
* 5 See also
* 6 References
* 7 Further reading

## History

The problem of multiple comparisons received increased attention in the 1950s with the work of statisticians such as Tukey and Scheffé. Over the ensuing decades, many procedures were developed to address the problem. In 1996, the first international conference on multiple comparison procedures took place in Tel Aviv.[3] Several researchers are active in this area, for example Emmanuel Candès and Vladimir Vovk.

## Definition

Production of a small p-value by multiple testing.
30 samples of 10 dots of random color (blue or red) are observed. On each sample, a two-tailed binomial test of the null hypothesis that blue and red are equally probable is performed. The first row shows the possible p-values as a function of the number of blue and red dots in the sample.
Although the 30 samples were all simulated under the null, one of the resulting p-values is small enough to produce a false rejection at the typical level 0.05 in the absence of correction.

Multiple comparisons arise when a statistical analysis involves multiple simultaneous statistical tests, each of which has a potential to produce a "discovery". A stated confidence level generally applies only to each test considered individually, but often it is desirable to have a confidence level for the whole family of simultaneous tests.[4] Failure to compensate for multiple comparisons can have important real-world consequences, as illustrated by the following examples:

* Suppose the treatment is a new way of teaching writing to students, and the control is the standard way of teaching writing. Students in the two groups can be compared in terms of grammar, spelling, organization, content, and so on. As more attributes are compared, it becomes increasingly likely that the treatment and control groups will appear to differ on at least one attribute due to random sampling error alone.
* Suppose we consider the efficacy of a drug in terms of the reduction of any one of a number of disease symptoms. As more symptoms are considered, it becomes increasingly likely that the drug will appear to be an improvement over existing drugs in terms of at least one symptom.

In both examples, as the number of comparisons increases, it becomes more likely that the groups being compared will appear to differ in terms of at least one attribute. Our confidence that a result will generalize to independent data should generally be weaker if it is observed as part of an analysis that involves multiple comparisons, rather than an analysis that involves only a single comparison.

For example, if one test is performed at the 5% level and the corresponding null hypothesis is true, there is only a 5% risk of incorrectly rejecting the null hypothesis. However, if 100 tests are each conducted at the 5% level and all corresponding null hypotheses are true, the expected number of incorrect rejections (also known as false positives or Type I errors) is 5. If the tests are statistically independent from each other (i.e. are performed on independent samples), the probability of at least one incorrect rejection is approximately 99.4%.

The multiple comparisons problem also applies to confidence intervals. A single confidence interval with a 95% coverage probability level will contain the true value of the parameter in 95% of samples. However, if one considers 100 confidence intervals simultaneously, each with 95% coverage probability, the expected number of non-covering intervals is 5. If the intervals are statistically independent from each other, the probability that at least one interval does not contain the population parameter is 99.4%.

Techniques have been developed to prevent the inflation of false positive rates and non-coverage rates that occur with multiple statistical tests.

### Classification of multiple hypothesis tests

The following table defines the possible outcomes when testing multiple null hypotheses. Suppose we have a number m of null hypotheses, denoted by: H1, H2, ..., Hm. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null hypothesis if the test is non-significant. Summing each type of outcome over all Hi yields the following random variables:

| Null hypothesis is true (H0)  | Alternative hypothesis is true (HA)  | Total

|
|
|

Test is declared significant  | V | S | R
Test is declared non-significant  | U | T | m−R
Total  | m0 | m−m0 | m

* m is the total number hypotheses tested
* m0 is the number of true null hypotheses, an unknown parameter
* m−m0 is the number of true alternative hypotheses
* V is the number of false positives (Type I error) (also called "false discoveries")
* S is the number of true positives (also called "true discoveries")
* T is the number of false negatives (Type II error)
* U is the number of true negatives
* R=V+S is the number of rejected null hypotheses (also called "discoveries", either true or false)

In m hypothesis tests of which m0 are true null hypotheses, R is an observable random variable, and S, T, U, and V are unobservable random variables.

## Controlling procedures

Probability that at least one null hypothesis is wrongly rejected, for αper comparison=0.05, as a function of the number of independent tests m.

### Multiple testing correction

Multiple testing correction refers to making statistical tests more stringent in order to counteract the problem of multiple testing. The best known such adjustment is the Bonferroni correction, but other methods have been developed. Such methods are typically designed to control the family-wise error rate or the false discovery rate.

If m independent comparisons are performed, the family-wise error rate (FWER), is given by

α¯=1−(1−α{per comparison})m.

Hence, unless the tests are perfectly positively dependent (i.e., identical), α¯ increases as the number of comparisons increases. If we do not assume that the comparisons are independent, then we can still say:

α¯≤m⋅α{per comparison},

which follows from Boole's inequality. Example: 0.2649=1−(1−.05)6≤.05×6=0.3

There are different ways to assure that the family-wise error rate is at most α. The most conservative method, which is free of dependence and distributional assumptions, is the Bonferroni correction α{per comparison}=α/m. A marginally less conservative correction can be obtained by solving the equation for the family-wise error rate of m independent comparisons for α{per comparison}. This yields α{per comparison}=1−(1−α)1/m, which is known as the Šidák correction. Another procedure is the Holm–Bonferroni method, which uniformly delivers more power than the simple Bonferroni correction, by testing only the lowest p-value (i=1) against the strictest criterion, and the higher p-values (i>1) against progressively less strict criteria.[5] α{per comparison}=α/(m−i+1).

The classical Bonferroni method is easy to understand but should not be used since it is overly conservative. There are online calculators to correct for multiple comparisons using the more modern Holm-Bonferroni method or the Benjamini-Hochberg procedure.[6]

For continuous problems, one can employ Bayesian logic to compute m from the prior-to-posterior volume ratio. Continuous generalizations of the Bonferroni and Šidák correction are presented in.[7]

## Large-scale multiple testing

Traditional methods for multiple comparisons adjustments focus on correcting for modest numbers of comparisons, often in an analysis of variance. A different set of techniques have been developed for "large-scale multiple testing", in which thousands or even greater numbers of tests are performed. For example, in genomics, when using technologies such as microarrays, expression levels of tens of thousands of genes can be measured, and genotypes for millions of genetic markers can be measured. Particularly in the field of genetic association studies, there has been a serious problem with non-replication — a result being strongly statistically significant in one study but failing to be replicated in a follow-up study. Such non-replication can have many causes, but it is widely considered that failure to fully account for the consequences of making multiple comparisons is one of the causes.[8] It has been argued that advances in measurement and information technology have made it far easier to generate large datasets for exploratory analysis, often leading to the testing of large numbers of hypotheses with no prior basis for expecting many of the hypotheses to be true. In this situation, very high false positive rates are expected unless multiple comparisons adjustments are made.

For large-scale testing problems where the goal is to provide definitive results, the family-wise error rate remains the most accepted parameter for ascribing significance levels to statistical tests. Alternatively, if a study is viewed as exploratory, or if significant results can be easily re-tested in an independent study, control of the false discovery rate (FDR)[9][10][11] is often preferred. The FDR, loosely defined as the expected proportion of false positives among all significant tests, allows researchers to identify a set of "candidate positives" that can be more rigorously evaluated in a follow-up study.[12]

The practice of trying many unadjusted comparisons in the hope of finding a significant one is a known problem, whether applied unintentionally or deliberately, is sometimes called "p-hacking".[13][14]

### Assessing whether any alternative hypotheses are true

A normal quantile plot for a simulated set of test statistics that have been standardized to be Z-scores under the null hypothesis. The departure of the upper tail of the distribution from the expected trend along the diagonal is due to the presence of substantially more large test statistic values than would be expected if all null hypotheses were true. The red point corresponds to the fourth largest observed test statistic, which is 3.13, versus an expected value of 2.06. The blue point corresponds to the fifth smallest test statistic, which is -1.75, versus an expected value of -1.96. The graph suggests that it is unlikely that all the null hypotheses are true, and that most or all instances of a true alternative hypothesis result from deviations in the positive direction.

A basic question faced at the outset of analyzing a large set of testing results is whether there is evidence that any of the alternative hypotheses are true. One simple meta-test that can be applied when it is assumed that the tests are independent of each other is to use the Poisson distribution as a model for the number of significant results at a given level α that would be found when all null hypotheses are true. If the observed number of positives is substantially greater than what should be expected, this suggests that there are likely to be some true positives among the significant results.

For example, if 1000 independent tests are performed, each at level α = 0.05, we expect 0.05 × 1000 = 50 significant tests to occur when all null hypotheses are true. Based on the Poisson distribution with mean 50, the probability of observing more than 61 significant tests is less than 0.05, so if more than 61 significant results are observed, it is very likely that some of them correspond to situations where the alternative hypothesis holds. A drawback of this approach is that it overstates the evidence that some of the alternative hypotheses are true when the test statistics are positively correlated, which commonly occurs in practice. . On the other hand, the approach remains valid even in the presence of correlation among the test statistics, as long as the Poisson distribution can be shown to provide a good approximation for the number of significant results. This scenario arises, for instance, when mining significant frequent itemsets from transactional datasets. Furthermore, a careful two stage analysis can bound the FDR at a pre-specified level.[15]

## See also

* q-value

Key concepts

* Family-wise error rate
* False positive rate
* False discovery rate (FDR)
* False coverage rate (FCR)
* Interval estimation
* Post-hoc analysis
* Experimentwise error rate
* Statistical hypothesis testing

Single step procedures

* Bonferroni correction
* Dunnett's test
* Hochberg's GT2
* Scheffé's method
* Šidák correction
* Tukey's HSD

Two-stage "protected" procedures

* Duncan's new multiple range test
* Fisher's least significant difference
* Fisher-Hayter procedure
* Student-Newman-Keuls test

Sequential procedures

* Step-down
* Campbell and Blackwell's step-down procedure
* Holland-Copenhaver procedure
* Holm–Bonferroni method
* Schaffer's modified Holm Procedure
* Welsch's step-down procedure (also known as the Ryan-Einot-Gabriel-Welsch test)
* Step-up
* Benjamini–Hochberg procedure
* Benjamini–Yekutieli procedure
* Dunnett and Tamhane's step-up procedure
* Hochberg's step-up procedure
* Hommel's procedure

General methods of alpha adjustment for multiple comparisons

* Boole–Bonferroni bound
* Closed testing procedure
* E-values
* Harmonic mean p-value procedure

Related concepts

* Testing hypotheses suggested by the data
* Texas sharpshooter fallacy
* Model selection
* Look-elsewhere effect
* Data dredging
* Birthday problem

## References

1. ↑ Miller, R.G. (1981). Simultaneous Statistical Inference 2nd Ed. Springer Verlag New York. ISBN 978-0-387-90548-8.
2. ↑ Benjamini, Y. (2010). "Simultaneous and selective inference: Current successes and future challenges". Biometrical Journal 52 (6): 708–721. doi:10.1002/bimj.200900299. PMID 21154895. Bibcode: 2010BiomJ..52..708B.
3. ↑ "Home". http://www.mcp-conference.org/.
4. ↑ Kutner, Michael; Nachtsheim, Christopher; Neter, John; Li, William (2005). Applied Linear Statistical Models. McGraw-Hill Irwin. pp. 744–745. ISBN 9780072386882. https://archive.org/details/appliedlinearsta00kutn_164.
5. ↑ Aickin, M; Gensler, H (May 1996). "Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods". Am J Public Health 86 (5): 726–728. doi:10.2105/ajph.86.5.726. PMID 8629727.
6. ↑ Gunnarsson, Ronny K. (2026). "Level of Significance". INFOVOICE.SE. https://infovoice.se/level-of-significance/.
7. ↑ Bayer, Adrian E.; Seljak, Uroš (2020). "The look-elsewhere effect from a unified Bayesian and frequentist perspective". Journal of Cosmology and Astroparticle Physics 2020 (10): 009. doi:10.1088/1475-7516/2020/10/009. Bibcode: 2020JCAP...10..009B. https://doi.org/10.1088/1475-7516/2020/10/009.
8. ↑ Qu, Hui-Qi; Tien, Matthew; Polychronakos, Constantin (2010-10-01). "Statistical significance in genetic association studies". Clinical and Investigative Medicine 33 (5): E266–E270. ISSN 0147-958X. PMID 20926032.
9. ↑ Benjamini, Yoav; Hochberg, Yosef (1995). "Controlling the false discovery rate: a practical and powerful approach to multiple testing". Journal of the Royal Statistical Society, Series B 57 (1): 125–133. doi:10.1111/j.2517-6161.1995.tb02031.x.
10. ↑ Storey, JD; Tibshirani, Robert (2003). "Statistical significance for genome-wide studies". PNAS 100 (16): 9440–9445. doi:10.1073/pnas.1530509100. PMID 12883005. Bibcode: 2003PNAS..100.9440S.
11. ↑ Efron, Bradley; Tibshirani, Robert; Storey, John D.; Tusher, Virginia (2001). "Empirical Bayes analysis of a microarray experiment". Journal of the American Statistical Association 96 (456): 1151–1160. doi:10.1198/016214501753382129. Bibcode: 2001JASA...96.1151E.
12. ↑ Noble, William S. (2009-12-01). "How does multiple testing correction work?" (in en). Nature Biotechnology 27 (12): 1135–1137. doi:10.1038/nbt1209-1135. ISSN 1087-0156. PMID 20010596. Bibcode: 2009NatBi..27.1135N.
13. ↑ Young, S. S., Karr, A. (2011). "Deming, data and observational studies". Significance 8 (3): 116–120. doi:10.1111/j.1740-9713.2011.00506.x. http://www.niss.org/sites/default/files/Young Karr Obs Study Problem.pdf.
14. ↑ Smith, G. D., Shah, E. (2002). "Data dredging, bias, or confounding". BMJ 325 (7378): 1437–1438. doi:10.1136/bmj.325.7378.1437. PMID 12493654.
15. ↑ Kirsch, A; Mitzenmacher, M; Pietracaprina, A; Pucci, G; Upfal, E; Vandin, F (June 2012). "An Efficient Rigorous Approach for Identifying Statistically Significant Frequent Itemsets". Journal of the ACM 59 (3): 12:1–12:22. doi:10.1145/2220357.2220359.

## Further reading

* F. Bretz, T. Hothorn, P. Westfall (2010), Multiple Comparisons Using R, CRC Press
* S. Dudoit and M. J. van der Laan (2008), Multiple Testing Procedures with Application to Genomics, Springer
* Farcomeni, A. (2008). "A Review of Modern Multiple Hypothesis Testing, with particular attention to the false discovery proportion". Statistical Methods in Medical Research 17 (4): 347–388. doi:10.1177/0962280206079046. PMID 17698936.
* Phipson, B.; Smyth, G. K. (2010). "Permutation P-values Should Never Be Zero: Calculating Exact P-values when Permutations are Randomly Drawn". Statistical Applications in Genetics and Molecular Biology 9: Article39. doi:10.2202/1544-6115.1585. PMID 21044043.
* P. H. Westfall and S. S. Young (1993), Resampling-based Multiple Testing: Examples and Methods for p-Value Adjustment, Wiley
* P. Westfall, R. Tobias, R. Wolfinger (2011) Multiple comparisons and multiple testing using SAS, 2nd edn, SAS Institute
* A gallery of examples of implausible correlations sourced by data dredging
* [1] An xkcd comic about the multiple comparisons problem, using jelly beans and acne as an example

* v
* t
* e

Design of experiments

Scientific
method|

* Scientific experiment
* Statistical design
* Control
* Internal and external validity
* Experimental unit
* Blinding

* Optimal design: Bayesian
* Random assignment
* Randomization
* Restricted randomization
* Replication versus subsampling
* Sample size

Treatment
and blocking|

* Treatment
* Effect size
* Contrast
* Interaction
* Confounding
* Orthogonality
* Blocking
* Covariate
* Nuisance variable

Models
and inference|

* Linear regression
* Ordinary least squares
* Bayesian

* Random effect
* Mixed model
* Hierarchical model: Bayesian

* Analysis of variance (Anova)
* Cochran's theorem
* Manova (multivariate)
* Ancova (covariance)

* Compare means
* Multiple comparison

Designs

Completely
randomized|

* Factorial
* Fractional factorial
* Plackett-Burman
* Taguchi

* Response surface methodology
* Polynomial and rational modeling
* Box-Behnken
* Central composite

* Block
* Generalized randomized block design (GRBD)
* Latin square
* Graeco-Latin square
* Orthogonal array
* Latin hypercube
Repeated measures design
* Crossover study

* Randomized controlled trial
* Sequential analysis
* Sequential probability ratio test

* Glossary
* Category
*   * Statistical outline
* Statistical topics

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Multiple comparisons problem. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Multiple_comparisons_problem&oldid=4411028"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template