Misuse of p-values

Short description: Misinterpretation of statistical significance

Misuse of p-values is common in scientific research and scientific education. p-values are often used or interpreted incorrectly;[1] the American Statistical Association states that p-values can indicate how incompatible the data are with a specified statistical model.[2] From a Neyman–Pearson hypothesis testing approach to statistical inferences, the data obtained by comparing the p-value to a significance level will yield one of two results: either the null hypothesis is rejected (which however does not prove that the null hypothesis is false), or the null hypothesis cannot be rejected at that significance level (which however does not prove that the null hypothesis is true). From a Fisherian statistical testing approach to statistical inferences, a low p-value means either that the null hypothesis is true and a highly improbable event has occurred or that the null hypothesis is false.

## Contents

* 1 Clarifications about p-values
* 2 Representing probabilities of hypotheses
* 3 Multiple comparisons problem
* 4 See also
* 5 References
* 6 Further reading

## Clarifications about p-values

The following list clarifies some issues that are commonly misunderstood regarding p-values:[2][3][4]

1. The p-value is not the probability that the null hypothesis is true, or the probability that the alternative hypothesis is false.[2] A p-value can indicate the degree of compatibility between a dataset and a particular hypothetical explanation (such as a null hypothesis). Specifically, the p-value can be taken as the probability of obtaining an effect that is at least as extreme as the observed effect, given that the null hypothesis is true. This should not be confused with the probability that the null hypothesis is true given the observed effect (see prosecutor's fallacy). In fact, frequentist statistics does not attach probabilities to hypotheses.
2. The p-value is not the probability that the observed effects were produced by random chance alone.[2] The p-value is computed under the assumption that a certain model, usually the null hypothesis, is true. This means that the p-value is a statement about the relation of the data to that hypothesis.[2]
3. The 0.05 significance level is merely a convention.[3][5] The 0.05 significance level (alpha level) is often used as the boundary between a statistically significant and a statistically non-significant p-value. However, this does not imply that there is generally a scientific reason to consider results on opposite sides of any threshold as qualitatively different.[3][6]
4. The p-value does not indicate the size or importance of the observed effect.[2] A small p-value can be observed for an effect that is not meaningful or important. In fact, the larger the sample size, the smaller the minimum effect needed to produce a statistically significant p-value (see effect size). Visualizing effect sizes is a critical component of a data-analysis method called estimation statistics.

## Representing probabilities of hypotheses

A frequentist approach rejects the validity of representing probabilities of hypotheses: hypotheses are true or false, not something that can be represented with a probability.[7]

Bayesian statistics actively models the likelihood of hypotheses. The p-value does not in itself allow reasoning about the probabilities of hypotheses, which requires multiple hypotheses or a range of hypotheses, with a prior distribution of likelihoods between them, in which case Bayesian statistics could be used. There, one uses a likelihood function for all possible values of the prior instead of the p-value for a single null hypothesis. The p-value describes a property of data when compared to a specific null hypothesis; it is not a property of the hypothesis itself. For the same reason, p-values do not give the probability that the data were produced by random chance alone.[2]

## Multiple comparisons problem

Main page: Multiple comparisons problem

The multiple comparisons problem occurs when one considers a set of statistical inferences simultaneously[8] or infers a subset of parameters selected based on the observed values.[9] It is also known as the look-elsewhere effect. Errors in inference, including confidence intervals that fail to include their corresponding population parameters or hypothesis tests that incorrectly reject the null hypothesis, are more likely to occur when one considers the set as a whole. Several statistical techniques have been developed to prevent this from happening, allowing significance levels for single and multiple comparisons to be directly compared. These techniques generally require a higher significance threshold for individual comparisons, so as to compensate for the number of inferences being made.[citation needed]

The webcomic xkcd satirized misunderstandings of p-values by portraying scientists investigating the claim that eating jellybeans caused acne.[10][11][12][13] After failing to find a significant (p < 0.05) correlation between eating jellybeans and acne, the scientists investigate 20 different colors of jellybeans individually, without adjusting for multiple comparisons. They find one color (green) nominally associated with acne (p < 0.05). The results are then reported by a newspaper as indicating that green jellybeans are linked to acne at a 95% confidence level—as if green were the only color tested. In fact, if 20 independent tests are conducted at the 0.05 significance level and all null hypotheses are true, there is a 64.2% chance of obtaining at least one false positive and the expected number of false positives is 1 (i.e. 0.05 × 20).

In general, the family-wise error rate (FWER)--the probability of obtaining at least one false positive--increases with the number of tests performed. The FWER when all null hypotheses are true for m independent tests, each conducted at significance level α, is:[12]

[math]\displaystyle{ \text{FWER}=1 - (1-\alpha)^m }[/math]

## See also

* Estimation statistics
* Replication crisis
* Metascience
* Misuse of statistics
* Statcheck

## References

1. ↑ "P-Values: Misunderstood and Misused". Frontiers in Physics 4 (6): 6. March 2016. doi:10.3389/fphy.2016.00006. Bibcode: 2016FrP.....4....6V.
2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 "The ASA's statement on p-values: context, process, and purpose". The American Statistician 70 (2): 129–133. 2016. doi:10.1080/00031305.2016.1154108. https://www.stat.berkeley.edu/~aldous/Real_World/ASA_statement.pdf.
3. ↑ 3.0 3.1 3.2 "Sifting the evidence-what's wrong with significance tests?". BMJ 322 (7280): 226–31. January 2001. doi:10.1136/bmj.322.7280.226. PMID 11159626.
4. ↑ "P values: What they are and what they are not". The American Statistician 50 (3): 203–206. 1996. doi:10.2307/2684655.
5. ↑ "Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise". BMC Medical Research Methodology 20 (1): 244. September 2020. doi:10.1186/s12874-020-01105-9. PMID 32998683.
6. ↑ "p > 0.05: significance thresholds and the crisis of unreplicable research". PeerJ 5: e3544. 2017. doi:10.7717/peerj.3544. PMID 28698825.
7. ↑ Chaput, Brigitte; Girard, Jean-Claude; Henry, Michel (2011). "Frequentist Approach: Modelling and Simulation in Statistics and Probability Teaching". Teaching Statistics in School Mathematics-Challenges for Teaching and Teacher Education. New ICMI Study Series. 14. pp. 85–95. doi:10.1007/978-94-007-1131-0_12. ISBN 978-94-007-1130-3.
8. ↑ Simultaneous Statistical Inference (2nd ed.). New York: Springer Verlag. 1981. ISBN 978-0-387-90548-8.
9. ↑ "Simultaneous and selective inference: Current successes and future challenges". Biometrical Journal. Biometrische Zeitschrift 52 (6): 708–21. December 2010. doi:10.1002/bimj.200900299. PMID 21154895.
10. ↑ "Significant". 6 April 2011. https://xkcd.com/882/.
11. ↑ "An investigation of the false discovery rate and the misinterpretation of p-values". Royal Society Open Science 1 (3): 140216. November 2014. doi:10.1098/rsos.140216. PMID 26064558. Bibcode: 2014RSOS....140216C.
12. ↑ 12.0 12.1 Statistics Done Wrong: The Woefully Complete Guide. No Starch Press. 2015. pp. 47–48. ISBN 978-1-59327-620-1. https://books.google.com/books?id=FFCIBwAAQBAJ&pg=PA48.
13. ↑ "Hypothesis testing and p values". 2 June 2014. http://blog.minitab.com/blog/statistics-in-the-field/hypothesis-testing-and-p-values.

## Further reading

* Wasserstein, Ronald L.; Schirm, Allen L.; Lazar, Nicole A. (2019-03-20). "Moving to a World Beyond "p < 0.05"". The American Statistician 73 (sup1): 1–19. doi:10.1080/00031305.2019.1583913. ISSN 0003-1305.
* "P-Values: Misunderstood and Misused". Frontiers in Physics 4 (6): 6. March 2016. doi:10.3389/fphy.2016.00006. Bibcode: 2016FrP.....4....6V.
* Ioannidis, John P. A. (29 March 2019). "What Have We (Not) Learnt from Millions of Scientific Papers with P Values?". The American Statistician 73 (sup1): 20–25. doi:10.1080/00031305.2018.1447512. ISSN 0003-1305.
* "A farewell to P-values?". Critical Care and Resuscitation 6 (2): 130–7. June 2004. PMID 16566700. https://www.cicm.org.au/CICM_Media/CICMSite/CICM-Website/Resources/Publications/CCR Journal/Previous Editions/June 2004/11_2004_Jun_Point-of-view.pdf.
* "Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P". British Journal of Pharmacology 166 (5): 1559–67. July 2012. doi:10.1111/j.1476-5381.2012.01931.x. PMID 22394284.
* "Scientific method: statistical errors". Nature 506 (7487): 150–2. February 2014. doi:10.1038/506150a. PMID 24522584. Bibcode: 2014Natur.506..150N.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Misuse of p-values. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Misuse_of_p-values&oldid=2637783"