Double descent

Short description: Concept in machine learning

An example of the double descent phenomenon in a two-layer neural network: as the ratio of parameters to data points increases, the test error first falls, then rises, then falls again.[1] The vertical line marks the "interpolation threshold" boundary between the underparametrized region (more data points than parameters) and the overparameterized region (more parameters than data points).Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

Double descent in statistics and machine learning is the phenomenon where a model's error rate on the test set initially decreases with the number of parameters, then peaks, then decreases again.[2] This phenomenon has been considered surprising, as it contradicts assumptions about overfitting in classical machine learning.[3]

The increase usually occurs near the interpolation threshold, where the number of parameters is the same as the number of training data points (the model is just large enough to fit the training data). Or, more precisely, it is the maximum number of samples on which the model/training procedure achieves approximately on average 0 training error.[4]

## Contents

* 1 History
* 2 Theoretical models
* 3 See also
* 4 References
* 5 Further reading
* 6 External links

## History

Early observations of what would later be called double descent in specific models date back to 1989.[5][6]

The term "double descent" was coined by Belkin et. al.[7] in 2019,[3] when the phenomenon gained popularity as a broader concept exhibited by many models.[8][9] The latter development was prompted by a perceived contradiction between the conventional wisdom that too many parameters in the model result in a significant overfitting error (an extrapolation of the bias–variance tradeoff),[10] and the empirical observations in the 2010s that some modern machine learning techniques tend to perform better with larger models.[7][11]

## Theoretical models

Double descent occurs in linear regression with isotropic Gaussian covariates and isotropic Gaussian noise.[12]

A model of double descent at the thermodynamic limit has been analyzed using the replica trick, and the result has been confirmed numerically.[13]

A number of works[14][15] have suggested that double descent can be explained using the concept of effective dimension: While a network may have a large number of parameters, in practice only a subset of those parameters are relevant for generalization performance, as measured by the local Hessian curvature. This explanation is formalized through PAC-Bayes compression-based generalization bounds,[16] which show that less complex models are expected to generalize better under a Solomonoff prior.

## See also

* Grokking (machine learning)

## References

1. ↑ Rocks, Jason W. (2022). "Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models". Physical Review Research 4 (1). doi:10.1103/PhysRevResearch.4.013201. PMID 36713351. Bibcode: 2022PhRvR...4a3201R.
2. ↑ "Deep Double Descent" (in en). 2019-12-05. https://openai.com/blog/deep-double-descent/.
3. ↑ 3.0 3.1 Schaeffer, Rylan; Khona, Mikail; Robertson, Zachary; Boopathy, Akhilan; Pistunova, Kateryna; Rocks, Jason W.; Fiete, Ila Rani; Koyejo, Oluwasanmi (2023-03-24). "Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle". arXiv:2303.14151v1 [cs.LG].
4. ↑ Nakkiran, Preetum; Kaplun, Gal; Bansal, Yamini; Yang, Tristan; Barak, Boaz; Sutskever, Ilya (2019-12-04). "Deep Double Descent: Where Bigger Models and More Data Hurt". arXiv:1912.02292 [cs.LG].
5. ↑ Vallet, F.; Cailton, J.-G.; Refregier, Ph (June 1989). "Linear and Nonlinear Extension of the Pseudo-Inverse Solution for Learning Boolean Functions" (in en). Europhysics Letters 9 (4): 315. doi:10.1209/0295-5075/9/4/003. ISSN 0295-5075. Bibcode: 1989EL......9..315V. https://dx.doi.org/10.1209/0295-5075/9/4/003.
6. ↑ Loog, Marco; Viering, Tom; Mey, Alexander; Krijthe, Jesse H.; Tax, David M. J. (2020-05-19). "A brief prehistory of double descent" (in en). Proceedings of the National Academy of Sciences 117 (20): 10625–10626. doi:10.1073/pnas.2001875117. ISSN 0027-8424. PMID 32371495. Bibcode: 2020PNAS..11710625L.
7. ↑ 7.0 7.1 Belkin, Mikhail; Hsu, Daniel; Ma, Siyuan; Mandal, Soumik (2019-08-06). "Reconciling modern machine learning practice and the bias-variance trade-off". Proceedings of the National Academy of Sciences 116 (32): 15849–15854. doi:10.1073/pnas.1903070116. ISSN 0027-8424. PMID 31341078.
8. ↑ Spigler, Stefano; Geiger, Mario; d'Ascoli, Stéphane; Sagun, Levent; Biroli, Giulio; Wyart, Matthieu (2019-11-22). "A jamming transition from under- to over-parametrization affects loss landscape and generalization". Journal of Physics A: Mathematical and Theoretical 52 (47): 474001. doi:10.1088/1751-8121/ab4c8b. ISSN 1751-8113.
9. ↑ Viering, Tom; Loog, Marco (2023-06-01). "The Shape of Learning Curves: A Review". IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (6): 7799–7819. doi:10.1109/TPAMI.2022.3220744. ISSN 0162-8828. PMID 36350870. Bibcode: 2023ITPAM..45.7799V.
10. ↑ Geman, Stuart; Bienenstock, Élie; Doursat, René (1992). "Neural networks and the bias/variance dilemma". Neural Computation 4: 1–58. doi:10.1162/neco.1992.4.1.1. http://web.mit.edu/6.435/www/Geman92.pdf.
11. ↑ Preetum Nakkiran; Gal Kaplun; Yamini Bansal; Tristan Yang; Boaz Barak; Ilya Sutskever (29 December 2021). "Deep double descent: where bigger models and more data hurt". Theory and Experiment (IOP Publishing Ltd and SISSA Medialab srl) 2021 (12): 124003. doi:10.1088/1742-5468/ac3a74. Bibcode: 2021JSMTE2021l4003N.
12. ↑ Nakkiran, Preetum (2019-12-16). "More Data Can Hurt for Linear Regression: Sample-wise Double Descent". arXiv:1912.07242v1 [stat.ML].
13. ↑ Advani, Madhu S.; Saxe, Andrew M.; Sompolinsky, Haim (2020-12-01). "High-dimensional dynamics of generalization error in neural networks". Neural Networks 132: 428–446. doi:10.1016/j.neunet.2020.08.022. ISSN 0893-6080. PMID 33022471.
14. ↑ Maddox, Wesley J.; Benton, Gregory W.; Wilson, Andrew Gordon (2020). "Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited". arXiv:2003.02139 [cs.LG].
15. ↑ Wilson, Andrew Gordon (2025). "Deep Learning is Not So Mysterious or Different". arXiv:2503.02113 [cs.LG].
16. ↑ Lotfi, Sanae; Finzi, Marc; Kapoor, Sanyam; Potapczynski, Andres; Goldblum, Micah; Wilson, Andrew G. (2022). "PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization". Advances in Neural Information Processing Systems. 35. pp. 31459–31473. https://proceedings.neurips.cc/paper_files/paper/2022/file/cbeec55c50c3367024bafab2438a021b-Paper-Conference.pdf.

## Further reading

* Mikhail Belkin; Daniel Hsu; Ji Xu (2020). "Two Models of Double Descent for Weak Features". SIAM Journal on Mathematics of Data Science 2 (4): 1167–1180. doi:10.1137/20M1336072.
* Mount, John (3 April 2024). "The m = n Machine Learning Anomaly". https://win-vector.com/2024/04/03/the-m-n-machine-learning-anomaly/.
* Preetum Nakkiran; Gal Kaplun; Yamini Bansal; Tristan Yang; Boaz Barak; Ilya Sutskever (29 December 2021). "Deep double descent: where bigger models and more data hurt". Theory and Experiment (IOP Publishing Ltd and SISSA Medialab srl) 2021 (12): 124003. doi:10.1088/1742-5468/ac3a74. Bibcode: 2021JSMTE2021l4003N.
* Song Mei; Andrea Montanari (April 2022). "The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve". Communications on Pure and Applied Mathematics 75 (4): 667–766. doi:10.1002/cpa.22008.
* Xiangyu Chang; Yingcong Li; Samet Oymak; Christos Thrampoulidis (2021). "Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks". Proceedings of the AAAI Conference on Artificial Intelligence 35 (8).
* Manuchehr Aminian: "Characterizations of Double Descent", SIAM News, Vol.58, No.10 (Dec.,2025).

## External links

* "Double Descent: Part 1: A Visual Introduction". https://mlu-explain.github.io/double-descent/.
* "Double Descent: Part 2: A Mathematical Explanation". https://mlu-explain.github.io/double-descent2/.
* Understanding "Deep Double Descent" at evhub.

* v
* t
* e

Statistics

* Outline
* Index

Descriptive statistics

| Continuous data|

| Center|

* Mean
* arithmetic
* geometric
* harmonic
* Median
* Mode

|

Dispersion|

* Variance
* Standard deviation
* Coefficient of variation
* Percentile
* Range
* Interquartile range

Shape|

* Central limit theorem
* Moments
* Skewness
* Kurtosis
* L-moments

Count data|

* Index of dispersion

Summary tables|

* Grouped data
* Frequency distribution
* Contingency table

Dependence|

* Pearson product-moment correlation
* Rank correlation
* Spearman's rho
* Kendall's tau
* Partial correlation
* Scatter plot

Graphics|

* Bar chart
* Biplot
* Box plot
* Control chart
* Correlogram
* Fan chart
* Forest plot
* Histogram
* Pie chart
* Q–Q plot
* Run chart
* Scatter plot
* Stem-and-leaf display
* Radar chart
* Violin plot

Data collection

| Study design|

* Population
* Statistic
* Effect size
* Statistical power
* Optimal design
* Sample size determination
* Replication
* Missing data

|

Survey methodology|

* Sampling
* stratified
* cluster
* Standard error
* Opinion poll
* Questionnaire

Controlled experiments|

* Scientific control
* Randomized experiment
* Randomized controlled trial
* Random assignment
* Blocking
* Interaction
* Factorial experiment

Adaptive Designs|

* Adaptive clinical trial
* Up-and-Down Designs
* Stochastic approximation

Observational Studies|

* Cross-sectional study
* Cohort study
* Natural experiment
* Quasi-experiment

Statistical inference

| Statistical theory|

* Population
* Statistic
* Probability distribution
* Sampling distribution
* Order statistic
* Empirical distribution
* Density estimation
* Statistical model
* Model specification
* Lp space
* Parameter
* location
* scale
* shape
* Parametric family
* Likelihood (monotone)
* Location–scale family
* Exponential family
* Completeness
* Sufficiency
* Statistical functional
* Bootstrap
* U
* V
* Optimal decision
* loss function
* Efficiency
* Statistical distance
* divergence
* Asymptotics
* Robustness

|

Frequentist inference|

| Point estimation|

* Estimating equations
* Maximum likelihood
* Method of moments
* M-estimator
* Minimum distance
* Unbiased estimators
* Mean-unbiased minimum-variance
* Rao–Blackwellization
* Lehmann–Scheffé theorem
* Median unbiased
* Plug-in

|

Interval estimation|

* Confidence interval
* Pivot
* Likelihood interval
* Prediction interval
* Tolerance interval
* Resampling
* Bootstrap
* Jackknife

Testing hypotheses|

* 1- & 2-tails
* Power
* Uniformly most powerful test
* Permutation test
* Randomization test
* Multiple comparisons

Parametric tests|

* Likelihood-ratio
* Score/Lagrange multiplier
* Wald

Specific tests|

* Z-test (normal)
* Student's t-test
* F-test

Goodness of fit|

* Chi-squared
* G-test
* Kolmogorov–Smirnov
* Anderson–Darling
* Lilliefors
* Jarque–Bera
* Normality (Shapiro–Wilk)
* Likelihood-ratio test
* Model selection
* Cross validation
* AIC
* BIC

Rank statistics|

* Sign
* Sample median
* Signed rank (Wilcoxon)
* Hodges–Lehmann estimator
* Rank sum (Mann–Whitney)
* Nonparametric anova
* 1-way (Kruskal–Wallis)
* 2-way (Friedman)
* Ordered alternative (Jonckheere–Terpstra)

Bayesian inference|

* Bayesian probability
* prior
* posterior
* Credible interval
* Bayes factor
* Bayesian estimator
* Maximum posterior estimator

* Correlation
* Regression analysis

| Correlation|

* Pearson product-moment
* Partial correlation
* Confounding variable
* Coefficient of determination

|

Regression analysis|

* Errors and residuals
* Regression validation
* Mixed effects models
* Simultaneous equations models
* Multivariate adaptive regression splines (MARS)

Linear regression|

* Simple linear regression
* Ordinary least squares
* General linear model
* Bayesian regression

Non-standard predictors|

* Nonlinear regression
* Nonparametric
* Semiparametric
* Isotonic
* Robust
* Heteroscedasticity
* Homoscedasticity

Generalized linear model|

* Exponential families
* Logistic (Bernoulli) / Binomial / Poisson regressions

Partition of variance|

* Analysis of variance (ANOVA, anova)
* Analysis of covariance
* Multivariate ANOVA
* Degrees of freedom

Categorical / Multivariate / Time-series / Survival analysis

| Categorical|

* Cohen's kappa
* Contingency table
* Graphical model
* Log-linear model
* McNemar's test

|

Multivariate|

* Regression
* Manova
* Principal components
* Canonical correlation
* Discriminant analysis
* Cluster analysis
* Classification
* Structural equation model
* Factor analysis
* Multivariate distributions
* Elliptical distributions
* Normal

Time-series|

| General|

* Decomposition
* Trend
* Stationarity
* Seasonal adjustment
* Exponential smoothing
* Cointegration
* Structural break
* Granger causality

|

Specific tests|

* Dickey–Fuller
* Johansen
* Q-statistic (Ljung–Box)
* Durbin–Watson
* Breusch–Godfrey

Time domain|

* Autocorrelation (ACF)
* partial (PACF)
* Cross-correlation (XCF)
* ARMA model
* ARIMA model (Box–Jenkins)
* Autoregressive conditional heteroskedasticity (ARCH)
* Vector autoregression (VAR)

Frequency domain|

* Spectral density estimation
* Fourier analysis
* Wavelet
* Whittle likelihood

Survival|

| Survival function|

* Kaplan–Meier estimator (product limit)
* Proportional hazards models
* Accelerated failure time (AFT) model
* First hitting time

|

Hazard function|

* Nelson–Aalen estimator

Test|

* Log-rank test

Applications

| Biostatistics|

* Bioinformatics
* Clinical trials / studies
* Epidemiology
* Medical statistics

|

Engineering statistics|

* Chemometrics
* Methods engineering
* Probabilistic design
* Process / quality control
* Reliability
* System identification

Social statistics|

* Actuarial science
* Census
* Crime statistics
* Demography
* Econometrics
* Jurimetrics
* National accounts
* Official statistics
* Population statistics
* Psychometrics

Spatial statistics|

* Cartography
* Environmental statistics
* Geographic information system
* Geostatistics
* Kriging

* Category
* '
* Commons
* WikiProject

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Glossary
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large
* NMT
* Reasoning
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Lethal autonomous weapons (LAWs)
* Generative artificial intelligence (GenAI)
* (Hypothetical: Artificial general intelligence (AGI))
* (Hypothetical: Artificial superintelligence (ASI))
* Agent2Agent protocol

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* GPT Image
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Seedance
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* 5.1
* 5.2
* Claude
* Gemini
* Gemini (language model)
* Gemma
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Christopher D. Manning
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Alex Krizhevsky
* Ilya Sutskever
* Oriol Vinyals
* Quoc V. Le
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* John Schulman
* Mustafa Suleyman
* Jan Leike
* Daniel Kokotajlo
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

Political|

* AI safety (Alignment)
* Ethics of AI
* EU AI Act
* Precautionary principle
* Regulation of AI
* Virtual politician

Social and economic|

* AI boom
* AI bubble
* AI literacy
* AI slop
* AI veganism
* AI winter
* Anthropomorphism
* In architecture
* In education
* In healthcare
* Chatbot psychosis
* Mental health
* In visual art

* Category

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Double descent. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Double_descent&oldid=4230996"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template