Double descent

Machine learning and data mining
Problems Classification Clustering Regression Anomaly detection AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean-shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA t-SNE
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection k-NN Local outlier factor
Artificial neural networks Autoencoder Deep learning DeepDream Multilayer perceptron RNN LSTM GRU Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net
Reinforcement learning Q-learning SARSA Temporal difference (TD)
Theory Bias–variance dilemma Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues NeurIPS ICML ML JMLR ArXiv:cs.LG
Glossary of artificial intelligence Glossary of artificial intelligence
Related articles List of datasets for machine-learning research Outline of machine learning
v t e

Double descent in statistics and machine learning is the phenomenon where a model with a small number of parameters and a model with an extremely large number of parameters both have a small training error, but a model whose number of parameters is about the same as the number of data points used to train the model will have a much greater test error than one with a much larger number of parameters.^[2] This phenomenon has been considered surprising, as it contradicts assumptions about overfitting in classical machine learning.^[3]

Early observations of what would later be called double descent in specific models date back to 1989.^[4][5]

The term "double descent" was coined by Belkin et. al.^[6] in 2019,^[3] when the phenomenon gained popularity as a broader concept exhibited by many models.^[7][8] The latter development was prompted by a perceived contradiction between the conventional wisdom that too many parameters in the model result in a significant overfitting error (an extrapolation of the bias–variance tradeoff),^[9] and the empirical observations in the 2010s that some modern machine learning techniques tend to perform better with larger models.^[6][10]

Double descent occurs in linear regression with isotropic Gaussian covariates and isotropic Gaussian noise.^[11]

A model of double descent at the thermodynamic limit has been analyzed using the replica trick, and the result has been confirmed numerically.^[12]

A number of works^[13][14] have suggested that double descent can be explained using the concept of effective dimension: While a network may have a large number of parameters, in practice only a subset of those parameters are relevant for generalization performance, as measured by the local Hessian curvature. This explanation is formalized through PAC-Bayes compression-based generalization bounds,^[15] which show that less complex models are expected to generalize better under a Solomonoff prior.

The scaling behavior of double descent has been found to follow a broken neural scaling law^[16] functional form.

• Grokking (machine learning)

• Mikhail Belkin; Daniel Hsu; Ji Xu (2020). "Two Models of Double Descent for Weak Features". SIAM Journal on Mathematics of Data Science 2 (4): 1167–1180. doi:10.1137/20M1336072. 
• Mount, John (3 April 2024). "The m = n Machine Learning Anomaly". https://win-vector.com/2024/04/03/the-m-n-machine-learning-anomaly/. 
• Preetum Nakkiran; Gal Kaplun; Yamini Bansal; Tristan Yang; Boaz Barak; Ilya Sutskever (29 December 2021). "Deep double descent: where bigger models and more data hurt". Theory and Experiment (IOP Publishing Ltd and SISSA Medialab srl) 2021 (12): 124003. doi:10.1088/1742-5468/ac3a74. Bibcode: 2021JSMTE2021l4003N. 
• Song Mei; Andrea Montanari (April 2022). "The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve". Communications on Pure and Applied Mathematics 75 (4): 667–766. doi:10.1002/cpa.22008. 
• Xiangyu Chang; Yingcong Li; Samet Oymak; Christos Thrampoulidis (2021). "Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks". Proceedings of the AAAI Conference on Artificial Intelligence 35 (8). 

• "Double Descent: Part 1: A Visual Introduction". https://mlu-explain.github.io/double-descent/. 
• "Double Descent: Part 2: A Mathematical Explanation". https://mlu-explain.github.io/double-descent2/. 
• Understanding "Deep Double Descent" at evhub.

Template:Artificial intelligence navbox

0.00 (0 votes) Original source: https://en.wikipedia.org/wiki/Double descent. Read more