Learning curve (machine learning)

This article provides insufficient context for those unfamiliar with the subject. Please help improve the article by providing more context for the reader. (March 2019) (Learn how and when to remove this template message)

|

Learning curve showing training score and cross validation score Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

In machine learning, a learning curve (or training curve) plots the optimal value of a model's loss function for a training set against this loss function evaluated on a validation data set with same parameters as produced the optimal function.[1] Synonyms include error curve, experience curve, improvement curve and generalization curve.[2]

More abstractly, the learning curve is a curve of (learning effort)-(predictive performance), where usually learning effort means number of training samples and predictive performance means accuracy on testing samples.[3]

The machine learning curve is useful for many purposes including comparing different algorithms,[4] choosing model parameters during design,[5] adjusting optimization to improve convergence, and determining the amount of data used for training.[6]

## Contents

* 1 Formal definition
* 1.1 Training curve for amount of data
* 1.2 Training curve for number of iterations
* 2 Choosing the size of the training dataset
* 3 See also
* 4 References

## Formal definition

One model of a machine learning is producing a function, f(x), which given some information, x, predicts some variable, y, from training data Xtrain and Ytrain. It is distinct from mathematical optimization because f should predict well for x outside of Xtrain.

We often constrain the possible functions to a parameterized family of functions, {fθ(x):θ∈Θ}, so that our function is more generalizable[7] or so that the function has certain properties such as those that make finding a good f easier, or because we have some a priori reason to think that these properties are true.[7]: 172

Given that it is not possible to produce a function that perfectly fits our data, it is then necessary to produce a loss function L(fθ(X),Y′) to measure how good our prediction is. We then define an optimization process which finds a θ which minimizes L(fθ(X,Y)) referred to as θ*(X,Y) .

### Training curve for amount of data

Then if our training data is {x1,x2,…,xn},{y1,y2,…yn} and our validation data is {x1′,x2′,…xm′},{y1′,y2′,…ym′} a learning curve is the plot of the two curves

1. i↦L(fθ*(Xi,Yi)(Xi),Yi)
2. i↦L(fθ*(Xi,Yi)(Xi′),Yi′)

where Xi={x1,x2,…xi}

### Training curve for number of iterations

Many optimization processes are iterative, repeating the same step until the process converges to an optimal value. Gradient descent is one such algorithm. If you define θi* as the approximation of the optimal θ after i steps, a learning curve is the plot of

1. i↦L(fθi*(X,Y)(X),Y)
2. i↦L(fθi*(X,Y)(X′),Y′)

## Choosing the size of the training dataset

It is a tool to find out how much a machine model benefits from adding more training data and whether the estimator suffers more from a variance error or a bias error. If both the validation score and the training score converge to a value that is too low with increasing size of the training set, it will not benefit much from more training data.[8]

In the machine learning domain, there are two implications of learning curves differing in the x-axis of the curves, with experience of the model graphed either as the number of training examples used for learning or the number of iterations used in training the model.[9]

## See also

* Overfitting
* Bias–variance tradeoff
* Model selection
* Cross-validation (statistics)
* Validity (statistics)
* Verification and validation
* Double descent

## References

1. ↑ "Mohr, Felix and van Rijn, Jan N. "Learning Curves for Decision Making in Supervised Machine Learning - A Survey." arXiv preprint arXiv:2201.12150 (2022).".
2. ↑ Viering, Tom; Loog, Marco (2023-06-01). "The Shape of Learning Curves: A Review". IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (6): 7799–7819. doi:10.1109/TPAMI.2022.3220744. ISSN 0162-8828. https://ieeexplore.ieee.org/document/9944190/.
3. ↑ Perlich, Claudia (2010), Sammut, Claude; Webb, Geoffrey I., eds., "Learning Curves in Machine Learning" (in en), Encyclopedia of Machine Learning (Boston, MA: Springer US): pp. 577–580, doi:10.1007/978-0-387-30164-8_452, ISBN 978-0-387-30164-8, https://doi.org/10.1007/978-0-387-30164-8_452, retrieved 2023-07-06
4. ↑ Madhavan, P.G. (1997). "A New Recurrent Neural Network Learning Algorithm for Time Series Prediction". Journal of Intelligent Systems. p. 113 Fig. 3. http://www.jininnovation.com/RecurrentNN_JIntlSys_PG.pdf.
5. ↑ "Machine Learning 102: Practical Advice". Tutorial: Machine Learning for Astronomy with Scikit-learn. https://astroml.github.com/sklearn_tutorial/practical.html#learning-curves.
6. ↑ Meek, Christopher; Thiesson, Bo; Heckerman, David (Summer 2002). "The Learning-Curve Sampling Method Applied to Model-Based Clustering". Journal of Machine Learning Research 2 (3): 397. http://connection.ebscohost.com/c/articles/7188676/learning-curve-sampling-method-applied-model-based-clustering.
7. ↑ 7.0 7.1 Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016-11-18) (in en). Deep Learning. MIT Press. pp. 108. ISBN 978-0-262-03561-3. https://books.google.com/books?id=Np9SDQAAQBAJ&q=deep learning goodfellow&pg=PA108.
8. ↑ scikit-learn developers. "Validation curves: plotting scores to evaluate models — scikit-learn 0.20.2 documentation". https://scikit-learn.org/stable/modules/learning_curve.html#learning-curve.
9. ↑ Sammut, Claude; Webb, Geoffrey I. (Eds.) (28 March 2011). Encyclopedia of Machine Learning (1st ed.). Springer. p. 578. ISBN 978-0-387-30768-8. https://books.google.com/books?id=i8hQhp1a62UC&q=neural+network+learning+curve&pg=PT604.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Learning curve (machine learning). Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Learning_curve_(machine_learning)&oldid=183939"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template