Rectifier (neural networks)

Type of activation function

Part of a series on

Machine learning
and data mining

Paradigms

* Supervised learning
* Unsupervised learning
* Semi-supervised learning
* Self-supervised learning
* Reinforcement learning
* Meta-learning
* Online learning
* Batch learning
* Curriculum learning
* Rule-based learning
* Neuro-symbolic AI
* Neuromorphic engineering
* Quantum machine learning

Problems

* Classification
* Generative modeling
* Regression
* Clustering
* Dimensionality reduction
* Density estimation
* Anomaly detection
* Data cleaning
* AutoML
* Association rules
* Semantic analysis
* Structured prediction
* Feature engineering
* Feature learning
* Learning to rank
* Grammar induction
* Ontology learning
* Multimodal learning

Supervised learning
(classification • regression)

* Apprenticeship learning
* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Fuzzy
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* PGD
* t-SNE
* SDL

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* RANSAC
* k-NN
* Local outlier factor
* Isolation forest

Neural networks

* Autoencoder
* Deep learning
* Feedforward neural network
* Recurrent neural network
* LSTM
* GRU
* ESN
* reservoir computing
* Boltzmann machine
* Restricted
* GAN
* Diffusion model
* SOM
* Convolutional neural network
* U-Net
* LeNet
* AlexNet
* DeepDream
* Neural field
* Neural radiance field
* Physics-informed neural networks
* Transformer
* Vision
* Mamba
* Spiking neural network
* Memtransistor
* Electrochemical RAM (ECRAM)

Reinforcement learning

* Q-learning
* Policy gradient
* SARSA
* Temporal difference (TD)
* Multi-agent
* Self-play

Learning with humans

* Active learning
* Crowdsourcing
* Human-in-the-loop
* Mechanistic interpretability
* RLHF

Model diagnostics

* Coefficient of determination
* Confusion matrix
* Learning curve
* ROC curve

Mathematical foundations

* Kernel machines
* Bias–variance tradeoff
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory
* Topological deep learning

Journals and conferences

* AAAI
* ECML PKDD
* NeurIPS
* ICML
* ICLR
* IJCAI
* ML
* JMLR

Related articles

* Glossary of artificial intelligence
* List of datasets for machine-learning research
* List of datasets in computer vision and image processing
* Outline of machine learning

* v
* t
* e

Plot of the ReLU (blue) and GELU (green) functions near x = 0

In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function[1][2] is an activation function defined as the non-negative part of its argument, i.e., the ramp function:

ReLU ⁡ ( x ) = x + = max ( 0 , x ) = x + | x | 2 = { x if x > 0 , 0 x ≤ 0 {\displaystyle \operatorname {ReLU} (x)=x^{+}=\max(0,x)={\frac {x+|x|}{2}}={\begin{cases}x&{\text{if }}x>0,\\\0&x\leq 0\end{cases}}}

where  x {\displaystyle x} is the input to a neuron. This is analogous to half-wave rectification in electrical engineering.

ReLU is one of the most popular activation functions for artificial neural networks,[3] and finds application in computer vision[4] and speech recognition[5][6] using deep neural nets and computational neuroscience.[7][8]

## History

[edit]

The ReLU was first used by Alston Householder in 1941 as a mathematical abstraction of biological neural networks.[9]

Kunihiko Fukushima in 1969 used ReLU in the context of visual feature extraction in hierarchical neural networks.[10][11] Thirty years later, Hahnloser et al. argued that ReLU approximates the biological relationship between neural firing rates and input current, in addition to enabling recurrent neural network dynamics to stabilise under weaker criteria.[12][13]

Prior to 2010, most activation functions used were the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more numerically efficient[14] counterpart, the hyperbolic tangent. Around 2010, the use of ReLU became common again.

Jarrett et al. (2009) noted that rectification by either absolute or ReLU (which they called "positive part") was critical for object recognition in convolutional neural networks (CNNs), specifically because it allows average pooling without neighboring filter outputs cancelling each other out. They hypothesized that the use of sigmoid or tanh was responsible for poor performance in previous CNNs.[15]

Nair and Hinton (2010) made a theoretical argument that the softplus activation function should be used, in that the softplus function numerically approximates the sum of an exponential number of linear models that share parameters. They then proposed ReLU as a good approximation to it. Specifically, they began by considering a single binary neuron in a Boltzmann machine that takes  x {\displaystyle x} as input, and produces 1 as output with probability  σ ( x ) = 1 1 + e − x {\displaystyle \sigma (x)={\frac {1}{1+e^{-x}}}} . They then considered extending its range of output by making infinitely many copies of it  X 1 , X 2 , X 3 , … {\displaystyle X_{1},X_{2},X_{3},\dots } , that all take the same input, offset by an amount  0.5 , 1.5 , 2.5 , … {\displaystyle 0.5,1.5,2.5,\dots } , then their outputs are added together as  ∑ i = 1 ∞ X i {\displaystyle \sum _{i=1}^{\infty }X_{i}} . They then demonstrated that  ∑ i = 1 ∞ X i {\displaystyle \sum _{i=1}^{\infty }X_{i}} is approximately equal to  N ( log ⁡ ( 1 + e x ) , σ ( x ) ) {\displaystyle {\mathcal {N}}(\log(1+e^{x}),\sigma (x))} , which is also approximately equal to  ReLU ⁡ ( N ( x , σ ( x ) ) ) {\displaystyle \operatorname {ReLU} ({\mathcal {N}}(x,\sigma (x)))} , where  N {\displaystyle {\mathcal {N}}} stands for the gaussian distribution.

They also argued for another reason for using ReLU: that it allows "intensity equivariance" in image recognition. That is, multiplying input image by a constant  k {\displaystyle k} multiplies the output also. In contrast, this is false for other activation functions like sigmoid or tanh. They found that ReLU activation allowed good empirical performance in restricted Boltzmann machines.[16]

Glorot et al (2011) argued that ReLU has the following advantages over sigmoid or tanh:

* ReLU is more similar to biological neurons' responses in their main operating regime.
* ReLU avoids vanishing gradients.
* ReLU is cheaper to compute.
* ReLU creates sparse representation naturally, because many hidden units output exactly zero for a given input.

They also found empirically that deep networks trained with ReLU can achieve strong performance without unsupervised pre-training, especially on large, purely supervised tasks.[4]

## Advantages

[edit]

Advantages of ReLU include:

* Sparse activation: for example, in a randomly initialized network, only about 50% of hidden units are activated (i.e. have a non-zero output).
* Better gradient propagation: fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions.[4]
* Efficiency: only requires comparison and addition.
* Scale-invariant (homogeneous, or "intensity equivariance"[16]):

max ( 0 , a x ) = a max ( 0 , x ) for a ≥ 0 {\displaystyle \max(0,ax)=a\max(0,x){\text{ for }}a\geq 0} .

## Potential problems

[edit]

Possible downsides can include:

* Non-differentiability at zero (however, it is differentiable anywhere else, and the value of the derivative at zero can be chosen to be 0 or 1 arbitrarily).
* Not zero-centered: ReLU outputs are always non-negative. This can make it harder for the network to learn during backpropagation, because gradient updates tend to push weights in one direction (positive or negative). Batch normalization can help address this.[citation needed]
* ReLU is unbounded.
* Redundancy of the parametrization: Because ReLU is scale-invariant, the network computes the exact same function by scaling the weights and biases in front of a ReLU activation by  k {\displaystyle k} , and the weights after by  1 / k {\displaystyle 1/k} .[4]
* Dying ReLU: ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state (it "dies"). This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity and potentially even halting the learning process. This problem typically arises when the learning rate is set too high. It may be mitigated by using "leaky" ReLU instead, where a small positive slope is assigned for  x < 0 {\displaystyle x<0} . However, depending on the task, performance may be reduced.

## Variants

[edit]

### Piecewise-linear variants

[edit]

Leaky ReLU (2014) allows a small, positive gradient when the unit is inactive,[6] helping to mitigate the vanishing gradient problem. This gradient is defined by a parameter  α {\displaystyle \alpha } , typically set to 0.01–0.3.[17][18]

f ( x ) = { x x > 0 , α x x ≤ 0 , f ′ ( x ) = { 1 x > 0 , α x ≤ 0. {\displaystyle f(x)={\begin{cases}x&x>0,\\\\\alpha x&x\leq 0,\end{cases}}\qquad f'(x)={\begin{cases}1&x>0,\\\\\alpha &x\leq 0.\end{cases}}}

The same function can also be expressed without the piecewise notation as:

f ( x ) = 1 + α 2 x + 1 − α 2 | x | {\displaystyle f(x)={\frac {1+\alpha }{2}}x+{\frac {1-\alpha }{2}}|x|}

Parametric ReLU (PReLU, 2016) takes this idea further by making  α {\displaystyle \alpha } a learnable parameter along with the other network parameters.[19]

Note that for  α ≤ 1 {\displaystyle \alpha \leq 1} , this is equivalent to

f ( x ) = max ( x , α x ) {\displaystyle f(x)=\max(x,\alpha x)}

and thus has a relation to "maxout" networks.[19]

Concatenated ReLU (CReLU, 2016) preserves positive and negative phase information by returning two values:[20]

f ( x ) = [ ReLU ⁡ ( x ) , ReLU ⁡ ( − x ) ] . {\displaystyle f(x)=[\operatorname {ReLU} (x),\operatorname {ReLU} (-x)].}

### Smooth variants

[edit]

#### Softplus

[edit]

Main article: Softplus

Plot of the softplus function and the ramp function

A smooth approximation to the rectifier is the analytic function

f ( x ) = ln ⁡ ( 1 + e x ) , f ′ ( x ) = e x 1 + e x = 1 1 + e − x {\displaystyle f(x)=\ln(1+e^{x}),\qquad f'(x)={\frac {e^{x}}{1+e^{x}}}={\frac {1}{1+e^{-x}}}}

which is called the softplus (2000)[21][4] or SmoothReLU function.[22] For large negative  x {\displaystyle x} it is roughly  ln ⁡ 1 {\displaystyle \ln 1} , so just above 0, while for large positive  x {\displaystyle x} it is roughly  ln ⁡ ( e x ) {\displaystyle \ln(e^{x})} , so just above  x {\displaystyle x} .

This function can be approximated as:

ln ⁡ ( 1 + e x ) ≈ { ln ⁡ 2 , x = 0 , x 1 − e − x / ln ⁡ 2 , x ≠ 0 {\displaystyle \ln \left(1+e^{x}\right)\approx {\begin{cases}\ln 2,&x=0,\\\\[6pt]{\frac {x}{1-e^{-x/\ln 2}}},&x\neq 0\end{cases}}}

By making the change of variables  x = y ln ⁡ ( 2 ) {\displaystyle x=y\ln(2)} , this is equivalent to

log 2 ⁡ ( 1 + 2 y ) ≈ { 1 , y = 0 , y 1 − e − y , y ≠ 0 {\displaystyle \log _{2}(1+2^{y})\approx {\begin{cases}1,&y=0,\\\\[6pt]{\frac {y}{1-e^{-y}}},&y\neq 0\end{cases}}}

A sharpness parameter  k {\displaystyle k} may be included:

f ( x ) = ln ⁡ ( 1 + e k x ) k , f ′ ( x ) = e k x 1 + e k x = 1 1 + e − k x {\displaystyle f(x)={\frac {\ln(1+e^{kx})}{k}},\qquad f'(x)={\frac {e^{kx}}{1+e^{kx}}}={\frac {1}{1+e^{-kx}}}}

The derivative of softplus is the logistic function. This in turn can be viewed as a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

L S E 0 + ⁡ ( x 1 , … , x n ) := LSE ⁡ ( 0 , x 1 , … , x n ) = ln ⁡ ( 1 + e x 1 + ⋯ + e x n ) {\displaystyle \operatorname {LSE_{0}} ^{+}(x_{1},\dots ,x_{n}):=\operatorname {LSE} (0,x_{1},\dots ,x_{n})=\ln(1+e^{x_{1}}+\cdots +e^{x_{n}})}

The LogSumExp function is

LSE ⁡ ( x 1 , … , x n ) = ln ⁡ ( e x 1 + ⋯ + e x n ) {\displaystyle \operatorname {LSE} (x_{1},\dots ,x_{n})=\ln(e^{x_{1}}+\cdots +e^{x_{n}})}

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

#### ELU

[edit]

Exponential linear units (2015) smoothly allow negative values. This is an attempt to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.[23]

f ( x ) = { x x > 0 , α ( e x − 1 ) x ≤ 0 f ′ ( x ) = { 1 x > 0 , α e x x ≤ 0 {\displaystyle f(x)={\begin{cases}x&x>0,\\\\\alpha \left(e^{x}-1\right)&x\leq 0\end{cases}}\qquad f'(x)={\begin{cases}1&x>0,\\\\\alpha e^{x}&x\leq 0\end{cases}}}

In these formulas,  α {\displaystyle \alpha } is a hyperparameter to be tuned with the constraint  α ≥ 0 {\displaystyle \alpha \geq 0} .

Given the same interpretation of  α {\displaystyle \alpha } , ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form  f ( x ) = max ( − α , x ) {\displaystyle f(x)=\max(-\alpha ,x)} .

#### Gaussian-error linear unit (GELU)

[edit]

GELU (2016) is a smooth approximation to the rectifier:

f ( x ) = x Φ ( x ) , {\displaystyle f(x)=x\Phi (x),}

f ′ ( x ) = x Φ ′ ( x ) + Φ ( x ) {\displaystyle f'(x)=x\Phi '(x)+\Phi (x)}

where  Φ ( x ) = P ( X ⩽ x ) {\displaystyle \Phi (x)=P(X\leqslant x)} is the cumulative distribution function of the standard normal distribution.

This activation function is illustrated in the figure at the start of this article. It has a "bump" with negative derivative to the left of x < 0\. It serves as the default activation for many transformer models such as BERT.[24]

#### SiLU

[edit]

Main article: Swish function

The swish function

The SiLU (sigmoid linear unit) or swish function[25] is another smooth approximation which uses the sigmoid (logistic) function, first introduced in the 2016 GELU paper:[24]

f ( x ) = x ⋅ sigmoid ⁡ ( x ) , {\displaystyle f(x)=x\cdot \operatorname {sigmoid} (x),}

f ′ ( x ) = x ⋅ sigmoid ′ ⁡ ( x ) + sigmoid ⁡ ( x ) {\displaystyle f'(x)=x\cdot \operatorname {sigmoid} '(x)+\operatorname {sigmoid} (x)}

It is cheaper to calculate than GELU. It also has a "bump".

#### Mish

[edit]

The mish function (2019) can also be used as a smooth approximation of the rectifier.[25] It is defined as

f ( x ) = x tanh ⁡ ( softplus ⁡ ( x ) ) , {\displaystyle f(x)=x\tanh {\big (}\operatorname {softplus} (x){\big )},}

where  tanh ⁡ ( x ) {\displaystyle \tanh(x)} is the hyperbolic tangent, and  softplus ⁡ ( x ) {\displaystyle \operatorname {softplus} (x)} is the softplus function.

Mish was obtained by experimenting with functions similar to Swish (SiLU, see above). It is non-monotonic (has a "bump") like Swish. The main new feature is that it exhibits a "self-regularizing" behavior attributed to a term in its first derivative.[25][26]

#### Squareplus

[edit]

Squareplus (2021)[27] is the function

f ( x ) = x + x 2 + b 2 {\displaystyle f(x)={\frac {x+{\sqrt {x^{2}+b}}}{2}}}

where  b ≥ 0 {\displaystyle b\geq 0} is a hyperparameter that determines the "size" of the curved region near  x = 0 {\displaystyle x=0} . (For example, letting  b = 0 {\displaystyle b=0} yields ReLU, and letting  b = 4 {\displaystyle b=4} yields the metallic mean function.) Squareplus shares many properties with softplus: It is monotonic, strictly positive, approaches 0 as  x -> − ∞ {\displaystyle x\to -\infty } , approaches the identity as  x -> + ∞ {\displaystyle x\to +\infty } , and is  C ∞ {\displaystyle C^{\infty }} smooth. However, squareplus can be computed using only algebraic functions, making it well-suited for settings where computational resources or instruction sets are limited. Additionally, squareplus requires no special consideration to ensure numerical stability when  x {\displaystyle x} is large.

#### DELU

[edit]

ExtendeD Exponential Linear Unit (DELU, 2023) is an activation function which is smoother within the neighborhood of zero and sharper for bigger values, allowing better allocation of neurons in the learning process for higher performance. Thanks to its unique design, it has been shown that DELU may obtain higher classification accuracy than ReLU and ELU.[28]

f ( x ) = { x x > x c , ( e a x − 1 ) / b x ≤ x c f ′ ( x ) = { 1 x > x c , ( a / b ) e a x x ≤ x c {\displaystyle f(x)={\begin{cases}x&x>x_{c},\\\\(e^{ax}-1)/b&x\leq x_{c}\end{cases}}\qquad f'(x)={\begin{cases}1&x>x_{c},\\\\(a/b)e^{ax}&x\leq x_{c}\end{cases}}}

In these formulas,  a {\displaystyle a} ,  b {\displaystyle b} and  x c {\displaystyle x_{c}} are hyperparameter values which could be set as default constraints  a = 1 {\displaystyle a=1} ,  b = 2 {\displaystyle b=2} and  x c = 1.25643 {\displaystyle x_{c}=1.25643} , as done in the original work.

## See also

[edit]

* Softmax function
* Sigmoid function
* Tobit model
* Layer (deep learning)

## References

[edit]

1. ^ Brownlee, Jason (8 January 2019). "A Gentle Introduction to the Rectified Linear Unit (ReLU)". Machine Learning Mastery. Retrieved 8 April 2021.
2. ^ Liu, Danqing (30 November 2017). "A Practical Guide to ReLU". Medium. Retrieved 8 April 2021.
3. ^ Ramachandran, Prajit; Barret, Zoph; Quoc, V. Le (October 16, 2017). "Searching for Activation Functions". arXiv:1710.05941 [cs.NE].
4. ^ a b c d e Xavier Glorot; Antoine Bordes; Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. "Rectifier and softplus activation functions. The second one is a smooth version of the first."
5. ^ László Tóth (2013). Phone Recognition with Deep Sparse Rectifier Neural Networks (PDF). ICASSP.
6. ^ a b Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). Rectifier Nonlinearities Improve Neural Network Acoustic Models.
7. ^ Hansel, D.; van Vreeswijk, C. (2002). "How noise contributes to contrast invariance of orientation tuning in cat visual cortex". J. Neurosci. 22 (12): 5118–5128\. doi:10.1523/JNEUROSCI.22-12-05118.2002. PMC 6757721. PMID 12077207.
8. ^ Kadmon, Jonathan; Sompolinsky, Haim (2015-11-19). "Transition to Chaos in Random Neuronal Networks". Physical Review X. 5 (4): 041030. arXiv:1508.06486. Bibcode:2015PhRvX...5d1030K. doi:10.1103/PhysRevX.5.041030. S2CID 7813832.
9. ^ Householder, Alston S. (June 1941). "A theory of steady-state activity in nerve-fiber networks: I. Definitions and preliminary lemmas". The Bulletin of Mathematical Biophysics. 3 (2): 63–69\. doi:10.1007/BF02478220. ISSN 0007-4985.
10. ^ Fukushima, K. (1969). "Visual feature extraction by a multilayered network of analog threshold elements". IEEE Transactions on Systems Science and Cybernetics. 5 (4): 322–333\. doi:10.1109/TSSC.1969.300225.
11. ^ Fukushima, K.; Miyake, S. (1982). "Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition". Competition and Cooperation in Neural Nets. Lecture Notes in Biomathematics. Vol. 45\. Springer. pp. 267–285\. doi:10.1007/978-3-642-46466-9_18. ISBN 978-3-540-11574-8.
12. ^ Hahnloser, R.; Sarpeshkar, R.; Mahowald, M. A.; Douglas, R. J.; Seung, H. S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit". Nature. 405 (6789): 947–951\. Bibcode:2000Natur.405..947H. doi:10.1038/35016072. PMID 10879535. S2CID 4399014.
13. ^ Hahnloser, Richard; Seung, H. Sebastian (2000). "Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks". Advances in Neural Information Processing Systems. 13. MIT Press.
14. ^ Yann LeCun; Leon Bottou; Genevieve B. Orr; Klaus-Robert Müller (1998). "Efficient BackProp" (PDF). In G. Orr; K. Müller (eds.). Neural Networks: Tricks of the Trade. Springer.
15. ^ Jarrett, Kevin; Kavukcuoglu, Koray; Ranzato, Marc'Aurelio; LeCun, Yann (September 2009). "What is the best multi-stage architecture for object recognition?". 2009 IEEE 12th International Conference on Computer Vision. pp. 2146–2153\. doi:10.1109/ICCV.2009.5459469. ISBN 978-1-4244-4420-5.
16. ^ a b Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (ICML-10). 2010.
17. ^ "PyTorch Leaky ReLU docs".
18. ^ "TensorFlow Leaky ReLU docs".
19. ^ a b He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification". arXiv:1502.01852 [cs.CV].
20. ^ Shang, Wenling; Sohn, Kihyuk; Almeida, Diogo; Lee, Honglak (2016-06-11). "Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units". Proceedings of the 33rd International Conference on Machine Learning. PMLR: 2217–2225\. arXiv:1603.05201.
21. ^ Dugas, Charles; Bengio, Yoshua; Bélisle, François; Nadeau, Claude; Garcia, René (2000-01-01). "Incorporating second-order functional knowledge for better option pricing" (PDF). Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS'00). MIT Press: 451–457\. "Since the sigmoid h has a positive first derivative, its primitive, which we call softplus, is convex."
22. ^ "Smooth Rectifier Linear Unit (SmoothReLU) Forward Layer". Developer Guide for Intel Data Analytics Acceleration Library. 2017. Retrieved 2018-12-04.
23. ^ Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp (2015). "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)". arXiv:1511.07289 [cs.LG].
24. ^ a b Hendrycks, Dan; Gimpel, Kevin (2016). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415 [cs.LG].
25. ^ a b c Diganta Misra (23 Aug 2019), Mish: A Self Regularized Non-Monotonic Activation Function (PDF), arXiv:1908.08681v1, retrieved 26 March 2022.
26. ^ Shaw, Sweta (2020-05-10). "Activation Functions Compared with Experiments". W&B. Retrieved 2022-07-11.
27. ^ Barron, Jonathan T. (22 December 2021). "Squareplus: A Softplus-Like Algebraic Rectifier". arXiv:2112.11687 [cs.NE].
28. ^ Çatalbaş, Burak; Morgül, Ömer (16 August 2023). "Deep learning with ExtendeD Exponential Linear Unit (DELU)". Neural Computing and Applications. 35 (30): 22705–22724\. doi:10.1007/s00521-023-08932-z. Retrieved 20 April 2025.

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large language model
* NMT
* Reasoning language model
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Artificial general intelligence (AGI)

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* Claude
* Gemini
* chatbot
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Ilya Sutskever
* Alex Krizhevsky
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* Mustafa Suleyman
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

* Category

*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template