Deep belief network

Short description: Type of artificial neural network

Schematic overview of a deep belief net. Arrows represent directed connections in the graphical model that the net represents.

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

In machine learning, a deep belief network (DBN) is a generative graphical model, or alternatively a class of deep neural network, composed of multiple layers of latent variables ("hidden units"), with connections between the layers but not between units within each layer.[1]

When trained on a set of examples without supervision, a DBN can learn to probabilistically reconstruct its inputs. The layers then act as feature detectors.[1] After this learning step, a DBN can be further trained with supervision to perform classification.[2]

DBNs can be viewed as a composition of simple, unsupervised networks such as restricted Boltzmann machines (RBMs)[1] or autoencoders,[3] where each sub-network's hidden layer serves as the visible layer for the next. An RBM is an undirected, generative energy-based model with a "visible" input layer and a hidden layer and connections between but not within layers. This composition leads to a fast, layer-by-layer unsupervised training procedure, where contrastive divergence is applied to each sub-network in turn, starting from the "lowest" pair of layers (the lowest visible layer is a training set).

The observation[2] that DBNs can be trained greedily, one layer at a time, led to one of the first effective deep learning algorithms.[4]:6 Overall, there are many attractive implementations and uses of DBNs in real-life applications and scenarios (e.g., electroencephalography,[5] drug discovery[6][7][8]).

## Contents

* 1 Training
* 2 See also
* 3 References
* 4 External links

## Training

The training method for RBMs proposed by Geoffrey Hinton for use with training "Product of Experts" models is called contrastive divergence (CD).[9] CD provides an approximation to the maximum likelihood method that would ideally be applied for learning the weights.[10][11] In training a single RBM, weight updates are performed with gradient descent via the following equation: [math]\displaystyle{ w_{ij}(t+1) = w_{ij}(t) + \eta\frac{\partial \log(p(v))}{\partial w_{ij}} }[/math]

where, [math]\displaystyle{ p(v) }[/math] is the probability of a visible vector, which is given by [math]\displaystyle{ p(v) = \frac{1}{Z}\sum_he^{-E(v,h)} }[/math]. [math]\displaystyle{ Z }[/math] is the partition function (used for normalizing) and [math]\displaystyle{ E(v,h) }[/math] is the energy function assigned to the state of the network. A lower energy indicates the network is in a more "desirable" configuration. The gradient [math]\displaystyle{ \frac{\partial \log(p(v))}{\partial w_{ij}} }[/math] has the simple form [math]\displaystyle{ \langle v_ih_j\rangle_\text{data} - \langle v_ih_j\rangle_\text{model} }[/math] where [math]\displaystyle{ \langle\cdots\rangle_p }[/math] represent averages with respect to distribution [math]\displaystyle{ p }[/math]. The issue arises in sampling [math]\displaystyle{ \langle v_ih_j\rangle_\text{model} }[/math] because this requires extended alternating Gibbs sampling. CD replaces this step by running alternating Gibbs sampling for [math]\displaystyle{ n }[/math] steps (values of [math]\displaystyle{ n = 1 }[/math] perform well). After [math]\displaystyle{ n }[/math] steps, the data are sampled and that sample is used in place of [math]\displaystyle{ \langle v_ih_j\rangle_\text{model} }[/math]. The CD procedure works as follows:[10]

1. Initialize the visible units to a training vector.
2. Update the hidden units in parallel given the visible units: [math]\displaystyle{ p(h_j = 1 \mid \textbf{V}) = \sigma(b_j + \sum_i v_iw_{ij}) }[/math]. [math]\displaystyle{ \sigma }[/math] is the sigmoid function and [math]\displaystyle{ b_j }[/math] is the bias of [math]\displaystyle{ h_j }[/math].
3. Update the visible units in parallel given the hidden units: [math]\displaystyle{ p(v_i = 1 \mid \textbf{H}) = \sigma(a_i + \sum_j h_jw_{ij}) }[/math]. [math]\displaystyle{ a_i }[/math] is the bias of [math]\displaystyle{ v_i }[/math]. This is called the "reconstruction" step.
4. Re-update the hidden units in parallel given the reconstructed visible units using the same equation as in step 2.
5. Perform the weight update: [math]\displaystyle{ \Delta w_{ij} \propto \langle v_ih_j\rangle_\text{data} - \langle v_ih_j\rangle_\text{reconstruction} }[/math].

Once an RBM is trained, another RBM is "stacked" atop it, taking its input from the final trained layer. The new visible layer is initialized to a training vector, and values for the units in the already-trained layers are assigned using the current weights and biases. The new RBM is then trained with the procedure above. This whole process is repeated until the desired stopping criterion is met.[12]

Although the approximation of CD to maximum likelihood is crude (does not follow the gradient of any function), it is empirically effective.[10]

A restricted Boltzmann machine (RBM) with fully connected visible and hidden units. Note there are no hidden-hidden or visible-visible connections.

## See also

* Bayesian network
* Convolutional deep belief network
* Deep learning
* Energy based model
* Stacked Restricted Boltzmann Machine

## References

1. ↑ 1.0 1.1 1.2 "Deep belief networks". Scholarpedia 4 (5): 5947. 2009. doi:10.4249/scholarpedia.5947. Bibcode: 2009SchpJ...4.5947H.
2. ↑ 2.0 2.1 "A fast learning algorithm for deep belief nets". Neural Computation 18 (7): 1527–54. July 2006. doi:10.1162/neco.2006.18.7.1527. PMID 16764513. http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf.
3. ↑ Bengio, Yoshua; Lamblin, Pascal; Popovici, Dan; Larochelle, Hugh (2007). "Greedy Layer-Wise Training of Deep Networks". NIPS. http://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf.
4. ↑ Bengio, Y. (2009). "Learning Deep Architectures for AI". Foundations and Trends in Machine Learning 2: 1–127. doi:10.1561/2200000006. http://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf.
5. ↑ "Deep Belief Networks for Electroencephalography: A Review of Recent Contributions and Future Outlooks" (in en-US). IEEE Journal of Biomedical and Health Informatics 22 (3): 642–652. May 2018. doi:10.1109/jbhi.2017.2727218. PMID 28715343.
6. ↑ Ghasemi, Pérez-Sánchez; Mehri, Pérez-Garrido (2018). "Neural network and deep-learning algorithms used in QSAR studies: merits and drawbacks". Drug Discovery Today 23 (10): 1784–1790. doi:10.1016/j.drudis.2018.06.016. PMID 29936244.
7. ↑ Ghasemi, Pérez-Sánchez; Mehri, fassihi (2016). "The Role of Different Sampling Methods in Improving Biological Activity Prediction Using Deep Belief Network". Journal of Computational Chemistry 38 (10): 1–8. doi:10.1002/jcc.24671. PMID 27862046.
8. ↑ "Deep Learning in Drug Discovery". Molecular Informatics 35 (1): 3–14. January 2016. doi:10.1002/minf.201501008. PMID 27491648.
9. ↑ "Training Product of Experts by Minimizing Contrastive Divergence". Neural Computation 14 (8): 1771–1800. 2002. doi:10.1162/089976602760128018. PMID 12180402. http://www.cs.toronto.edu/~fritz/absps/nccd.pdf.
10. ↑ 10.0 10.1 10.2 "A Practical Guide to Training Restricted Boltzmann Machines". Tech. Rep. UTML TR 2010-003. 2010. https://www.researchgate.net/publication/221166159.
11. ↑ "Training Restricted Boltzmann Machines: An Introduction". Pattern Recognition 47 (1): 25–39. 2014. doi:10.1016/j.patcog.2013.05.025. Bibcode: 2014PatRe..47...25F. http://image.diku.dk/igel/paper/TRBMAI.pdf. Retrieved 2017-07-02.
12. ↑ Bengio, Yoshua (2009). "Learning Deep Architectures for AI". Foundations and Trends in Machine Learning 2 (1): 1–127. doi:10.1561/2200000006. http://sanghv.com/download/soft/machine learning, artificial intelligence, mathematics ebooks/ML/learning deep architectures for AI (2009).pdf. Retrieved 2017-07-02.

## External links

* "Deep Belief Networks". Deep Learning Tutorials. http://deeplearning.net/tutorial/DBN.html.
* "Deep Belief Network Example". Deeplearning4j Tutorials. http://deeplearning4j.org/deepbeliefnetwork.html.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Deep belief network. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Deep_belief_network&oldid=3383184"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template