Multimodal learning

Short description: Machine learning methods using multiple input modalities

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

Multimodal learning, in the context of machine learning, is a type of deep learning using a combination of various modalities of data, often arising in real-world applications. An example of multi-modal data is data that combines text (typically represented as feature vector) with imaging data consisting of pixel intensities and annotation tags. As these modalities have fundamentally different statistical properties, combining them is non-trivial, which is why specialized modelling strategies and algorithms are required. The model is then trained to able to understand and work with multiple forms of data.

## Contents

* 1 Motivation
* 2 Background: Boltzmann machine
* 2.1 Restricted Boltzmann machine
* 2.2 Gaussian-Bernoulli RBM
* 2.3 Replicated Softmax Model
* 3 Deep Boltzmann machines
* 4 Multimodal deep Boltzmann machines
* 4.1 Inference and learning
* 5 Application
* 6 Multimodal transformers
* 7 See also
* 8 References

## Motivation

Many models and algorithms have been implemented to retrieve and classify certain types of data, e.g. image or text (where humans who interact with machines can extract images in form of pictures and texts that could be any message etc.). However, data usually come with different modalities (it is the degree to which a system's components may be separated or combined) which carry different information. For example, it is very common to caption an image to convey the information not presented in the image itself. Similarly, sometimes it is more straightforward to use an image to describe the information which may not be obvious from texts. As a result, if different words appear in similar images, then these words likely describe the same thing. Conversely, if a word is used to describe seemingly dissimilar images, then these images may represent the same object. Thus, in cases dealing with multi-modal data, it is important to use a model which is able to jointly represent the information such that the model can capture the correlation structure between different modalities. Moreover, it should also be able to recover missing modalities given observed ones (e.g. predicting possible image object according to text description). The Multimodal Deep Boltzmann Machine model satisfies the above purposes.

## Background: Boltzmann machine

A Boltzmann machine is a type of stochastic neural network invented by Geoffrey Hinton and Terry Sejnowski in 1985. Boltzmann machines can be seen as the stochastic, generative counterpart of Hopfield nets. They are named after the Boltzmann distribution in statistical mechanics. The units in Boltzmann machines are divided into two groups: visible units and hidden units. General Boltzmann machines allow connection between any units. However, learning is impractical using general Boltzmann Machines because the computational time is exponential to the size of the machine[citation needed]. A more efficient architecture is called restricted Boltzmann machine where connection is only allowed between hidden unit and visible unit, which is described in the next section.

### Restricted Boltzmann machine

A restricted Boltzmann machine[1] is an undirected graph model with stochastic visible variables and stochastic hidden variables. Each visible variable is connected to each hidden variable. The energy function of the model is defined as

E(𝐯,𝐡;θ)=−∑i=1D∑j=1FWijvihj−∑i=1Dbivi−∑j=1Fajhj

where θ={𝐯,𝐡;θ} are model parameters: Wij represents the symmetric interaction term between visible unit i and hidden unit j; bi and aj are bias terms. The joint distribution of the system is defined as

P(𝐯;θ)=1𝒵(θ)∑𝐡exp(−E(𝐯,𝐡;θ))

where 𝒵(θ) is a normalizing constant. The conditional distribution over hidden 𝐡 and 𝐯 can be derived as logistic function in terms of model parameters.

P(𝐡|𝐯;θ)=∏j=1Fp(hj|𝐯), with p(hj=1|𝐯)=g(∑i=1DWijvi+aj)
P(𝐯|𝐡;θ)=∏i=1Dp(vi|𝐡), with p(vi=1|𝐡)=g(∑j=1FWijhj+bi)

where g(x)=1(1+exp(−x)) is the logistic function.

The derivative of the log-likelihood with respect to the model parameters can be decomposed as the difference between the model's expectation and data-dependent expectation.

### Gaussian-Bernoulli RBM

Gaussian-Bernoulli RBMs[2] are a variant of restricted Boltzmann machine used for modeling real-valued vectors such as pixel intensities. It is usually used to model the image data. The energy of the system of the Gaussian-Bernoulli RBM is defined as

E(𝐯,𝐡;θ)=∑i=1D(vi−bi)22σi2−∑i=1D∑j=1FviσiWijvihj−∑i=1Dbivi−∑j=1Fajhj

where θ={𝐚,𝐛,𝐰,σ} are the model parameters. The joint distribution is defined the same as the one in restricted Boltzmann machine. The conditional distributions now become

P(𝐡|𝐯;θ)=∏j=1Fp(hj|𝐯), with p(hj=1|𝐯)=g(∑i=1DWijviσi+aj)
P(𝐯|𝐡;θ)=∏i=1Dp(vi|𝐡), with p(vi|𝐡)∼𝒩(σi∑j=1FWijhj+bi,σi2)

In Gaussian-Bernoulli RBM, the visible unit conditioned on hidden units is modeled as a Gaussian distribution.

### Replicated Softmax Model

The Replicated Softmax Model[3] is also a variant of restricted Boltzmann machine and commonly used to model word count vectors in a document. In a typical text mining problem, let K be the dictionary size, and M be the number of words in the document. Let 𝐕 be a M×K binary matrix with vik=1 only when the ith word in the document is the kth word in the dictionary. v^k denotes the count for the kth word in the dictionary. The energy of the state {𝐕,𝐡} for a document contains M words is defined as

E(𝐕,𝐡)=−∑j=1F∑k=1KWjkv^khj−∑k=1Kbkv^k−M∑j=1Fajhj

The conditional distributions are given by

p(hj=1|𝐕)=g(Maj+∑k=1Kv^kWjk)
p(vik=1|𝐡)=exp(bk+∑j=1FhjWjk∑q=1Kexp(bq+∑j=1FhjWjq)

## Deep Boltzmann machines

A deep Boltzmann machine[4] has a sequence of layers of hidden units. There are only connections between adjacent hidden layers, as well as between visible units and hidden units in the first hidden layer. The energy function of the system adds layer interaction terms to the energy function of general restricted Boltzmann machine and is defined by E(𝐯,𝐡;θ)=−∑i=1D∑j=1F1Wij(1)vihj(1)−∑j=1F1∑l=1F2Wjl(2)hj(1)hl(2)−∑l=1F2∑p=1F3Wlp(3)hl(2)hp(3)−∑i=1Dbivi−∑j=1F1bj(1)hj(1)−∑l=1F2bl(2)hl(2)−∑p=1F3bp(3)hp(3)

The joint distribution is

P(𝐯;θ)=1𝒵(θ)∑𝐡exp(−E(𝐯,𝐡(1),𝐡(2),𝐡(3);θ))

## Multimodal deep Boltzmann machines

Multimodal deep Boltzmann machine[5][6] uses an image-text bi-modal DBM where the image pathway is modeled as Gaussian-Bernoulli DBM and text pathway as Replicated Softmax DBM, and each DBM has two hidden layers and one visible layer. The two DBMs join together at an additional top hidden layer. The joint distribution over the multi-modal inputs defined as P(𝐯m,𝐯t;θ)=∑𝐡(2m),𝐡(2t),𝐡(3)P(𝐡(2m),𝐡(2t),𝐡(3))(∑𝐡(1m)P(𝐯m,𝐡(1m)|𝐡(2m)))(∑𝐡(1t)P(𝐯t,𝐡(1t)|𝐡(2t)))=1𝒵M(θ)∑𝐡exp(∑kjWkj(1t)vkthj(1t)+∑jlWjl(2t)hj(1t)hl(2t)+∑kbktvkt+M∑jbj(1t)hj(1t)+∑lbl(2t)hl(2t)−∑i(vim−bim)22σ2+∑ijvimσiWij(1m)hj(1m)+∑jlWjl(2m)hj(1m)hl(2m)+∑jbj(1m)hj(1m)+∑lbl(2m)hl(2m)+∑lpW(3t)hl(2t)hp(3)+∑lpW(3m)hl(2m)hp(3)+∑pbp(3)hp(3)

The conditional distributions over the visible and hidden units are

p(hj(1m)=1|𝐯m,𝐡(2m))=g(∑i=1DWij(1m)vimσi+∑l=1F2mWjl(2m)hl(2m)+bj(1m))
p(hl(2m)=1|𝐡(1m),𝐡(3))=g(∑j=1F1mWjl(2m)hj(1m)+∑p=1F3Wlp(3m)hp(3)+bl(2m))
p(hj(1t)=1|𝐯t,𝐡(2t))=g(∑k=1KWkl(1t)vk(t)+∑l=1F2tWjl(2t)hl(2t)+Mbj(1t))
p(hl(2t)=1|𝐡(1t),𝐡(3))=g(∑j=1F1tWjl(2t)hj(1t)+∑p=1F3Wlp(3t)hp(3)+bl(2t))
p(hp3)=1|𝐡(2))=g(∑l=1F2mWlp(3m)hl(2m)+∑l=1F2tWlp(3t)hl(2t)+bp(3))
p(vikt=1|𝐡(1t))=exp(∑j=1F1thj(1t)Wjk(1t)+bkt)∑q=1Kexp(∑j=1F1thj(1t)Wjq(1t)+bkt)
p(vim|𝐡(1m))∼𝒩(σi∑j=1F1mWij(1m)hj(1m)+bim,σi2)

### Inference and learning

Exact maximum likelihood learning in this model is intractable, but approximate learning of DBMs can be carried out by using a variational approach, where mean-field inference is used to estimate data-dependent expectations and an MCMC based stochastic approximation procedure is used to approximate the model’s expected sufficient statistics.[7]

## Application

Multimodal deep Boltzmann machines are successfully used in classification and missing data retrieval. The classification accuracy of multimodal deep Boltzmann machine outperforms support vector machines, latent Dirichlet allocation and deep belief network, when models are tested on data with both image-text modalities or with single modality.[citation needed] Multimodal deep Boltzmann machine is also able to predict missing modalities given the observed ones with reasonably good precision.[citation needed] Self Supervised Learning brings a more interesting and powerful model for multimodality. OpenAI developed CLIP and DALL-E models that revolutionized multimodality.

Multimodal deep learning is used for cancer screening – at least one system under development integrates such different types of data.[8][9]

## Multimodal transformers

This section is an excerpt from Transformer (machine learning model) § Multimodality[ edit ]

## See also

* Hopfield network
* Markov random field
* Markov chain Monte Carlo

## References

1. ↑ "Restricted Boltzmann Machine". 1986. https://www.cs.cmu.edu/~bhiksha/courses/deeplearning/Fall.2014/pdfs/Smolensky.1986.pdf.
2. ↑ "Gaussian-Bernoulli RBM". 1994. http://cseweb.ucsd.edu/~yfreund/papers/freund94unsupervised.pdf.
3. ↑ "Replicated Softmax Model". 2009a. http://papers.nips.cc/paper/3856-replicated-softmax-an-undirected-topic-model.pdf.
4. ↑ "Deep Boltzmann Machine". 2009b. http://www.utstat.toronto.edu/~rsalakhu/papers/dbm.pdf.
5. ↑ "Multimodal Learning with Deep Boltzmann Machine". 2012. http://papers.nips.cc/paper/4683-multimodal-learning-with-deep-boltzmann-machines.pdf.
6. ↑ "Multimodal Learning with Deep Boltzmann Machine". 2014. http://www.jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf.
7. ↑ "Approximations to the Likelihood Gradient". 2008. http://icml2008.cs.helsinki.fi/papers/638.pdf.
8. ↑ Quach, Katyanna. "Harvard boffins build multimodal AI system to predict cancer" (in en). The Register. https://www.theregister.com/2022/08/09/ai_cancer_multimodal/.
9. ↑ Chen, Richard J.; Lu, Ming Y.; Williamson, Drew F. K.; Chen, Tiffany Y.; Lipkova, Jana; Noor, Zahra; Shaban, Muhammad; Shady, Maha et al. (8 August 2022). "Pan-cancer integrative histology-genomic analysis via multimodal deep learning" (in English). Cancer Cell 40 (8): 865–878.e6. doi:10.1016/j.ccell.2022.07.004. ISSN 1535-6108. PMID 35944502.
* Teaching hospital press release: "New AI technology integrates multiple data types to predict cancer outcomes" (in en). Brigham and Women's Hospital via medicalxpress.com. https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html.

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Multimodal learning. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Multimodal_learning&oldid=214406"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template