Multimodal learning

Short description: Machine learning methods using multiple input modalities

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval,[1] text-to-image generation,[2] aesthetic ranking,[3] and image captioning.[4]

Multimodal learning was proposed in 2011 at the beginning of the deep learning period.[5] Large multimodal models, such as Google Gemini and GPT-4o, have become increasingly popular since 2023, enabling increased versatility and a broader understanding of real-world phenomena.[6]

## Contents

* 1 Motivation
* 2 Multimodal transformers
* 3 Multimodal deep Boltzmann machines
* 4 Applications
* 5 See also
* 6 References

## Motivation

Data usually comes with different modalities which carry different information. For example, it is very common to caption an image to convey the information not presented in the image itself. Similarly, sometimes it is more straightforward to use an image to describe information which may not be obvious from text. As a result, if different words appear in similar images, then these words likely describe the same thing. Conversely, if a word is used to describe seemingly dissimilar images, then these images may represent the same object. Thus, in cases dealing with multi-modal data, it is important to use a model which is able to jointly represent the information such that the model can capture the combined information from different modalities.

## Multimodal transformers

This section is an excerpt from Large language model § Multimodality[ edit ]

Multimodality means having multiple modalities, where a "modality" refers to a type of input or output, such as video, image, audio, text, proprioception, etc.[7] For example, Google PaLM model was fine-tuned into a multimodal model and applied to robotic control.[8] LLaMA models have also been turned multimodal using the tokenization method, to allow image inputs,[9] and video inputs.[10] GPT-4o can process and generate text, audio and images.[11] Such models are sometimes called large multimodal models (LMMs).[12]

A common method to create multimodal models out of an LLM is to "tokenize" the output of a trained encoder. Concretely, one can construct an LLM that can understand images as follows: take a trained LLM, and take a trained image encoder E. Make a small multilayered perceptron f, so that for any image y, the post-processed vector f(E(y)) has the same dimensions as an encoded token. That is an "image token". Then, one can interleave text tokens and image tokens. The compound model is then fine-tuned on an image-text dataset. This basic construction can be applied with more sophistication to improve the model. The image encoder may be frozen to improve stability.[13] This type of method, where embeddings from multiple modalities are fused and the predictor is trained on the combined embeddings, is called early fusion.

Another method, called intermediate fusion, involves each modality being first processed independently to obtain modality-specific representations; then these intermediate representations are fused together.[14] In general, cross-attention is used for integrating information from different modalities. As an example, the model Flamingo uses cross-attention layers to inject visual information into its pre-trained language model.[15]

This section is an excerpt from Transformer (machine learning model) § Multimodality[ edit ]

Models such as CLIP (Contrastive Language–Image Pretraining) learn joint representations of images and text by optimizing contrastive objectives, allowing the model to match images with their corresponding textual descriptions.[16]

## Multimodal deep Boltzmann machines

A Boltzmann machine is a type of stochastic neural network invented by Geoffrey Hinton and Terry Sejnowski in 1985. Boltzmann machines can be seen as the stochastic, generative counterpart of Hopfield nets. They are named after the Boltzmann distribution in statistical mechanics. The units in Boltzmann machines are divided into two groups: visible units and hidden units. Each unit is like a neuron with a binary output that represents whether it is activated or not.[17] General Boltzmann machines allow connection between any units. However, learning is impractical using general Boltzmann Machines because the computational time is exponential to the size of the machine . A more efficient architecture is called restricted Boltzmann machine where connection is only allowed between hidden unit and visible unit, which is described in the next section.

Multimodal deep Boltzmann machines can process and learn from different types of information, such as images and text, simultaneously. This can notably be done by having a separate deep Boltzmann machine for each modality, for example one for images and one for text, joined at an additional top hidden layer.[18]

## Applications

Multimodal machine learning has numerous applications across various domains:

* Cross-modal retrieval: cross-modal retrieval allows users to search for data across different modalities (e.g., retrieving images based on text descriptions), improving multimedia search engines and content recommendation systems.[19]
* Classification and missing data retrieval: multimodal Deep Boltzmann Machines outperform traditional models like support vector machines and latent Dirichlet allocation in classification tasks and can predict missing data in multimodal datasets, such as images and text.
* Healthcare diagnostics: multimodal models integrate medical imaging, genomic data, and patient records to improve diagnostic accuracy and early disease detection, especially in cancer screening.[20][21][22]
* Content generation: models like DALL·E generate images from textual descriptions, benefiting creative industries, while cross-modal retrieval enables dynamic multimedia searches.[23]
* Robotics and human-computer interaction: multimodal learning improves interaction in robotics and AI by integrating sensory inputs like speech, vision, and touch, aiding autonomous systems and human-computer interaction.
* Emotion recognition: combining visual, audio, and text data, multimodal systems enhance sentiment analysis and emotion recognition, applied in customer service, social media, and marketing.

## See also

* Cross-modal retrieval
* Vision-language model
* Hopfield network
* Markov random field
* Markov chain Monte Carlo

## References

1. ↑ Hendriksen, Mariya; Bleeker, Maurits; Vakulenko, Svitlana; van Noord, Nanne; Kuiper, Ernst; de Rijke, Maarten (2021). "Extending CLIP for Category-to-image Retrieval in E-commerce". arXiv:2112.11294 [cs.CV].
2. ↑ "Stable Diffusion Repository on GitHub". CompVis - Machine Vision and Learning Research Group, LMU Munich. 17 September 2022. https://github.com/CompVis/stable-diffusion.
3. ↑ LAION-AI/aesthetic-predictor, LAION AI, 2024-09-06, https://github.com/LAION-AI/aesthetic-predictor, retrieved 2024-09-08
4. ↑ Mokady, Ron; Hertz, Amir; Bermano, Amit H. (2021). "ClipCap: CLIP Prefix for Image Captioning". arXiv:2111.09734 [cs.CV].
5. ↑ Ngiam, Jiquan; Khosla, Aditya; Kim, Mingyu; Nam, Juhan; Lee, Honglak; Ng, Andrew Y. (2011-06-28). "Multimodal deep learning". Proceedings of the 28th International Conference on International Conference on Machine Learning. ICML'11 (Madison, WI, USA: Omnipress): 689–696. ISBN 978-1-4503-0619-5. https://dl.acm.org/doi/10.5555/3104482.3104569.
6. ↑ Zia, Tehseen (January 8, 2024). "Unveiling of Large Multimodal Models: Shaping the Landscape of Language Models in 2024". https://www.unite.ai/unveiling-of-large-multimodal-models-shaping-the-landscape-of-language-models-in-2024/.
7. ↑ Kiros, Ryan; Salakhutdinov, Ruslan; Zemel, Rich (2014-06-18). "Multimodal Neural Language Models". Proceedings of the 31st International Conference on Machine Learning (PMLR): 595–603. https://proceedings.mlr.press/v32/kiros14.html. Retrieved 2023-07-02.
8. ↑ Driess, Danny; Xia, Fei; Sajjadi, Mehdi S. M.; Lynch, Corey; Chowdhery, Aakanksha; Ichter, Brian; Wahid, Ayzaan; Tompson, Jonathan et al. (2023-03-01). "PaLM-E: An Embodied Multimodal Language Model". ICML 202: 8469–8488. https://dl.acm.org/doi/10.5555/3618408.3618748.
9. ↑ Liu, Haotian; Li, Chunyuan; Wu, Qingyang; Lee, Yong Jae (2023-04-01). "Visual Instruction Tuning". NeurIPS.
10. ↑ Zhang, Hang; Li, Xin; Bing, Lidong (2023-06-01). "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding". EMNLP.
11. ↑ "OpenAI says natively multimodal GPT-4o eats text, visuals, sound – and emits the same". The Register. 2024-05-13. https://www.theregister.com/2024/05/13/openai_gpt4o/.
12. ↑ Zia, Dr Tehseen (2024-01-08). "Unveiling of Large Multimodal Models: Shaping the Landscape of Language Models in 2024" (in en-US). https://www.unite.ai/unveiling-of-large-multimodal-models-shaping-the-landscape-of-language-models-in-2024/.
13. ↑ Li, Junnan; Li, Dongxu; Savarese, Silvio; Hoi, Steven (2023-01-01). "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models". ICML 202: 19730–19742. https://dl.acm.org/doi/10.5555/3618408.3619222.
14. ↑ Kumar, Puneet; Khokher, Vedanti; Gupta, Yukti; Raman, Balasubramanian (2021). Hybrid Fusion Based Approach for Multimodal Emotion Recognition with Insufficient Labeled Data. pp. 314–318. doi:10.1109/ICIP42928.2021.9506714. ISBN 978-1-6654-4115-5.
15. ↑ Alayrac, Jean-Baptiste; Donahue, Jeff; Luc, Pauline; Miech, Antoine; Barr, Iain; Hasson, Yana; Lenc, Karel; Mensch, Arthur et al. (2022-12-06). "Flamingo: a Visual Language Model for Few-Shot Learning". Advances in Neural Information Processing Systems 35: 23716–23736. https://proceedings.neurips.cc/paper_files/paper/2022/hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html. Retrieved 2023-07-02.
16. ↑ Radford, Alec; Kim, Jong Wook; Hallacy, Chris; Ramesh, Aditya; Goh, Gabriel; Agarwal, Sandhini; Sastry, Girish; Askell, Amanda; Mishkin, Pamela; Clark, Jack; Krueger, Gretchen; Sutskever, Ilya (2021). "Learning Transferable Visual Models From Natural Language Supervision". arXiv:2103.00020 [cs.CV].
17. ↑ Dey, Victor (2021-09-03). "Beginners Guide to Boltzmann Machine" (in en-US). https://analyticsindiamag.com/beginners-guide-to-boltzmann-machines/.
18. ↑ "Multimodal Learning with Deep Boltzmann Machine". 2014. http://www.jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf.
19. ↑ Hendriksen, Mariya; Vakulenko, Svitlana; Kuiper, Ernst; de Rijke, Maarten (2023). "Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study". arXiv:2301.05174 [cs.CV].
20. ↑ Quach, Katyanna. "Harvard boffins build multimodal AI system to predict cancer" (in en). The Register. https://www.theregister.com/2022/08/09/ai_cancer_multimodal/.
21. ↑ Chen, Richard J.; Lu, Ming Y.; Williamson, Drew F. K.; Chen, Tiffany Y.; Lipkova, Jana; Noor, Zahra; Shaban, Muhammad; Shady, Maha et al. (8 August 2022). "Pan-cancer integrative histology-genomic analysis via multimodal deep learning" (in English). Cancer Cell 40 (8): 865–878.e6. doi:10.1016/j.ccell.2022.07.004. ISSN 1535-6108. PMID 35944502.
* Teaching hospital press release: "New AI technology integrates multiple data types to predict cancer outcomes" (in en). Brigham and Women's Hospital via medicalxpress.com. https://medicalxpress.com/news/2022-08-ai-technology-multiple-cancer-outcomes.html.
22. ↑ Shi, Yuge; Siddharth, N.; Paige, Brooks; Torr, Philip HS (2019). "Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models". arXiv:1911.03393 [cs.LG].
23. ↑ Shi, Yuge; Siddharth, N.; Paige, Brooks; Torr, Philip HS (2019). "Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models". arXiv:1911.03393 [cs.LG].

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Multimodal learning. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Multimodal_learning&oldid=4297749"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template