Vision transformer

Short description: Machine learning model for vision processing

File:Vision Transformer.pngThe architecture of vision transformer. An input image is divided into patches, each of which is linearly mapped through a patch embedding layer, before entering a standard Transformer encoder.

A vision transformer (ViT) is a transformer designed for computer vision.[1] A ViT decomposes an input image into a series of patches (rather than text into tokens), serializes each patch into a vector, and maps it to a smaller dimension with a single matrix multiplication. These vector embeddings are then processed by a transformer encoder as if they were token embeddings.

ViTs were designed as alternatives to convolutional neural networks (CNNs) in computer vision applications. They have different inductive biases, training stability, and data efficiency.[2] Compared to CNNs, ViTs are less data efficient, but have higher capacity. Some of the largest modern computer vision models are ViTs, such as one with 22B parameters.[3][4]

Subsequent to its publication, many variants were proposed, with hybrid architectures with both features of ViTs and CNNs [5]. ViTs have found application in image recognition, image segmentation, weather prediction, and autonomous driving.[6][7]

## Contents

* 1 History
* 2 Overview
* 3 Variants
* 3.1 Original ViT
* 3.2 Architectural improvements
* 3.2.1 Pooling
* 3.3 Masked Autoencoder
* 3.4 DINO
* 3.5 Swin Transformer
* 3.6 TimeSformer
* 3.7 ViT-VQGAN
* 3.8 Others
* 4 Comparison with CNNs
* 5 Applications
* 6 See also
* 7 References
* 8 Further reading

## History

Transformers were introduced in Attention Is All You Need (2017),[8] and have found widespread use in natural language processing. A 2019 paper[9] applied ideas from the Transformer to computer vision. Specifically, they started with a ResNet, a standard convolutional neural network used for computer vision, and replaced all convolutional kernels by the self-attention mechanism found in a Transformer. It resulted in superior performance. However, it is not a Vision Transformer.

In 2020, an encoder-only Transformer was adapted for computer vision, yielding the ViT, which reached state of the art in image classification, overcoming the previous dominance of CNN.[1] The masked autoencoder (2022) extended ViT to work with unsupervised training. The vision transformer and the masked autoencoder, in turn, stimulated new developments in convolutional neural networks.[10][11]

Subsequently, there was cross-fertilization between the previous CNN approach and the ViT approach.

In 2021, some important variants of the Vision Transformers were proposed. These variants are mainly intended to be more efficient, more accurate or better suited to a specific domain. Two studies [12][13] improved efficiency and robustness of ViT by adding a CNN as a preprocessor. The Swin Transformer[14] achieved state-of-the-art results on some object detection datasets such as COCO, by using convolution-like sliding windows of attention mechanism, and the pyramid process in classical computer vision.

## Overview

File:Vision Transformer.svgVision Transformer architecture, showing the encoder-only Transformer blocks inside

The basic architecture, used by the original 2020 paper,[1] is as follows. In summary, it is a BERT-like encoder-only Transformer.

The input image is of type ℝH×W×C, where H,W,C are height, width, channel (RGB). It is then split into square-shaped patches of type ℝP×P×C.

For each patch, the patch is pushed through a linear operator, to obtain a vector ("patch embedding"). The position of the patch is also transformed into a vector by "position encoding". The two vectors are added, then pushed through several Transformer encoders.

The attention mechanism in a ViT repeatedly transforms representation vectors of image patches, incorporating more and more semantic relations between image patches in an image. This is analogous to how in natural language processing, as representation vectors flow through a transformer, they incorporate more and more semantic relations between words, from syntax to semantics.

The above architecture turns an image into a sequence of vector representations. To use these for downstream applications, an additional head needs to be trained to interpret them.

For example, to use it for classification, one can add a shallow MLP on top of it that outputs a probability distribution over classes. The original paper uses a linear-GeLU-linear-softmax network.[1]

## Variants

### Original ViT

The original ViT was an encoder-only Transformer supervise-trained to predict the image label from the patches of the image. As in the case of BERT, it uses a special token `<CLS>` in the input side, and the corresponding output vector is used as the only input of the final output MLP head. The special token is an architectural hack to allow the model to compress all information relevant for predicting the image label into one vector.

Animation of ViT. The 0th token is the special `<CLS>`. The other 9 patches are projected by a linear layer before being fed into the Transformer encoder as input tokens 1 to 9.

Transformers found their initial applications in natural language processing tasks, as demonstrated by language models such as BERT and GPT-3. By contrast the typical image processing system uses a convolutional neural network (CNN). Well-known projects include Xception, ResNet, EfficientNet,[15] DenseNet,[16] and Inception.[17]

Transformers measure the relationships between pairs of input tokens (words in the case of text strings), termed attention. The cost is quadratic in the number of tokens. For images, the basic unit of analysis is the pixel. However, computing relationships for every pixel pair in a typical image is prohibitive in terms of memory and computation. Instead, ViT computes relationships among pixels in various small sections of the image (e.g., 16x16 pixels), at a drastically reduced cost. The sections (with positional embeddings) are placed in a sequence. The embeddings are learnable vectors. Each section is arranged into a linear sequence and multiplied by the embedding matrix. The result, with the position embedding is fed to the transformer.[17]

### Architectural improvements

#### Pooling

Main article: Pooling layer

After the ViT processes an image, it produces some embedding vectors. These must be converted to a single class probability prediction by some kind of network. In the original ViT and Masked Autoencoder, they used a dummy `[CLS]` token , in emulation of the BERT language model. The output at `[CLS]` is the classification token, which is then processed by a LayerNorm-feedforward-softmax module into a probability distribution.

Global average pooling (GAP) does not use the dummy token, but simply takes the average of all output tokens as the classification token. It was mentioned in the original ViT as being equally good.[1]

Multihead attention pooling (MAP) applies a multiheaded attention block to pooling. Specifically, it takes as input a list of vectors x1,x2,…,xn, which might be thought of as the output vectors of a layer of a ViT. The output from MAP is MultiheadedAttention(Q,V,V), where q is a trainable query vector, and V is the matrix with rows being x1,x2,…,xn.[18] This was first proposed in the Set Transformer architecture.[19]

Later papers demonstrated that GAP and MAP both perform better than BERT-like pooling.[18][20] A variant of MAP was proposed as class attention, which applies MAP, then feedforward, then MAP again.[21]

Re-attention was proposed to allow training deep ViT. It changes the multiheaded attention module.[22]

### Masked Autoencoder

File:Masked Autoencoder.svgMasked Autoencoder architecture

The Masked Autoencoder[23] took inspiration from denoising autoencoders and context encoders.[24] It has two ViTs put end-to-end. The first one ("encoder") takes in image patches with positional encoding, and outputs vectors representing each patch. The second one (called "decoder", even though it is still an encoder-only Transformer) takes in vectors with positional encoding and outputs image patches again.

Training During training, input images (224px x 224 px in the original implementation) are split along a designated number of lines on each axis, producing image patches[25]. A certain percentage of patches are selected to be masked out by mask tokens, while all others are retained in the image. The network is tasked with reconstructing the image from the remaining unmasked patches. Mask tokens in the original implementation are learnable vector quantities[26]. A linear projection with positional embeddings is then applied to the vector of unmasked patches. Experiments varying mask ratio on networks trained on the ImageNet-1K dataset found 75% mask ratios[27] achieved high performance on both finetuning and linear-probing of the encoder's [Latent space|latent space]. The MAE processes only unmasked patches during training, increasing the efficiency of data processing in the encoder and lowering the memory usage of the [[Transformer (deep learning architecture) |transformer]][28].

A less computationally-intensive ViT is used for the decoder in the original implementation of the MAE. Masked patches are added back to the output of the encoder block as mask tokens and both are fed into the decoder. A reconstruction loss is computed for the masked patches to assess network performance.

Prediction In prediction, the decoder architecture is discarded entirely. The input image is split into patches by the same algorithm as in training, but no patches are masked out. A linear projection with a positional embedding is applied to each patch, and the resulting embedding vector representations of each patch are fed to the encoder.

Uses and Derivatives Several derivatives of the original MAE have been explored. The MAE has been applied for self-supervised pretraining in medical contexts, including for chest X-ray interpretation[29]. Derivatives of the MAE have been applied in this context to better serve as pretraining in medical contexts.

* Medically Supervised MAE[30]

Medically Supervised MAE seeks to address the application of MAE's high mask ratios when applied to medical lesion datasets and uses a [[Supervised learning|supervised training] set to create local attention maps for medical images in order to constrain which patches are masked out. Medically Supervised MAE achieved state-of-the-art performance as of Jan. 2025 on the classification of medical lesions on the Messidor-2, BTMD, HAM10000, DeepLesion, and ChestXRay2017 datasets[31]

* Gray Level Co-occurrence Matrix MAE (GLCM-MAE)[32]: GCLM-MAE uses GCLM to extract texture information from images in order to preserve texture information. It addresses an issue in which a classic MAE oversmooths images, causing a loss of granular detail that may be important in medical contexts. GLCM-MAE achieves state-of-the-art performance on the identification of gallbladder cancer, breast cancer imaged from ultrasound, pneumonia imaged from X-rays, and COVID-19 imaged from computed tomography as of Jul. 2025.

* Region-aware MAE[33] R-MAE: R-MAE replaces patch-generating step in the original MAE with an algorithm for assigning individual pixels to regions of interest in an image, which are masked out together. The region encoding architecture is standalone, but can be combined with the MAE for region reconstruction.

* Siamese MAEs (SiamMAE)[34]

SiamMAE is a network designed to apply MAEs to video data. Samples two frames from a video (compared to one in the original MAE), and labels them as "past" and "future." The network masks out a majority of the patches (~95%) in the future frame, leaves the past frame untouched, and passes both through the MAE encoder block. The decoder architecture is replaced with attention blocks that map patches from the past frame to the future frame for reconstruction. SiamMAE achieves competitive performance against larger models on segmentation and propagation in videos.

A similar architecture was BERT ViT (BEiT), published concurrently.[35]

### DINO

Like the Masked Autoencoder, the DINO (self-distillation with no labels) method is a way to train a ViT by self-supervision.[36] DINO is a form of teacher-student self-distillation. In DINO, the student is the model itself, and the teacher is an exponential average of the student's past states. The method is similar to previous works like momentum contrast[37] and bootstrap your own latent (BYOL).[38]

The loss function used in DINO is the cross-entropy loss between the output of the teacher network (fθ't) and the output of the student network (fθt). The teacher network is an exponentially decaying average of the student network's past parameters: θ't=αθt+α(1−α)θt−1+⋯. The inputs to the networks are two different crops of the same image, represented as T(x) and T′(x), where x is the original image. The loss function is written asL(fθ't(T(x)),fθt(T′(x)))One issue is that the network can "collapse" by always outputting the same value (y), regardless of the input. To prevent this collapse, DINO employs two strategies:

* Sharpening: The teacher network's output is sharpened using a softmax function with a lower temperature. This makes the teacher more "confident" in its predictions, forcing the student to learn more meaningful representations to match the teacher's sharpened output.
* Centering: The teacher network's output is centered by averaging it with its previous outputs. This prevents the teacher from becoming biased towards any particular output value, encouraging the student to learn a more diverse set of features.

In January 2024, Meta AI Research released an updated version called DINOv2[39] with improvements in architecture, loss function, and optimization technique. It was trained on a larger and more diverse dataset. The features learned by DINOv2 were more transferable, meaning it had better performance in downstream tasks.

In August 2025, Meta AI Research released DINOv3, an update to DINOv2. It introduced image-text alignment like CLIP. It scaled up the model to 7B parameters and the training dataset to 1.7B images (obtained by diversity-sampling an initial dataset with 17B images). Architecturally, it introduced two improvements: Gram anchoring and axial RoPE (Rotary Positional Embeddings) with jittering. Gram anchoring applies teacher-student self-distillation for the Gram matrix between the feature vectors of the patches of an image. It avoids the previously observed problem of degradation of dense feature maps: While performance on global tasks (like classification) continued to improve, performance on dense tasks (like segmentation) would peak early and then decline, with feature maps becoming noisy. Axial RoPE makes the model more robust to varying image resolutions, scales, and aspect ratios.[40][41]

### Swin Transformer

The Swin Transformer ("Shifted windows")[14] took inspiration from standard CNNs:

* Instead of performing self-attention over the entire sequence of tokens, one for each patch, it performs "shifted window based" self-attention, which means only performing attention over square-shaped blocks of patches. One block of patches is analogous to the receptive field of one convolution.
* After every few attention blocks, there is a "merge layer", which merges neighboring 2x2 tokens into a single token. This is analogous to pooling (by 2x2 convolution kernels, with stride 2). Merging means concatenation followed by multiplication with a matrix.

It is improved by Swin Transformer V2,[42] which modifies upon the ViT by a different attention mechanism[14]Template:Pg:

* LayerNorm immediately after each attention and feedforward layer ("res-post-norm");
* scaled cosine attention to replace the original dot product attention;
* log-spaced continuous relative position bias, which allows transfer learning across different window resolutions.

### TimeSformer

The TimeSformer[43] was designed for video understanding tasks, and it applied a factorized self-attention, similar to the factorized convolution kernels found in the Inception CNN architecture.[44] Schematically, it divides a video into frames, and each frame into a square grid of patches (same as ViT). Let each patch coordinate be denoted by x,y,t, denoting horizontal, vertical, and time.

* A space attention layer is a self-attention layer where each query patch qx,y,t attends to only the key and value patches kx′,y′,t′,vx′,y′,t′ such that t=t′.
* A time attention layer is where the requirement is x′=x,y′=y instead.

The TimeSformer also considered other attention layer designs, such as the "height attention layer" where the requirement is x′=x,t′=t. However, they found empirically that the best design interleaves one space attention layer and one time attention layer.

### ViT-VQGAN

In ViT-VQGAN,[45] there are two ViT encoders and a discriminator. One encodes 8x8 patches of an image into a list of vectors, one for each patch. The vectors can only come from a discrete set of "codebook", as in vector quantization. Another encodes the quantized vectors back to image patches. The training objective attempts to make the reconstruction image (the output image) faithful to the input image. The discriminator (usually a convolutional network, but other networks are allowed) attempts to decide if an image is an original real image, or a reconstructed image by the ViT.

The idea is essentially the same as vector quantized variational autoencoder (VQVAE) plus generative adversarial network (GAN).

After such a ViT-VQGAN is trained, it can be used to code an arbitrary image into a list of symbols, and code an arbitrary list of symbols into an image. The list of symbols can be used to train into a standard autoregressive transformer (like GPT), for autoregressively generating an image. Further, one can take a list of caption-image pairs, convert the images into strings of symbols, and train a standard GPT-style transformer. Then at test time, one can just give an image caption, and have it autoregressively generate the image. This is the structure of Google Parti.[46]

### Others

Other examples include the visual transformer,[47] CoAtNet,[48] CvT,[49] the data-efficient ViT (DeiT),[50] etc.

In the Transformer in Transformer architecture, each layer applies a vision Transformer layer on each image patch embedding, add back the resulting tokens to the embedding, then applies another vision Transformer layer.[51]

## Comparison with CNNs

Typically, ViT uses patch sizes larger than standard CNN kernels (3x3 to 7x7). ViT is more sensitive to the choice of the optimizer, hyperparameters, and network depth. Preprocessing with a layer of smaller-size, overlapping (stride < size) convolutional filters helps with performance and stability.[13]

This different behavior seems to derive from the different inductive biases they possess.

CNN applies the same set of filters for processing the entire image. This allows them to be more data efficient and less sensitive to local perturbations.[2] ViT applies self-attention, allowing them to easily capture long-range relationships between patches [52]. They also require more data to train, but they can ingest more training data compared to CNN, which might not improve after training on a large enough training dataset. ViT also appears more robust to input image distortions such as adversarial patches or permutations.[53]

## Applications

ViT have been used in many computer vision tasks with excellent results and in some cases even state-of-the-art, such as in image classification, object detection, video deepfake detection,[54] image segmentation,[55] anomaly detection, image synthesis, cluster analysis, autonomous driving.[6][7]

ViT had been used for image generation as backbones for GAN[56] and for diffusion models (diffusion transformer, or DiT).[57]

DINO[36] has been demonstrated to learn useful representations for clustering images and exploring morphological profiles on biological datasets, such as images generated with the Cell Painting assay.[58]

In 2024, a 113 billion-parameter ViT model was proposed (the largest ViT to date) for weather and climate prediction, and trained on the Frontier supercomputer with a throughput of 1.6 exaFLOPs.[59]

## See also

* Transformer (machine learning model)
* Convolutional neural network
* Attention (machine learning)
* Perceiver
* Deep learning
* PyTorch
* TensorFlow

## References

1. ↑ 1.0 1.1 1.2 1.3 1.4 Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob (2021-06-03). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale". arXiv:2010.11929 [cs.CV].
2. ↑ 2.0 2.1 Raghu, Maithra; Unterthiner, Thomas; Kornblith, Simon; Zhang, Chiyuan; Dosovitskiy, Alexey (2021-08-19). "Do Vision Transformers See Like Convolutional Neural Networks?". arXiv:2108.08810 [cs.CV].
3. ↑ Dehghani, Mostafa; Djolonga, Josip; Mustafa, Basil; Padlewski, Piotr; Heek, Jonathan; Gilmer, Justin; Steiner, Andreas; Caron, Mathilde et al. (2023-02-10), Scaling Vision Transformers to 22 Billion Parameters
4. ↑ "Scaling vision transformers to 22 billion parameters" (in en). http://research.google/blog/scaling-vision-transformers-to-22-billion-parameters/.
5. ↑ Koresh, Ella; Gross, Ronit D.; Meir, Yuval; Tzach, Yarden; Halevi, Tal; Kanter, Ido (2025). "Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi". Physica A: Statistical Mechanics and Its Applications 666. doi:10.1016/j.physa.2025.130529. ISSN 0378-4371. Bibcode: 2025PhyA..66630529K. https://www.sciencedirect.com/science/article/pii/S0378437125001815.
6. ↑ 6.0 6.1 Han, Kai; Wang, Yunhe; Chen, Hanting; Chen, Xinghao; Guo, Jianyuan; Liu, Zhenhua; Tang, Yehui; Xiao, An et al. (2023-01-01). "A Survey on Vision Transformer". IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (1): 87–110. doi:10.1109/TPAMI.2022.3152247. ISSN 0162-8828. PMID 35180075. Bibcode: 2023ITPAM..45...87H.
7. ↑ 7.0 7.1 Khan, Salman; Naseer, Muzammal; Hayat, Munawar; Zamir, Syed Waqas; Khan, Fahad Shahbaz; Shah, Mubarak (2022-09-13). "Transformers in Vision: A Survey". ACM Comput. Surv. 54 (10s): 200:1–200:41. doi:10.1145/3505244. ISSN 0360-0300.
8. ↑ Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All you Need". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 30. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
9. ↑ Ramachandran, Prajit; Parmar, Niki; Vaswani, Ashish; Bello, Irwan; Levskaya, Anselm; Shlens, Jon (2019). "Stand-Alone Self-Attention in Vision Models". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 32. https://proceedings.neurips.cc/paper/2019/hash/3416a75f4cea9109507cacd8e2f2aefc-Abstract.html.
10. ↑ Liu, Zhuang; Mao, Hanzi; Wu, Chao-Yuan; Feichtenhofer, Christoph; Darrell, Trevor; Xie, Saining (2022) (in en). A ConvNet for the 2020s. pp. 11976–11986. https://openaccess.thecvf.com/content/CVPR2022/html/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper.html.
11. ↑ Woo, Sanghyun; Debnath, Shoubhik; Hu, Ronghang; Chen, Xinlei; Liu, Zhuang; Kweon, In So; Xie, Saining (2023) (in en). ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders. pp. 16133–16142. https://openaccess.thecvf.com/content/CVPR2023/html/Woo_ConvNeXt_V2_Co-Designing_and_Scaling_ConvNets_With_Masked_Autoencoders_CVPR_2023_paper.html.
12. ↑ Wu, Bichen; Xu, Chenfeng; Dai, Xiaoliang; Wan, Alvin; Zhang, Peizhao; Yan, Zhicheng; Masayoshi, Tomizuka; Gonzalez, Joseph; Keutzer, Kurt; Vajda, Peter (2020). "Visual Transformers: Token-based Image Representation and Processing for Computer Vision". arXiv:2006.03677 [cs.CV].
13. ↑ 13.0 13.1 Xiao, Tete; Singh, Mannat; Mintun, Eric; Darrell, Trevor; Dollár, Piotr; Girshick, Ross (2021-06-28). "Early Convolutions Help Transformers See Better". arXiv:2106.14881 [cs.CV].
14. ↑ 14.0 14.1 14.2 Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021-03-25). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". arXiv:2103.14030 [cs.CV].
15. ↑ Tan, Mingxing; Le, Quoc (23 June 2021). "EfficientNetV2: Smaller Models and Faster Training". Proceedings of the 38th International Conference on Machine Learning (PMLR) 139: 10096–10106. https://proceedings.mlr.press/v139/tan21a/tan21a.pdf. Retrieved 31 October 2023.
16. ↑ Huang, Gao; Liu, Zhuang; van der Maaten, Laurens; Q. Weinberger, Kilian (28 Jan 2018). "Densely Connected Convolutional Networks". arXiv:1608.06993 [cs.CV].
17. ↑ 17.0 17.1 Sarkar, Arjun (2021-05-20). "Are Transformers better than CNN's at Image Recognition?" (in en). https://towardsdatascience.com/are-transformers-better-than-cnns-at-image-recognition-ced60ccc7c8.
18. ↑ 18.0 18.1 Zhai, Xiaohua; Kolesnikov, Alexander; Houlsby, Neil; Beyer, Lucas (June 2022). "Scaling Vision Transformers". 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 1204–1213. doi:10.1109/cvpr52688.2022.01179. ISBN 978-1-6654-6946-3. http://dx.doi.org/10.1109/cvpr52688.2022.01179.
19. ↑ Lee, Juho; Lee, Yoonho; Kim, Jungtaek; Kosiorek, Adam; Choi, Seungjin; Teh, Yee Whye (2019-05-24). "Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks" (in en). Proceedings of the 36th International Conference on Machine Learning (PMLR): 3744–3753. https://proceedings.mlr.press/v97/lee19d.html.
20. ↑ Karamcheti, Siddharth; Nair, Suraj; Chen, Annie S.; Kollar, Thomas; Finn, Chelsea; Sadigh, Dorsa; Liang, Percy (2023-02-24), Language-Driven Representation Learning for Robotics
21. ↑ Touvron, Hugo; Cord, Matthieu; Sablayrolles, Alexandre; Synnaeve, Gabriel; Jégou, Hervé (2021) (in en). Going Deeper With Image Transformers. pp. 32–42. https://openaccess.thecvf.com/content/ICCV2021/html/Touvron_Going_Deeper_With_Image_Transformers_ICCV_2021_paper.html.
22. ↑ Zhou, Daquan; Kang, Bingyi; Jin, Xiaojie; Yang, Linjie; Lian, Xiaochen; Jiang, Zihang; Hou, Qibin; Feng, Jiashi (2021-04-19), DeepViT: Towards Deeper Vision Transformer
23. ↑ He, Kaiming; Chen, Xinlei; Xie, Saining; Li, Yanghao; Dollár, Piotr; Girshick, Ross (2021). "Masked Autoencoders Are Scalable Vision Learners". arXiv:2111.06377 [cs.CV].
24. ↑ Pathak, Deepak; Krahenbuhl, Philipp; Donahue, Jeff; Darrell, Trevor; Efros, Alexei A. (June 2016). "Context Encoders: Feature Learning by Inpainting". 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 2536–2544. doi:10.1109/CVPR.2016.278. ISBN 978-1-4673-8851-1.
25. ↑ Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". arXiv:2111.06377 [cs.CV].
26. ↑ Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". arXiv:2111.06377 [cs.CV].
27. ↑ Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". arXiv:2111.06377 [cs.CV].
28. ↑ Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". arXiv:2111.06377 [cs.CV].
29. ↑ Zhou, Lei; Liu, Huidong; Bae, Joseph; He, Junjun; Samaras, Dimitris; Prasanna, Prateek (2022). "Self Pre-Training with Masked Autoencoders for Medical Image Classification and Segmentation". arXiv:2203.05573 [eess.IV].
30. ↑ Chen, Rui; Yang, Xiaotong; Li, Yue; Peng, Guocheng; Zhu, Qiuyue; Zhang, Zhenyu; Jiang, Hong (2024). "Deep learning-based model for automatic detection and grading of meibomian gland dysfunction from infrared images". Applied Soft Computing 164. doi:10.1016/j.asoc.2024.112905.
31. ↑ Chen, Rui; Yang, Xiaotong; Li, Yue; Peng, Guocheng; Zhu, Qiuyue; Zhang, Zhenyu; Jiang, Hong (2024). "Deep learning-based model for automatic detection and grading of meibomian gland dysfunction from infrared images". Applied Soft Computing 164.
32. ↑ Madan, Chetan; Satia, Aarjav; Basu, Soumen; Gupta, Pankaj; Dutta, Usha; Arora, Chetan (2025). "Focus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classification". arXiv:2507.10869 [eess.IV].
33. ↑ Nguyen, Duy Kien; Li, Yanghao; Aggarwal, Vaibhav; Oswald, Martin R.; Kirillov, Alexander; Snoek, Cees G. M.; Chen, Xinlei (2024). "R-MAE: Regions Meet Masked Autoencoders" (PDF). International Conference on Learning Representations. https://openreview.net/pdf?id=ba84RDHFnz.
34. ↑ Gupta, Agrim; Wu, Jiajun; Deng, Jia; Fei-Fei, Li (2023). "Siamese Masked Autoencoders". arXiv:2305.14344 [cs.CV].
35. ↑ Bao, Hangbo; Dong, Li; Piao, Songhao; Wei, Furu (2021-10-06). "BEiT: BERT Pre-Training of Image Transformers" (in en). International Conference on Learning Representations. https://openreview.net/forum?id=p-BhZSz59o4.
36. ↑ 36.0 36.1 Caron, Mathilde; Touvron, Hugo; Misra, Ishan; Jegou, Herve; Mairal, Julien; Bojanowski, Piotr; Joulin, Armand (October 2021). "Emerging Properties in Self-Supervised Vision Transformers". 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE. pp. 9630–9640. doi:10.1109/iccv48922.2021.00951. ISBN 978-1-6654-2812-5. http://dx.doi.org/10.1109/iccv48922.2021.00951.
37. ↑ He, Kaiming; Fan, Haoqi; Wu, Yuxin; Xie, Saining; Girshick, Ross (2020). Momentum Contrast for Unsupervised Visual Representation Learning. pp. 9729–9738. https://openaccess.thecvf.com/content_CVPR_2020/html/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.html.
38. ↑ Grill, Jean-Bastien; Strub, Florian; Altché, Florent; Tallec, Corentin; Richemond, Pierre; Buchatskaya, Elena; Doersch, Carl; Avila Pires, Bernardo et al. (2020). "Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 33: 21271–21284. https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html.
39. ↑ Oquab, Maxime; Darcet, Timothée; Moutakanni, Théo; Vo, Huy; Szafraniec, Marc; Khalidov, Vasil; Fernandez, Pierre; Haziza, Daniel; Massa, Francisco (2023-04-14). "DINOv2: Learning Robust Visual Features without Supervision". arXiv:2304.07193 [cs.CV].
40. ↑ "DINOv3: Self-supervised learning for vision at unprecedented scale" (in en). https://ai.meta.com/blog/dinov3-self-supervised-vision-model/.
41. ↑ Siméoni, Oriane; Vo, Huy V.; Seitzer, Maximilian; Baldassarre, Federico; Oquab, Maxime; Jose, Cijo; Khalidov, Vasil; Szafraniec, Marc; Yi, Seungeun; Ramamonjisoa, Michaël; Massa, Francisco; Haziza, Daniel; Wehrstedt, Luca; Wang, Jianyuan; Darcet, Timothée; Moutakanni, Théo; Sentana, Leonel; Roberts, Claire; Vedaldi, Andrea; Tolan, Jamie; Brandt, John; Couprie, Camille; Mairal, Julien; Jégou, Hervé; Labatut, Patrick; Bojanowski, Piotr (2025). "DINOv3". arXiv:2508.10104 [cs.CV].
42. ↑ Liu, Ze; Hu, Han; Lin, Yutong; Yao, Zhuliang; Xie, Zhenda; Wei, Yixuan; Ning, Jia; Cao, Yue et al. (2022). "Swin Transformer V2: Scaling Up Capacity and Resolution" (in en). Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12009–12019. https://openaccess.thecvf.com/content/CVPR2022/html/Liu_Swin_Transformer_V2_Scaling_Up_Capacity_and_Resolution_CVPR_2022_paper.html.
43. ↑ Bertasius, Gedas; Wang, Heng; Torresani, Lorenzo (2021-02-09). "Is Space-Time Attention All You Need for Video Understanding?". arXiv:2102.05095 [cs.CV].
44. ↑ Szegedy, Christian; Vanhoucke, Vincent; Ioffe, Sergey; Shlens, Jon; Wojna, Zbigniew (2016). Rethinking the Inception Architecture for Computer Vision. pp. 2818–2826. https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html.
45. ↑ Yu, Jiahui; Li, Xin; Koh, Jing Yu; Zhang, Han; Pang, Ruoming; Qin, James; Ku, Alexander; Xu, Yuanzhong; Baldridge, Jason; Wu, Yonghui (2021). "Vector-quantized Image Modeling with Improved VQGAN". arXiv:2110.04627 [cs.CV].
46. ↑ "Parti: Pathways Autoregressive Text-to-Image Model". https://sites.research.google/parti/.
47. ↑ Wu, Bichen; Xu, Chenfeng; Dai, Xiaoliang; Wan, Alvin; Zhang, Peizhao; Yan, Zhicheng; Tomizuka, Masayoshi; Gonzalez, Joseph et al. (2020-11-19), Visual Transformers: Token-based Image Representation and Processing for Computer Vision
48. ↑ Dai, Zihang; Liu, Hanxiao; Le, Quoc V.; Tan, Mingxing (2021-06-09). "CoAtNet: Marrying Convolution and Attention for All Data Sizes". arXiv:2106.04803 [cs.CV].
49. ↑ Wu, Haiping; Xiao, Bin; Codella, Noel; Liu, Mengchen; Dai, Xiyang; Yuan, Lu; Zhang, Lei (2021-03-29). "CvT: Introducing Convolutions to Vision Transformers". arXiv:2103.15808 [cs.CV].
50. ↑ Touvron, Hugo; Cord, Matthieu; Jégou, Hervé (2022). "DeiT III: Revenge of the ViT". in Avidan, Shai; Brostow, Gabriel; Cissé, Moustapha et al. (in en). Computer Vision – ECCV 2022. Lecture Notes in Computer Science. 13684. Cham: Springer Nature Switzerland. pp. 516–533. doi:10.1007/978-3-031-20053-3_30. ISBN 978-3-031-20053-3.
51. ↑ Han, Kai; Xiao, An; Wu, Enhua; Guo, Jianyuan; XU, Chunjing; Wang, Yunhe (2021). "Transformer in Transformer". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 34: 15908–15919. https://proceedings.neurips.cc/paper/2021/hash/854d9fca60b4bd07f9bb215d59ef5561-Abstract.html.
52. ↑ Gross, Ronit D.; Halevi, Tal; Koresh, Ella; Tzach, Yarden; Kanter, Ido (2025). "Low-latency vision transformers via large-scale multi-head attention". Physica A: Statistical Mechanics and Its Applications 675. doi:10.1016/j.physa.2025.130835. ISSN 0378-4371. Bibcode: 2025PhyA..67530835G. https://www.sciencedirect.com/science/article/pii/S037843712500487X.
53. ↑ Naseer, Muzammal; Ranasinghe, Kanchana; Khan, Salman; Hayat, Munawar; Khan, Fahad Shahbaz; Yang, Ming-Hsuan (2021-05-21). "Intriguing Properties of Vision Transformers". arXiv:2105.10497 [cs.CV].
54. ↑ Coccomini, Davide; Messina, Nicola; Gennaro, Claudio; Falchi, Fabrizio (2022). "Combining Efficient Net and Vision Transformers for Video Deepfake Detection" (in en). Image Analysis and Processing – ICIAP 2022. Lecture Notes in Computer Science. 13233. pp. 219–229. doi:10.1007/978-3-031-06433-3_19. ISBN 978-3-031-06432-6.
55. ↑ Kirillov, Alexander; Mintun, Eric; Ravi, Nikhila; Mao, Hanzi; Rolland, Chloe; Gustafson, Laura; Xiao, Tete; Whitehead, Spencer et al. (2023) (in en). Segment Anything. pp. 4015–4026. https://openaccess.thecvf.com/content/ICCV2023/html/Kirillov_Segment_Anything_ICCV_2023_paper.html.
56. ↑ Jiang, Yifan; Chang, Shiyu; Wang, Zhangyang (2021). "TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 34: 14745–14758. https://proceedings.neurips.cc/paper_files/paper/2021/hash/7c220a2091c26a7f5e9f1cfb099511e3-Abstract.html.
57. ↑ Peebles, William; Xie, Saining (March 2023). "Scalable Diffusion Models with Transformers". arXiv:2212.09748v2 [cs.CV].
58. ↑ Doron, Michael; Moutakanni, Théo; Chen, Zitong S.; Moshkov, Nikita; Caron, Mathilde; Touvron, Hugo; Bojanowski, Piotr; Pernice, Wolfgang M. et al. (2023-06-18). "Unbiased single-cell morphology with self-supervised vision transformers". BioRxiv: The Preprint Server for Biology. doi:10.1101/2023.06.16.545359. PMID 37398158.
59. ↑ Wang, Xiao; Liu, Siyan; Tsaris, Aristeidis; Choi, Jong-Youl; Aji, Ashwin; Fan, Ming; Zhang, Wei; Yin, Junqi; Ashfaq, Moetasim; Lu, Dan; Balaprakash, Prasanna (2024). "ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability". arXiv:2404.14712 [physics.ao-ph].

## Further reading

* Zhang, Aston; Lipton, Zachary; Li, Mu; Smola, Alexander J. (2024). "11.8. Transformers for Vision". Dive into deep learning. Cambridge New York Port Melbourne New Delhi Singapore: Cambridge University Press. ISBN 978-1-009-38943-3. https://d2l.ai/chapter_attention-mechanisms-and-transformers/vision-transformer.html.
* Steiner, Andreas; Kolesnikov, Alexander; Zhai, Xiaohua; Wightman, Ross; Uszkoreit, Jakob; Beyer, Lucas (June 18, 2021). "How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers". arXiv:2106.10270 [cs.CV].

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Glossary
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large
* NMT
* Reasoning
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Lethal autonomous weapons (LAWs)
* Generative artificial intelligence (GenAI)
* (Hypothetical: Artificial general intelligence (AGI))
* (Hypothetical: Artificial superintelligence (ASI))
* Agent2Agent protocol

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* GPT Image
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Seedance
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* 5.1
* 5.2
* Claude
* Gemini
* Gemini (language model)
* Gemma
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Christopher D. Manning
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Alex Krizhevsky
* Ilya Sutskever
* Oriol Vinyals
* Quoc V. Le
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* John Schulman
* Mustafa Suleyman
* Jan Leike
* Daniel Kokotajlo
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

Political|

* AI safety (Alignment)
* Ethics of AI
* EU AI Act
* Precautionary principle
* Regulation of AI
* Virtual politician

Social and economic|

* AI boom
* AI bubble
* AI literacy
* AI slop
* AI veganism
* AI winter
* Anthropomorphism
* In architecture
* In education
* In healthcare
* Chatbot psychosis
* Mental health
* In visual art

* Category

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Vision transformer. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Vision_transformer&oldid=4151687"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template