Transformer (machine learning model)

Machine learning algorithm used for natural-language processing

This article relies excessively on references to primary sources. Please improve this article by adding secondary or tertiary sources.
Find sources: "Transformer" machine learning model – news · newspapers · books · scholar · JSTOR (February 2023) (Learn how and when to remove this template message)

|

"Transformer architecture" redirects here. For the design of electrical transformers, see Transformer.

Part of a series on

Machine learning
and data mining

Paradigms

* Supervised learning
* Unsupervised learning
* Online learning
* Batch learning
* Meta-learning
* Semi-supervised learning
* Self-supervised learning
* Reinforcement learning
* Rule-based learning
* Quantum machine learning

Problems

* Classification
* Generative model
* Regression
* Clustering
* dimension reduction
* density estimation
* Anomaly detection
* Data Cleaning
* AutoML
* Association rules
* Semantic analysis
* Structured prediction
* Feature engineering
* Feature learning
* Learning to rank
* Grammar induction
* Ontology learning
* Multimodal learning

Supervised learning
(classification • regression)

* Apprenticeship learning
* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Fuzzy
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* PGD
* t-SNE
* SDL

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* RANSAC
* k-NN
* Local outlier factor
* Isolation forest

Artificial neural network

* Autoencoder
* Cognitive computing
* Deep learning
* DeepDream
* Feedforward neural network
* Recurrent neural network
* LSTM
* GRU
* ESN
* reservoir computing
* Restricted Boltzmann machine
* GAN
* Diffusion model
* SOM
* Convolutional neural network
* U-Net
* Transformer
* Vision
* Spiking neural network
* Memtransistor
* Electrochemical RAM (ECRAM)

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)
* Multi-agent
* Self-play

Learning with humans

* Active learning
* Crowdsourcing
* Human-in-the-loop

Model diagnostics

* Learning curve

Mathematical foundations

* Kernel machines
* Bias–variance tradeoff
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* ECML PKDD
* NeurIPS
* ICML
* ICLR
* IJCAI
* ML
* JMLR

Related articles

* Glossary of artificial intelligence
* List of datasets for machine-learning research
* List of datasets in computer vision and image processing
* Outline of machine learning

* v
* t
* e

A transformer is a deep learning architecture, initially proposed in 2017, that relies on the parallel multi-head attention mechanism.[1] It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM),[2] and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.[3] Input text is split into n-grams encoded as tokens and each token is converted into a vector via looking up from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism allowing the signal for key tokens to be amplified and less important tokens to be diminished. Though the transformer paper was published in 2017, the softmax-based attention mechanism was proposed in 2014 for machine translation,[4][5] and the Fast Weight Controller, similar to a transformer, was proposed in 1992.[6][7][8]

This architecture is now used not only in natural language processing and computer vision,[9] but also in audio[10] and multi-modal processing. It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs)[11] and BERT[12] (Bidirectional Encoder Representations from Transformers).

Timeline of natural language processing models

## Timeline[edit]

* In 1990, the Elman network, using a recurrent neural network, encoded each word in a training set as a vector, called a word embedding, and the whole vocabulary as a vector database, allowing it to perform such tasks as sequence-predictions that are beyond the power of a simple multilayer perceptron. A shortcoming of the static embeddings was that they didn't differentiate between multiple meanings of same-spelt words.[13]
* In 1992, the Fast Weight Controller was published by Jürgen Schmidhuber.[6] It learns to answer queries by programming the attention weights of another neural network through outer products of key vectors and value vectors called FROM and TO. The Fast Weight Controller was later shown to be closely related to the Linear Transformer.[8][7][14] The terminology "learning internal spotlights of attention" was introduced in 1993.[15]

* In 1993, the IBM alignment models were used for statistical machine translation.[16]

* In 1997, a precursor of large language model, using recurrent neural networks, such as long short-term memory, was proposed.

* In 2001, a one-billion-word large text corpus, scraped from the Internet, referred to as "very very large" at the time, was used for word disambiguation.[17]

* In 2012, AlexNet demonstrated the effectiveness of large neural networks for image recognition, encouraging large artificial neural networks approach instead of older, statistical approaches.

* In 2014, a 380M-parameter seq2seq model for machine translation using two LSTMs networks was proposed by Sutskever at al.[18] The architecture consists of two parts. The encoder is an LSTM that takes in a sequence of tokens and turns it into a vector. The decoder is another LSTM that converts the vector into a sequence of tokens.

* In 2014, gating proved to be useful in a 130M-parameter seq2seq model, which used a simplified gated recurrent units (GRUs). Bahdanau et al[19] showed that GRUs are neither better nor worse than gated LSTMs.[20][21]

* In 2014, Bahdanau et al.[22] improved the previous seq2seq model by using an "additive" kind of attention mechanism in-between two LSTM networks. It was, however, not yet the parallelizable (scaled "dot product") kind of attention, later proposed in the 2017 transformer paper.
* In 2015, the relative performance of Global and Local (widowed) attention model architectures were assessed by Luong et al, a mixed attention architecture found to improve on the translations offered by Bahdanau's architecture, while the use of a local attention architecture reduced translation time. [23]

* In 2016, Google Translate gradually replaced the older statistical machine translation approach with the newer neural-networks-based approach that included a seq2seq model combined by LSTM and the "additive" kind of attention mechanism. They achieved a higher level of performance than the statistical approach, which took ten years to develop, in only nine months.[24][25]

* In 2017, the original (100M-sized) encoder-decoder transformer model with a faster (parallelizable or decomposable) attention mechanism was proposed in the "Attention is all you need" paper. As the model had difficulties converging, it was suggested that the learning rate should be linearly scaled up from 0 to maximal value for the first part of the training (i.e. 2% of the total number of training steps). The intent of the transformer model is to take a seq2seq model and remove its recurrent neural networks, but preserve its additive attention mechanism.[1]

* In 2018, in the ELMo paper, an entire sentence was processed before an embedding vector was assigning to each word in the sentence. A bi-directional LSTM was used to calculate such, deep contextualized embeddings for each word, improving upon the line of research from bag of words and word2vec.

* In 2018, an encoder-only transformer was used in the (more than 1B-sized) BERT model, improving upon ELMo.[26]

* In 2020, vision transformer[27] and speech-processing convolution-augmented transformer[28] outperformed recurrent neural networks, previously used for vision and speech.

* In 2020, difficulties with converging the original transformer were solved by normalizing layers before (instead of after) multiheaded attention by Xiong et al. This is called pre-LN Transformer.[29]

* In 2023, uni-directional ("autoregressive") transformers were being used in the (more than 100B-sized) GPT-3 and other OpenAI GPT models.[30][31]

### Predecessors[edit]

Before transformers, predecessors of attention mechanism were added to gated recurrent neural networks, such as LSTMs and gated recurrent units (GRUs), which processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. In 1992, fast weight controller was proposed as an alternative to recurrent neural networks that can learn "internal spotlights of attention".[15][6] In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.

The performance of old models was enhanced by adding an attention mechanism, which allowed a model to access any preceding point along the sequence. The attention layer weighs all previous states according to a learned measure of relevance, providing relevant information about far-away tokens. This proved to be especially useful in language translation, where far-away context can be essential for the meaning of a word in a sentence. The state vector has been accessible only after the last English word was processed while, for example, translating it from French by a LSTM model. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved. If an attention mechanism is added, the decoder is given access to the state vectors of every input word, not just the last, and can learn attention weights that dictate how much to attend to each input state vector. The augmentation of seq2seq models with the attention mechanism was first implemented in the context of machine translation by Bahdanau, Cho, and Bengio in 2014.[4][5]

### Decomposable attention[edit]

In 2016, highly parallelizable decomposable attention was successfully combined with a feedforward network.[32] This indicated that attention mechanisms were powerful in themselves and that sequential recurrent processing of data was not necessary to achieve the quality gains of recurrent neural networks with attention. In 2017, Vaswani et al. also proposed replacing recurrent neural networks with self-attention and started the effort to evaluate that idea.[1] Transformers, using an attention mechanism, processing all tokens simultaneously, calculated "soft" weights between them in successive layers. Since the attention mechanism only uses information about other tokens from lower layers, it can be computed for all tokens in parallel, which leads to improved training speed.

## Training[edit]

### Methods for stabilizing training[edit]

The plain transformer architecture had difficulty converging. In the original paper[1] the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of the training (usually recommended to be 2% of the total number of training steps), before decaying again.

A 2020 paper found that using layer normalization before (instead of after) multiheaded attention and feedforward layers stabilizes training, not requiring learning rate warmup.[29]

The GT3 model integrates CWTE, SWTE, and TTE using a self-adaptive gate layer, enabling efficient and effective fusion of three types of features for end-to-end text-driven stock market prediction.[33]

### Pretrain-finetune[edit]

Transformers typically undergo self-supervised learning involving unsupervised pretraining followed by supervised fine-tuning. Pretraining is typically done on a larger dataset than fine-tuning, due to the limited availability of labeled training data. Tasks for pretraining and fine-tuning commonly include:

* language modeling[12]
* next-sentence prediction[12]
* question answering[3]
* reading comprehension
* sentiment analysis[1]
* paraphrasing[1]

The T5 transformer paper[34] documents a large number of pretraining tasks. Some examples are:

* restoring corrupted text: `Thank you <X> me to your party <Y> week.` -> `<X> for inviting <Y> last <Z>` where the `<Z>` means "end of output".
* translation: `translate English to German: That is good."` -> `"Das ist gut."`.
* judging the grammatical acceptability of a sentence: `cola sentence: "The course is jumping well."` -> `not acceptable` .

## Applications[edit]

The transformer has had great success in natural language processing (NLP), for example the tasks of machine translation and time series prediction. Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. These may include:

* machine translation
* document summarization
* document generation
* named entity recognition (NER)[35]
* biological sequence analysis
* writing computer code based on requirements expressed in natural language.
* video understanding.

In addition to the NLP applications, it has also been successful in other fields, such as computer vision, or the protein folding applications (such as AlphaFold).

As an illustrative example, Ithaca is an encoder-only transformer with three output heads. It takes as input ancient Greek inscription as sequences of characters, but with illegible characters replaced with "-". Its three output heads respectively outputs probability distributions over Greek characters, location of inscription, and date of inscription.[36]

## Implementations[edit]

The transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch.

Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.[11]

## Architecture[edit]

An illustration of main components of the transformer model from the original paper, where layers were normalized after (instead of before) multiheaded attention.

All transformers have the same primary components:

* Tokenizers, which convert text into tokens.
* A single embedding layer, which converts tokens and positions of the tokens into vector representations.
* Transformer layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. These consist of alternating attention and feedforward layers.
* (optional) Un-embedding layer, which converts the final vector representations back to a probability distribution over the tokens.

Transformer layers can be one of two types, encoder and decoder. In the original paper both of them were used, while later models included only one type of them. BERT is an example of an encoder-only model; GPT are decoder-only models.

### Input[edit]

The input text is parsed into tokens by a tokenizer, most often a byte pair encoding tokenizer, and each token is converted into a vector via looking up from a word embedding table. Then, positional information of the token is added to the word embedding.

### Encoder-decoder architecture[edit]

Like earlier seq2seq models, the original transformer model used an encoder-decoder architecture. The encoder consists of encoding layers that process the input tokens iteratively one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output as well as the decoder output's tokens so far.

The function of each encoder layer is to generate contextualized token representations, where each representation corresponds to a token that "mixes" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e., the tokens generated so far during inference time).[37] [38]

Both the encoder and decoder layers have a feed-forward neural network for additional processing of the outputs and contain residual connections and layer normalization steps.[38]

### Scaled dot-product attention[edit]

The transformer building blocks are scaled dot-product attention units. For each attention unit, the transformer model learns three weight matrices: the query weights  W Q {\displaystyle W_{Q}} , the key weights  W K {\displaystyle W_{K}} , and the value weights  W V {\displaystyle W_{V}} . For each token  i {\displaystyle i} , the input token representation  x i {\displaystyle x_{i}} is multiplied with each of the three weight matrices to produce a query vector  q i = x i W Q {\displaystyle q_{i}=x_{i}W_{Q}} , a key vector  k i = x i W K {\displaystyle k_{i}=x_{i}W_{K}} , and a value vector  v i = x i W V {\displaystyle v_{i}=x_{i}W_{V}} . Attention weights are calculated using the query and key vectors: the attention weight  a i j {\displaystyle a_{ij}} from token  i {\displaystyle i} to token  j {\displaystyle j} is the dot product between  q i {\displaystyle q_{i}} and  k j {\displaystyle k_{j}} . The attention weights are divided by the square root of the dimension of the key vectors,  d k {\displaystyle {\sqrt {d_{k}}}} , which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that  W Q {\displaystyle W_{Q}} and  W K {\displaystyle W_{K}} are different matrices allows attention to be non-symmetric: if token  i {\displaystyle i} attends to token  j {\displaystyle j} (i.e.  q i ⋅ k j {\displaystyle q_{i}\cdot k_{j}} is large), this does not necessarily mean that token  j {\displaystyle j} will attend to token  i {\displaystyle i} (i.e.  q j ⋅ k i {\displaystyle q_{j}\cdot k_{i}} could be small). The output of the attention unit for token  i {\displaystyle i} is the weighted sum of the value vectors of all tokens, weighted by  a i j {\displaystyle a_{ij}} , the attention from token  i {\displaystyle i} to each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices  Q {\displaystyle Q} ,  K {\displaystyle K} and  V {\displaystyle V} are defined as the matrices where the  i {\displaystyle i} th rows are vectors  q i {\displaystyle q_{i}} ,  k i {\displaystyle k_{i}} , and  v i {\displaystyle v_{i}} respectively. Then we can represent the attention as

Attention ( Q , K , V ) = softmax ( Q K T d k ) V {\displaystyle {\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}}

where softmax is taken over the horizontal axis.

#### Multi-head attention[edit]

One set of  ( W Q , W K , W V ) {\displaystyle \left(W_{Q},W_{K},W_{V}\right)} matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". In addition, the influence field representing relevance can become progressively dilated in successive layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects.[39] The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feed-forward neural network layers.

Concretely, let the multiple attention heads be indexed by  i {\displaystyle i} , then we have

MultiheadedAttention ( Q , K , V ) = Concat i ∈ [ # h e a d s ] ( Attention ( X W i Q , X W i K , X W i V ) ) W O {\displaystyle {\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\\#heads]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}}

where the matrix  X {\displaystyle X} is the concatenation of word embeddings, and the matrices  W i Q , W i K , W i V {\displaystyle W_{i}^{Q},W_{i}^{K},W_{i}^{V}} are "projection matrices" owned by individual attention head  i {\displaystyle i} , and  W O {\displaystyle W^{O}} is a final projection matrix owned by the whole multi-headed attention head.

#### Masked attention[edit]

It may be necessary to cut out attention links between some word-pairs. For example, the decoder for token position  t {\displaystyle t} should not have access to token position  t + 1 {\displaystyle t+1} . This may be accomplished before the softmax stage by adding a mask matrix  M {\displaystyle M} that is  − ∞ {\displaystyle -\infty } at entries where the attention link must be cut, and  0 {\displaystyle 0} at other places:

MaskedAttention ( Q , K , V ) = softmax ( M + Q K T d k ) V {\displaystyle {\begin{aligned}{\text{MaskedAttention}}(Q,K,V)={\text{softmax}}\left(M+{\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}}

### Encoder[edit]

Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders.

The first encoder takes positional information and embeddings of the input sequence as its input, rather than encodings. The positional information is necessary for the transformer to make use of the order of the sequence, because no other part of the transformer makes use of this.[1]

The encoder is bidirectional. Attention can be placed on tokens before and after the current token. Tokens are used instead of words to account for polysemy.

A diagram of a sinusoidal positional encoding with parameters  N = 10000 , d = 100 {\displaystyle N=10000,d=100}

#### Positional encoding[edit]

A positional encoding is a fixed-size vector representation that encapsulates the relative positions of tokens within a target sequence: it provides the transformer model with information about where the words are in the input sequence.

The positional encoding is defined as a function of type  f : R -> R d ; d ∈ Z , d > 0 {\displaystyle f:\mathbb {R} \to \mathbb {R} ^{d};d\in \mathbb {Z} ,d>0} , where  d {\displaystyle d} is a positive even integer. The full positional encoding – as defined in the original paper – is given by the equation:

( f ( t ) 2 k , f ( t ) 2 k + 1 ) = ( sin ⁡ ( θ ) , cos ⁡ ( θ ) ) ∀ k ∈ { 0 , 1 , … , d / 2 − 1 } {\displaystyle (f(t)_{2k},f(t)_{2k+1})=(\sin(\theta ),\cos(\theta ))\quad \forall k\in \\{0,1,\ldots ,d/2-1\\}}

where  θ = t r k , r = N 2 / d {\displaystyle \theta ={\frac {t}{r^{k}}},r=N^{2/d}} .

Here,  N {\displaystyle N} is a free parameter that should be significantly larger than the biggest  k {\displaystyle k} that would be input into the positional encoding function. In the original paper,[1] the authors chose  N = 10000 {\displaystyle N=10000} .

The function is in a simpler form when written as a complex function of type  f : R -> C d / 2 {\displaystyle f:\mathbb {R} \to \mathbb {C} ^{d/2}}

f ( t ) = ( e i t / r k ) k = 0 , 1 , … , d 2 − 1 {\displaystyle f(t)=\left(e^{it/r^{k}}\right)_{k=0,1,\ldots ,{\frac {d}{2}}-1}}

where  r = N 2 / d {\displaystyle r=N^{2/d}} .

The main reason the authors chose this as the positional encoding function is that it allows one to perform shifts as linear transformations:

f ( t + Δ t ) = d i a g ( f ( Δ t ) ) f ( t ) {\displaystyle f(t+\Delta t)=\mathrm {diag} (f(\Delta t))f(t)}

where  Δ t ∈ R {\displaystyle \Delta t\in \mathbb {R} } is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations:

∑ j c j f ( t + Δ t j ) = ( ∑ j c j d i a g ( f ( Δ t j ) ) ) f ( t ) {\displaystyle \sum _{j}c_{j}f(t+\Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (f(\Delta t_{j}))\right)f(t)}

for any constants  c j {\displaystyle c_{j}} . This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position".

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

### Decoder[edit]

Each decoder consists of three major components: a self-attention mechanism, an attention mechanism over the encodings, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention.[1][38]

Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow.[1] This allows for autoregressive text generation. For all attention heads, attention can't be placed on following tokens. The last decoder is followed by a final linear transformation and softmax layer, to produce the output probabilities over the vocabulary.

All members of OpenAI's GPT series have a decoder-only architecture.

### Terminology[edit]

In large language model, the terminology is somewhat different than the terminology used in the original Transformer paper:[40]

* "encoder only": full encoder, full decoder.

* "encoder-decoder": full encoder, autoregressive decoder.
* "decoder only": autoregressive encoder, autoregressive decoder.

Here "autoregressive" means that a mask is inserted in the attention head to zero out all attention from one token to all tokens following it, as described in the "masked attention" section.

Generally, Transformer-based language models are of two types: causal (or "autoregressive") and masked. The GPT series is causal and decoder only. BERT is masked and encoder only.[41][42] The T5 series is encoder-decoder, with a full encoder and autoregressive decoder.[34]

## Subsequent work[edit]

### Alternative functions[edit]

The original transformer uses ReLU activation function. Other activation functions were developed, such as SwiGLU.[43]

### Alternative positional encodings[edit]

Transformers may use other positional encoding methods than sinusoidal.[44]

RoPE (rotary positional embedding),[45] is best explained by considering a list of 2-dimensional vectors  [ ( x 1 ( 1 ) , x 1 ( 2 ) ) , ( x 2 ( 1 ) , x 2 ( 2 ) ) , ( x 3 ( 1 ) , x 3 ( 2 ) ) , . . . ] {\displaystyle [(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),...]} . Now pick some angle  θ {\displaystyle \theta } . Then RoPE encoding is

RoPE ( x m ( 1 ) , x m ( 2 ) , m ) = ( cos ⁡ m θ − sin ⁡ m θ sin ⁡ m θ cos ⁡ m θ ) ( x m ( 1 ) x m ( 2 ) ) = ( x m ( 1 ) cos ⁡ m θ − x m ( 2 ) sin ⁡ m θ x m ( 2 ) cos ⁡ m θ + x m ( 1 ) sin ⁡ m θ ) {\displaystyle {\text{RoPE}}{\big (}x_{m}^{(1)},x_{m}^{(2)},m{\big )}={\begin{pmatrix}\cos m\theta &-\sin m\theta \\\\\sin m\theta &\cos m\theta \end{pmatrix}}{\begin{pmatrix}x_{m}^{(1)}\\\x_{m}^{(2)}\\\\\end{pmatrix}}={\begin{pmatrix}x_{m}^{(1)}\cos m\theta -x_{m}^{(2)}\sin m\theta \\\x_{m}^{(2)}\cos m\theta +x_{m}^{(1)}\sin m\theta \\\\\end{pmatrix}}}

Equivalently, if we write the 2-dimensional vectors as complex numbers  z m := x m ( 1 ) + i x m ( 2 ) {\displaystyle z_{m}:=x_{m}^{(1)}+ix_{m}^{(2)}} , then RoPE encoding is just multiplication by an angle:

RoPE ( z m , m ) = e i m θ z m {\displaystyle {\text{RoPE}}{\big (}z_{m},m{\big )}=e^{im\theta }z_{m}}

For a list of  2 n {\displaystyle 2n} -dimensional vectors, a RoPE encoder is defined by a sequence of angles  θ ( 1 ) , . . . , θ ( n ) {\displaystyle \theta ^{(1)},...,\theta ^{(n)}} . Then the RoPE encoding is applied to each pair of coordinates.

The benefit of RoPE is that the dot-product between two vectors depends on their relative location only:

RoPE ( x , m ) T RoPE ( y , n ) = RoPE ( x , m + k ) T RoPE ( y , n + k ) {\displaystyle {\text{RoPE}}{\big (}x,m{\big )}^{T}{\text{RoPE}}{\big (}y,n{\big )}={\text{RoPE}}{\big (}x,m+k{\big )}^{T}{\text{RoPE}}{\big (}y,n+k{\big )}}

for any integer  k {\displaystyle k} .

ALiBi (Attention with Linear Biases)[46] is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is

Attention ( Q , K , V ) = softmax ( Q K T d k + s B ) V {\displaystyle {\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}+sB\right)V\end{aligned}}}

Here,  s {\displaystyle s} is a real number ("scalar"), and  B {\displaystyle B} is the linear bias matrix defined by

B = ( 0 1 2 3 ⋯ − 1 0 1 2 ⋯ − 2 − 1 0 1 ⋯ − 3 − 2 − 1 0 ⋯ ⋮ ⋮ ⋮ ⋮ ⋱ ) {\displaystyle B={\begin{pmatrix}0&1&2&3&\cdots \\\\-1&0&1&2&\cdots \\\\-2&-1&0&1&\cdots \\\\-3&-2&-1&0&\cdots \\\\\vdots &\vdots &\vdots &\vdots &\ddots \\\\\end{pmatrix}}}

in other words,  B i , j = j − i {\displaystyle B_{i,j}=j-i} .

ALiBi allows pretraining on short context windows, then finetuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

Relative Position Encodings[47] is similar to ALiBi, but more generic:

Attention ( Q , K , V ) = softmax ( Q K T d k + B ) V {\displaystyle {\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}+B\right)V\end{aligned}}}

where  B {\displaystyle B} is a Toeplitz matrix, that is,  B i , j = B i ′ , j ′ {\displaystyle B_{i,j}=B_{i',j'}} whenever  i − j = i ′ − j ′ {\displaystyle i-j=i'-j'} .

### Efficient implementation[edit]

#### FlashAttention[edit]

FlashAttention[48] is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow).

An improved version, FlashAttention-2,[49][50][51] was developed to cater to the rising demand for language models capable of handling longer context lengths. It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.

Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).

Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.

#### Multi-Query Attention[edit]

Multi-Query Attention changes the multiheaded attention mechanism.[52] Whereas normally,

MultiheadedAttention ( Q , K , V ) = Concat i ∈ [ # h e a d s ] ( Attention ( X W i Q , X W i K , X W i V ) ) W O {\displaystyle {\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\\#heads]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}}

with Multi-Query Attention, there is just one  W K , W V {\displaystyle W^{K},W^{V}} , thus:

MultiQueryAttention ( Q , K , V ) = Concat i ∈ [ # h e a d s ] ( Attention ( X W i Q , X W K , X W V ) ) W O {\displaystyle {\text{MultiQueryAttention}}(Q,K,V)={\text{Concat}}_{i\in [\\#heads]}({\text{Attention}}(XW_{i}^{Q},XW^{K},XW^{V}))W^{O}}

This has a neutral effect on model quality and training speed, but increases inference speed.

### Sub-quadratic transformers[edit]

Training transformer-based architectures can be expensive, especially for long inputs.[53] Alternative architectures include the Reformer (which reduces the computational load from  O ( N 2 ) {\displaystyle O(N^{2})} to  O ( N ln ⁡ N ) {\displaystyle O(N\ln N)} [53]), or models like ETC/BigBird (which can reduce it to  O ( N ) {\displaystyle O(N)} )[54] where  N {\displaystyle N} is the length of the sequence. This is done using locality-sensitive hashing and reversible layers.[55][56]

Ordinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers[57] reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.

Long Range Arena (2020)[58] is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

Random Feature Attention (2021)[59] uses Fourier random features:

φ ( x ) = 1 D [ cos ⁡ ⟨ w 1 , x ⟩ , sin ⁡ ⟨ w 1 , x ⟩ , ⋯ cos ⁡ ⟨ w D , x ⟩ , sin ⁡ ⟨ w D , x ⟩ ] T {\displaystyle \varphi (x)={\frac {1}{\sqrt {D}}}[\cos \langle w_{1},x\rangle ,\sin \langle w_{1},x\rangle ,\cdots \cos \langle w_{D},x\rangle ,\sin \langle w_{D},x\rangle ]^{T}}

where  w 1 , . . . , w D {\displaystyle w_{1},...,w_{D}} are independent samples from the normal distribution  N ( 0 , σ 2 I ) {\displaystyle N(0,\sigma ^{2}I)} . This choice of parameters satisfy  E [ ⟨ φ ( x ) , φ ( y ) ⟩ ] = e ‖ x − y ‖ 2 2 σ 2 {\displaystyle \mathbb {E} [\langle \varphi (x),\varphi (y)\rangle ]=e^{\frac {\|x-y\|^{2}}{2\sigma ^{2}}}} , or

e ⟨ x , y ⟩ / σ 2 = E [ ⟨ e ‖ x ‖ 2 / 2 σ 2 φ ( x ) , e ‖ y ‖ 2 / 2 σ 2 φ ( y ) ⟩ ] ≈ ⟨ e ‖ x ‖ 2 / 2 σ 2 φ ( x ) , e ‖ y ‖ 2 / 2 σ 2 φ ( y ) ⟩ {\displaystyle e^{\langle x,y\rangle /\sigma ^{2}}=\mathbb {E} [\langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle ]\approx \langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle }

Consequently, the one-headed attention, with one query, can be written as

Attention ( q , K , V ) = softmax ( q K T d k ) V ≈ φ ( q ) T ∑ i e ‖ k i ‖ 2 / 2 σ 2 φ ( k i ) v i T φ ( q ) T ∑ i e ‖ k i ‖ 2 / 2 σ 2 φ ( k i ) {\displaystyle {\text{Attention}}(q,K,V)={\text{softmax}}\left({\frac {qK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx {\frac {\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})v_{i}^{T}}{\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})}}}

where  σ = d K 1 / 4 {\displaystyle \sigma =d_{K}^{1/4}} . Similarly for multiple queries, and for multiheaded attention.

This approximation can be computed in linear time, as we can compute the matrix  φ ( k i ) v i T {\displaystyle \varphi (k_{i})v_{i}^{T}} first, then multiply it with the query. In essence, we have managed to obtain a more precise version of

Attention ( Q , K , V ) = softmax ( Q K T d k ) V ≈ Q ( K T V / d k ) {\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx Q(K^{T}V/{\sqrt {d_{k}}})}

Performer (2022)[60] uses the same Random Feature Attention, but  w 1 , . . . , w D {\displaystyle w_{1},...,w_{D}} are first independently sampled from the normal distribution  N ( 0 , σ 2 I ) {\displaystyle N(0,\sigma ^{2}I)} , then they are Gram-Schmidt processed.

### Multimodality[edit]

Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.

Vision transformers[27] adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like tokens in a standard transformer.

Conformer[28] and later Whisper[61] follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i.e. broken down into a series of patches, turned into vectors and treated like tokens in a standard transformer.

Perceivers by Andrew Jaegle et al. (2021)[62][63] can learn from large amounts of heterogeneous data.

Regarding image outputs, Peebles et al introduced a diffusion transformer (DiT) which facilitates use of the transformer architecture for diffusion-based image production.[64] Also, Google released a transformer-centric image generator called "Muse" based on parallel decoding and masked generative transformer technology.[65] (Transformers played a less-central role with prior image-producing technologies,[66] albeit still a significant one.[67])

## See also[edit]

* Perceiver – Machine learning algorithm for non-textual data
* BERT (language model) – Language model developed by Google
* GPT-3 – 2020 large language model
* GPT-4 – 2023 text-generating language model
* ChatGPT – AI chatbot developed by OpenAI
* Wu Dao – Chinese multimodal artificial intelligence program
* Vision transformer – Machine learning algorithm for vision processing
* BLOOM (language model) – Open-access multilingual language model

## References[edit]

1. ^ a b c d e f g h i j Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All you Need" (PDF). Advances in Neural Information Processing Systems. Curran Associates, Inc. 30.
2. ^ Hochreiter, Sepp; Schmidhuber, Jürgen (1 November 1997). "Long Short-Term Memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. ISSN 0899-7667. PMID 9377276. S2CID 1915014.
3. ^ a b "Better Language Models and Their Implications". OpenAI. 2019-02-14. Archived from the original on 2020-12-19. Retrieved 2019-08-25.
4. ^ a b Bahdanau; Cho, Kyunghyun; Bengio, Yoshua (September 1, 2014). "Neural Machine Translation by Jointly Learning to Align and Translate". arXiv:1409.0473 [cs.CL].
5. ^ a b Luong, Minh-Thang; Pham, Hieu; Manning, Christopher D. (August 17, 2015). "Effective Approaches to Attention-based Neural Machine Translation". arXiv:1508.04025 [cs.CL].
6. ^ a b c Schmidhuber, Jürgen (1992). "Learning to control fast-weight memories: an alternative to recurrent nets". Neural Computation. 4 (1): 131–139. doi:10.1162/neco.1992.4.1.131. S2CID 16683347.
7. ^ a b Schlag, Imanol; Irie, Kazuki; Schmidhuber, Jürgen (2021). "Linear Transformers Are Secretly Fast Weight Programmers". ICML 2021. Springer. pp. 9355–9366.
8. ^ a b Katharopoulos, Angelos; Vyas, Apoorv; Pappas, Nikolaos; Fleuret, François (2020). "Transformers are RNNs: Fast autoregressive Transformers with linear attention". ICML 2020. PMLR. pp. 5156–5165.
9. ^ He, Cheng (31 December 2021). "Transformer in CV". Transformer in CV. Towards Data Science. Archived from the original on 16 April 2023. Retrieved 19 June 2021.
10. ^ Radford, Alec; Jong Wook Kim; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). "Robust Speech Recognition via Large-Scale Weak Supervision". arXiv:2212.04356 [eess.AS].
11. ^ a b Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Pierric; Rault, Tim; Louf, Remi; Funtowicz, Morgan; Davison, Joe; Shleifer, Sam; von Platen, Patrick; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander (2020). "Transformers: State-of-the-Art Natural Language Processing". Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. doi:10.18653/v1/2020.emnlp-demos.6. S2CID 208117506.
12. ^ a b c "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing". Google AI Blog. 2 November 2018. Archived from the original on 2021-01-13. Retrieved 2019-08-25.
13. ^ Elman, Jeffrey L. (March 1990). "Finding Structure in Time". Cognitive Science. 14 (2): 179–211. doi:10.1207/s15516709cog1402_1. S2CID 2763403.
14. ^ Choromanski, Krzysztof; Likhosherstov, Valerii; Dohan, David; Song, Xingyou; Gane, Andreea; Sarlos, Tamas; Hawkins, Peter; Davis, Jared; Mohiuddin, Afroz; Kaiser, Lukasz; Belanger, David; Colwell, Lucy; Weller, Adrian (2020). "Rethinking Attention with Performers". arXiv:2009.14794 [cs.CL].
15. ^ a b Schmidhuber, Jürgen (1993). "Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets". ICANN 1993. Springer. pp. 460–463.
16. ^ Brown, Peter F. (1993). "The mathematics of statistical machine translation: Parameter estimation". Computational Linguistics (19): 263–311.
17. ^ Banko, Michele; Brill, Eric (2001). "Scaling to very very large corpora for natural language disambiguation". Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01. Morristown, NJ, USA: Association for Computational Linguistics: 26–33. doi:10.3115/1073012.1073017. S2CID 6645623.
18. ^ Sutskever, Ilya; Vinyals, Oriol; Le, Quoc V (2014). "Sequence to Sequence Learning with Neural Networks". Advances in Neural Information Processing Systems. Curran Associates, Inc. 27. arXiv:1409.3215.
19. ^ Cho, Kyunghyun; van Merrienboer, Bart; Bahdanau, Dzmitry; Bengio, Yoshua (2014). "On the Properties of Neural Machine Translation: Encoder–Decoder Approaches". Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Stroudsburg, PA, USA: Association for Computational Linguistics: 103–111. arXiv:1409.1259. doi:10.3115/v1/w14-4012. S2CID 11336213.
20. ^ Chung, Junyoung; Gulcehre, Caglar; Cho, KyungHyun; Bengio, Yoshua (2014). "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling". arXiv:1412.3555 [cs.NE].
21. ^ Gruber, N.; Jockisch, A. (2020), "Are GRU cells more specific and LSTM cells more sensitive in motive classification of text?", Frontiers in Artificial Intelligence, 3: 40, doi:10.3389/frai.2020.00040, PMC 7861254, PMID 33733157, S2CID 220252321
22. ^ Bahdanau, Dzmitry; Cho, Kyunghyun; Bengio, Yoshua (2014-09-01). "Neural Machine Translation by Jointly Learning to Align and Translate". arXiv:1409.0473 [cs.CL].
23. ^ "Google Scholar". scholar.google.com. Retrieved 2023-08-13.
24. ^ Lewis-Kraus, Gideon (2016-12-14). "The Great A.I. Awakening". The New York Times. ISSN 0362-4331. Archived from the original on 24 May 2023. Retrieved 2023-06-22.
25. ^ Wu, Yonghui; et al. (2016-09-01). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". arXiv:1609.08144 [cs.CL].
26. ^ Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 October 2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2 [cs.CL].
27. ^ a b Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn, Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa; Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob (2021-06-03). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale". arXiv:2010.11929 [cs.CV].
28. ^ a b Gulati, Anmol; Qin, James; Chiu, Chung-Cheng; Parmar, Niki; Zhang, Yu; Yu, Jiahui; Han, Wei; Wang, Shibo; Zhang, Zhengdong; Wu, Yonghui; Pang, Ruoming (2020). "Conformer: Convolution-augmented Transformer for Speech Recognition". arXiv:2005.08100 [eess.AS].
29. ^ a b Xiong, Ruibin; Yang, Yunchang; He, Di; Zheng, Kai; Zheng, Shuxin; Xing, Chen; Zhang, Huishuai; Lan, Yanyan; Wang, Liwei; Liu, Tie-Yan (2020-06-29). "On Layer Normalization in the Transformer Architecture". arXiv:2002.04745 [cs.LG].
30. ^ "Improving language understanding with unsupervised learning". openai.com. June 11, 2018. Archived from the original on 2023-03-18. Retrieved 2023-03-18.
31. ^ finetune-transformer-lm, OpenAI, June 11, 2018, retrieved 2023-05-01
32. ^ "Papers with Code – A Decomposable Attention Model for Natural Language Inference". paperswithcode.com.
33. ^ Chen, Jia; Chen, Tao; Shen, Mengqi; Shi, Yunhai; Wang, Dongjing; Zhang, Xin (2022-09-01). "Gated three-tower transformer for text-driven stock market prediction". Multimedia Tools and Applications. 81 (21): 30093–30119. doi:10.1007/s11042-022-11908-1. ISSN 1573-7721.
34. ^ a b Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michael; Zhou, Yanqi; Li, Wei; Liu, Peter J. (2020-01-01). "Exploring the limits of transfer learning with a unified text-to-text transformer". The Journal of Machine Learning Research. 21 (1): 140:5485–140:5551. ISSN 1532-4435.
35. ^ Kariampuzha, William; Alyea, Gioconda; Qu, Sue; Sanjak, Jaleal; Mathé, Ewy; Sid, Eric; Chatelaine, Haley; Yadaw, Arjun; Xu, Yanji; Zhu, Qian (2023). "Precision information extraction for rare disease epidemiology at scale". Journal of Translational Medicine. 21 (1): 157. doi:10.1186/s12967-023-04011-y. PMC 9972634. PMID 36855134.
36. ^ Assael, Yannis; Sommerschield, Thea; Shillingford, Brendan; Bordbar, Mahyar; Pavlopoulos, John; Chatzipanagiotou, Marita; Androutsopoulos, Ion; Prag, Jonathan; de Freitas, Nando (March 2022). "Restoring and attributing ancient texts using deep neural networks". Nature. 603 (7900): 280–283. Bibcode:2022Natur.603..280A. doi:10.1038/s41586-022-04448-z. ISSN 1476-4687. PMC 8907065. PMID 35264762.
37. ^ "Sequence Modeling with Neural Networks (Part 2): Attention Models". Indico. 2016-04-18. Archived from the original on 2020-10-21. Retrieved 2019-10-15.
38. ^ a b c Alammar, Jay. "The Illustrated Transformer". jalammar.github.io. Archived from the original on 2020-10-18. Retrieved 2019-10-15.
39. ^ Clark, Kevin; Khandelwal, Urvashi; Levy, Omer; Manning, Christopher D. (August 2019). "What Does BERT Look at? An Analysis of BERT's Attention". Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Florence, Italy: Association for Computational Linguistics: 276–286. doi:10.18653/v1/W19-4828. Archived from the original on 2020-10-21. Retrieved 2020-05-20.
40. ^ LeCun, Yann (Apr 28, 2023). "A survey of LLMs with a practical guide and evolutionary tree". Twitter. Archived from the original on 23 Jun 2023. Retrieved 2023-06-23.
41. ^ "Masked language modeling". huggingface.co. Retrieved 2023-10-05.
42. ^ "Causal language modeling". huggingface.co. Retrieved 2023-10-05.
43. ^ Shazeer, Noam (2020-02-01). "GLU Variants Improve Transformer". arXiv:2002.05202 [cs.LG].
44. ^ Dufter, Philipp; Schmitt, Martin; Schütze, Hinrich (2022-06-06). "Position Information in Transformers: An Overview". Computational Linguistics. 48 (3): 733–763. doi:10.1162/coli_a_00445. ISSN 0891-2017. S2CID 231986066.
45. ^ Su, Jianlin; Lu, Yu; Pan, Shengfeng; Murtadha, Ahmed; Wen, Bo; Liu, Yunfeng (2021-04-01). "RoFormer: Enhanced Transformer with Rotary Position Embedding". arXiv:2104.09864 [cs.CL].
46. ^ Press, Ofir; Smith, Noah A.; Lewis, Mike (2021-08-01). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". arXiv:2108.12409 [cs.CL].
47. ^ Shaw, Peter; Uszkoreit, Jakob; Vaswani, Ashish (2018). "Self-Attention with Relative Position Representations". arXiv:1803.02155 [cs.CL].
48. ^ Dao, Tri; Fu, Dan; Ermon, Stefano; Rudra, Atri; Ré, Christopher (2022-12-06). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". Advances in Neural Information Processing Systems. 35: 16344–16359. arXiv:2205.14135.
49. ^ "Stanford CRFM". crfm.stanford.edu. Retrieved 2023-07-18.
50. ^ "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning". Princeton NLP. 2023-06-17. Retrieved 2023-07-18.
51. ^ "Introducing Together AI Chief Scientist Tri Dao, as he releases FlashAttention-2 to speed up model training and inference". TOGETHER. Retrieved 2023-07-18.
52. ^ Chowdhery, Aakanksha; Narang, Sharan; Devlin, Jacob; Bosma, Maarten; Mishra, Gaurav; Roberts, Adam; Barham, Paul; Chung, Hyung Won; Sutton, Charles; Gehrmann, Sebastian; Schuh, Parker; Shi, Kensen; Tsvyashchenko, Sasha; Maynez, Joshua; Rao, Abhishek (2022-04-01). "PaLM: Scaling Language Modeling with Pathways". arXiv:2204.02311 [cs.CL].
53. ^ a b Kitaev, Nikita; Kaiser, Łukasz; Levskaya, Anselm (2020). "Reformer: The Efficient Transformer". arXiv:2001.04451 [cs.LG].
54. ^ "Constructing Transformers For Longer Sequences with Sparse Attention Methods". Google AI Blog. 25 March 2021. Archived from the original on 2021-09-18. Retrieved 2021-05-28.
55. ^ "Tasks with Long Sequences – Chatbot". Coursera. Archived from the original on 2020-10-26. Retrieved 2020-10-22.
56. ^ "Reformer: The Efficient Transformer". Google AI Blog. 16 January 2020. Archived from the original on 2020-10-22. Retrieved 2020-10-22.
57. ^ Zhai, Shuangfei; Talbott, Walter; Srivastava, Nitish; Huang, Chen; Goh, Hanlin; Zhang, Ruixiang; Susskind, Josh (2021-09-21). "An Attention Free Transformer". arXiv:2105.14103 [cs.LG].
58. ^ Tay, Yi; Dehghani, Mostafa; Abnar, Samira; Shen, Yikang; Bahri, Dara; Pham, Philip; Rao, Jinfeng; Yang, Liu; Ruder, Sebastian; Metzler, Donald (2020-11-08). "Long Range Arena: A Benchmark for Efficient Transformers". arXiv:2011.04006 [cs.LG].
59. ^ Peng, Hao; Pappas, Nikolaos; Yogatama, Dani; Schwartz, Roy; Smith, Noah A.; Kong, Lingpeng (2021-03-19). "Random Feature Attention". arXiv:2103.02143 [cs.CL].
60. ^ Choromanski, Krzysztof; Likhosherstov, Valerii; Dohan, David; Song, Xingyou; Gane, Andreea; Sarlos, Tamas; Hawkins, Peter; Davis, Jared; Belanger, David; Colwell, Lucy; Weller, Adrian (2020-09-30). "Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers". arXiv:2006.03555 [cs.LG].
61. ^ Radford, Alec; Kim, Jong Wook; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). "Robust Speech Recognition via Large-Scale Weak Supervision". arXiv:2212.04356 [eess.AS].
62. ^ Jaegle, Andrew; Gimeno, Felix; Brock, Andrew; Zisserman, Andrew; Vinyals, Oriol; Carreira, Joao (2021-06-22). "Perceiver: General Perception with Iterative Attention". arXiv:2103.03206 [cs.CV].
63. ^ Jaegle, Andrew; Borgeaud, Sebastian; Alayrac, Jean-Baptiste; Doersch, Carl; Ionescu, Catalin; Ding, David; Koppula, Skanda; Zoran, Daniel; Brock, Andrew; Shelhamer, Evan; Hénaff, Olivier (2021-08-02). "Perceiver IO: A General Architecture for Structured Inputs & Outputs". arXiv:2107.14795 [cs.LG].
64. ^ Peebles, William; Xie, Saining (March 2, 2023). "Scalable Diffusion Models with Transformers". arXiv:2212.09748 [cs.CV].
65. ^ "Google AI Unveils Muse, a New Text-to-Image Transformer Model". InfoQ.
66. ^ "Using Diffusion Models to Create Superior NeRF Avatars". January 5, 2023.
67. ^ Islam, Arham (November 14, 2022). "How Do DALL·E 2, Stable Diffusion, and Midjourney Work?".

## Further reading[edit]

* Hubert Ramsauer et al. (2020), "Hopfield Networks is All You Need" Archived 2021-09-18 at the Wayback Machine, preprint submitted for ICLR 2021\. arXiv:2008.02217; see also authors' blog Archived 2021-09-18 at the Wayback Machine

– Discussion of the effect of a transformer layer as equivalent to a Hopfield update, bringing the input closer to one of the fixed points (representable patterns) of a continuous-valued Hopfield network

* Alexander Rush, The Annotated transformer Archived 2021-09-22 at the Wayback Machine, Harvard NLP group, 3 April 2018
* Phuong, Mary; Hutter, Marcus (2022), Formal Algorithms for Transformers, arXiv:2207.09238

* v
* t
* e

Google

* Alphabet Inc.
* History
* List of Android apps
* List of Easter eggs
* April Fools' Day
* List of mergers and acquisitions

Company

| Divisions|

* Ads
* AI
* Brain
* DeepMind
* Android
* booting process
* recovery mode
* software development
* version history
* China
* Goojje
* Chrome
* Cloud
* Glass
* Google.org
* Crisis Response
* Public Alerts
* RechargeIT
* Health
* Maps
* Pixel
* Search
* Timeline
* Sidewalk Labs
* Sustainability
* YouTube
* History
* "Me at the zoo"
* Social impact
* YouTuber

|

People|

| Current|

* Krishna Bharat
* Vint Cerf
* Jeff Dean
* John Doerr
* Sanjay Ghemawat
* Al Gore
* John L. Hennessy
* Urs Hölzle
* Salar Kamangar
* Ray Kurzweil
* Ann Mather
* Alan Mulally
* Rick Osterloh
* Sundar Pichai (CEO)
* Ruth Porat (CFO)
* Rajen Sheth
* Hal Varian
* Susan Wojcicki
* Neal Mohan

|

Former|

* Andy Bechtolsheim
* Sergey Brin (Founder)
* David Cheriton
* Matt Cutts
* David Drummond
* Alan Eustace
* Timnit Gebru
* Omid Kordestani
* Paul Otellini
* Larry Page (Founder)
* Patrick Pichette
* Eric Schmidt
* Ram Shriram
* Amit Singhal
* Shirley M. Tilghman
* Rachel Whetstone

Real estate|

* 111 Eighth Avenue
* Androidland
* Barges
* Binoculars Building
* Central Saint Giles
* Chelsea Market
* Chrome Zone
* Data centers
* Modular
* Googleplex
* Mayfield Mall
* Pier 57
* Sidewalk Toronto
* St. John's Terminal
* YouTube Space
* YouTube Theater

Design|

* Fonts
* Croscore
* Noto
* Product Sans
* Roboto
* Logo
* Doodle
* Doodle Champion Island Games
* Magic Cat Academy
* Material Design

Events|

* Android
* Developer Challenge
* Developer Day
* Developer Lab
* Code-in
* Code Jam
* Developer Day
* Developers Live
* Doodle4Google
* G-Day
* I/O
* Jigsaw
* Living Stories
* Lunar XPRIZE
* Mapathon
* Science Fair
* Summer of Code
* Talks at Google

YouTube|

* Awards
* CNN/YouTube presidential debates
* Comedy Week
* Live
* Music Awards
* Space Lab
* Symphony Orchestra

Projects and
initiatives|

* 20% project
* Area 120
* Reply
* Tables
* ATAP
* Business Groups
* Computing University Initiative
* Data Liberation Front
* Data Transfer Project
* Developer Expert
* Digital Garage
* Digital News Initiative
* Digital Unlocked
* Dragonfly
* Founders' Award
* Free Zone
* Get Your Business Online
* Google for Education
* Google for Startups
* Labs
* Liquid Galaxy
* Made with Code
* Māori
* ML FairnessNative Client
* News Lab
* Nightingale
* OKR
* PowerMeter
* Privacy Sandbox
* Quantum Artificial Intelligence Lab
* RechargeIT
* Shield
* Silicon Initiative
* Solve for X
* Starline
* Student Ambassador Program
* Submarine communications cables
* Dunant
* Grace Hopper
* Sunroof
* Versus Debates
* YouTube
* Creator Awards
* Next Lab and Audience Development Group
* Original Channel Initiative
* Zero

Criticism|

* 2018 data breach
* 2018 walkouts
* Alphabet Workers Union
* Censorship
* DeGoogle
* "Did Google Manipulate Search for Hillary?"
* Dragonfly
* FairSearch
* "Ideological Echo Chamber" memo
* Litigation
* Privacy concerns
* Street View
* San Francisco tech bus protests
* Services outages
* Smartphone patent wars
* Worker organization

YouTube|

* Back advertisement controversy
* Censorship
* Copyright issues
* Copyright strike
* Elsagate
* Fantastic Adventures scandal
* Headquarters shooting
* Kohistan video case
* Reactions to Innocence of Muslims
* Slovenian government incident

Development

| Operating systems|

* Android
* Automotive
* Glass OS
* Go
* gLinux
* Goobuntu
* Things
* TV
* Wear OS
* ChromeOS
* ChromiumOS
* Neverware
* Fuchsia
* TV

|

Libraries/
frameworks|

* ALTS
* AMP
* Angular
* JS
* ARCore
* APIs
* Blockly
* Chart API
* Charts
* Dialogflow
* Exposure Notification
* Fast Pair
* Federated Learning of Cohorts
* File System
* FlatBuffers
* Flutter
* Gears
* gRPC
* Gson
* Guava
* Guice
* Guetzli
* JAX
* gVisor
* MapReduce
* Matter
* Mobile Services
* Neural Machine Translation
* OpenSocial
* Pack
* Polymer
* Protocol Buffers
* Reqwireless
* Shell
* Skia Graphics Engine
* Tango
* TensorFlow
* Test
* WaveNet
* Weave
* Web Accelerator
* WebRTC

Platforms|

* App Engine
* AppJet
* Apps Script
* Cloud Platform
* Anvato
* Firebase
* Cloud Messaging
* Crashlytics
* Global IP Solutions
* Internet Low Bitrate Codec
* Internet Speech Audio Codec
* Gridcentric, Inc.
* ITA Software
* Kubernetes
* LevelDB
* Neatx
* SageTV

Apigee|

* Bigtable
* Bitium
* Chronicle
* VirusTotal
* Compute Engine
* Connect
* Dataflow
* Datastore
* Kaggle
* Looker
* Mandiant
* Messaging
* Orbitera
* Shell
* Stackdriver
* Storage

Tools|

* American Fuzzy Lop
* Android Cloud to Device Messaging
* Android Debug Bridge
* Android Studio
* App Maker
* App Runtime for Chrome
* AppSheet
* Bazel
* Chrome Frame
* Closure Tools
* Cpplint
* Data Protocol
* Gadgets
* Gerrit
* GYP
* Kythe
* Lighthouse
* MIT App Inventor
* Mashup Editor
* Native Client
* Optimize
* OpenRefine
* OR-Tools
* PageSpeed
* Plugin for Eclipse
* Programmable Search Engine
* Public DNS
* reCAPTCHA
* Schema.org
* Search Console
* Sitemaps
* Swiffy
* Tesseract (software)
* Trendalyzer
* VisBug
* Wave Federation Protocol
* Web Toolkit

Search algorithms|

* Hummingbird
* PageRank
* applications in biochemistry
* Matrix
* Panda
* Penguin
* Pigeon
* RankBrain

Others|

* BERT
* BigQuery
* Chrome Experiments
* Flutter
* Gemini
* Googlebot
* Keyhole Markup Language
* LaMDA
* Open Location Code
* PaLM
* Programming languages
* Caja
* Carbon
* Dart
* Go
* Sawzall
* Transformer
* Viewdle
* Webdriver Torso
* Web Server

File formats|

* AAB
* APK
* AV1
* On2 Technologies
* VP3
* VP6
* VP8
* libvpx
* VP9
* WebM
* WebP
* WOFF2

Products

| Entertainment|

* Currents (news app)
* Green Throttle Games
* Owlchemy Labs
* Oyster
* PaperofRecord.com
* Podcasts
* Quick, Draw!
* Santa Tracker
* Songza
* Stadia
* games
* Typhoon Studios
* TV
* Vevo
* Video

Play|

* Books
* Games
* most downloaded apps
* Music
* Newsstand
* Pass
* Services

YouTube|

* BandPage
* BrandConnect
* Content ID
* Instant
* Kids
* Music
* Official channel
* Preferred
* Premium
* original programming
* YouTube Rewind
* RightsFlow
* Shorts
* Studio
* TV

Communication|

* Allo
* Bump
* Buzz
* Chat
* Contacts
* Currents (social app)
* Dodgeball
* Duo
* Fi Wireless
* Friend Connect
* Gizmo5
* Google+
* Gmail
* History
* Inbox
* Interface
* Groups
* Hangouts
* Helpouts
* IME
* Japanese
* Pinyin
* Transliteration
* Jaiku
* Marratech
* Meebo
* Meet
* Messages
* Moderator
* Neotonic Software
* Orkut
* Postini
* Quest Visual
* Word Lens
* Schemer
* Spaces
* Sparrow
* Talk
* Translate
* Translator Toolkit
* Voice
* Voice Local Search
* Wave

Search|

* Aardvark
* Alerts
* Answers
* Base
* BeatThatQuote.com
* Blog Search
* Books
* Ngram Viewer
* Code Search
* Data Commons
* Dataset Search
* Dictionary
* Directory
* Fast Flip
* Flu Trends
* Finance
* Goggles
* Google.by
* Images
* Image Labeler
* Image Swirl
* Kaltix
* Knowledge Graph
* Freebase
* Metaweb
* Like.com
* News
* Archive
* Weather
* Patents
* People Cards
* Personalized Search
* Public Data Explorer
* Questions and Answers
* SafeSearch
* Scholar
* Searchwiki
* Shopping
* Catalogs
* Express
* Squared
* Tenor
* Travel
* Flights
* Trends
* Insights for Search
* Voice Search
* WDYL

Navigation|

* Earth
* Endoxon
* ImageAmerica
* Maps
* Latitude
* Map Maker
* Navigation
* Pin
* Street View
* Coverage
* Trusted
* Waze

Business
and finance|

* Ad Manager
* AdMob
* Ads
* Adscape
* AdSense
* Attribution
* BebaPay
* Checkout
* Contributor
* DoubleClick
* Affiliate Network
* Invite Media
* Marketing Platform
* Analytics
* Looker Studio
* Urchin
* Pay (mobile app)
* Wallet
* Pay (payment method)
* Send
* Tez
* PostRank
* Primer
* Softcard
* Wildfire Interactive
* Widevine

Organization
and productivity|

* Bookmarks
* Browser Sync
* Calendar
* Cloud Search
* Desktop
* Drive
* Etherpad
* fflick
* Files
* iGoogle
* Jamboard
* Notebook
* One
* Photos
* Quickoffice
* Quick Search Box
* Surveys
* Sync
* Tasks
* Toolbar

Docs Editors|

* Docs
* Drawings
* Forms
* Fusion Tables
* Keep
* Sheets
* Slides
* Sites

Publishing|

* Apture
* Blogger
* Pyra Labs
* Domains
* FeedBurner
* One Pass
* Page Creator
* Sites
* Web Designer

Education|

* Classroom
* Grasshopper
* Socratic
* Photomath
* Read Along
* Workspace
* Marketplace

Others|

* Account
* Dashboard
* Takeout
* Android Auto
* Android Beam
* Arts & Culture
* Assistant
* Authenticator
* Bard
* Body
* BufferBox
* Building Maker
* BumpTop
* Cast
* List of supported apps
* Cloud Print
* Crowdsource
* Expeditions
* Family Link
* Find My Device
* Fit
* Google Fonts
* Gboard
* Gesture Search
* Impermium
* Knol
* Lively
* Live Transcribe
* MyTracks
* Nearby Share
* Now
* Offers
* Opinion Rewards
* Person Finder
* PlinkArt
* Poly
* Question Hub
* Reader
* Safe Browsing
* Sidewiki
* SlickLogin
* Sound Amplifier
* Speech Services
* Station
* Store
* TalkBack
* Tilt Brush
* URL Shortener
* Voice Access
* Wavii
* Web Light
* WiFi

Chrome|

* Apps
* Chromium
* Dinosaur Game
* GreenBorder
* Remote Desktop
* Web Store
* V8

Images and
photography|

* Camera
* Lens
* Snapseed
* Nik Software
* Panoramio
* Photos
* Picasa
* Web Albums
* Picnik

Hardware

| Smartphones|

* Android Dev Phone
* Android One
* Nexus
* Nexus One
* S
* Galaxy Nexus
* 4
* 5
* 6
* 5X
* 6P
* Comparison
* Pixel
* Pixel
* 2
* 3
* 3a
* 4
* 4a
* 5
* 5a
* 6
* 6a
* 7
* 7a
* Fold
* 8
* Comparison
* Play Edition
* Project Ara

|

Laptops and tablets|

* Chromebook
* Nexus
* 7 (2012)
* 7 (2013)
* 10
* 9
* Comparison
* Pixel
* Chromebook Pixel
* Pixelbook
* Pixelbook Go
* C
* Slate
* Tablet

Wearables|

* Fitbit
* List of products
* Pixel Buds
* Pixel Watch
* Pixel Watch 2
* Project Iris (unreleased)
* Virtual reality
* Cardboard
* Contact Lens
* Daydream
* Glass

Others|

* Chromebit
* Chromebox
* Clips
* Digital media players
* Chromecast
* Nexus Player
* Nexus Q
* Dropcam
* Liquid Galaxy
* Nest
* Smart Speakers
* Thermostat
* Wifi
* OnHub
* Pixel Visual Core
* Search Appliance
* Sycamore processor
* Tensor
* Tensor Processing Unit
* Titan Security Key

Related

| Terms and phrases|

* "Don't be evil"
* Gayglers
* Google (verb)
* Google bombing
* 2004 U.S. presidential election
* Google effect
* Googlefight
* Google hacking
* Googleshare
* Google tax
* Googlewhack
* Googlization
* "Illegal flower tribute"
* Rooting
* Search engine manipulation effect
* Sitelink
* Site reliability engineering
* YouTube poop

|

Documentaries|

* AlphaGo
* Google: Behind the Screen
* Google Maps Road Trip
* Google and the World Brain
* The Creepy Line

Books|

* Google Hacks
* The Google Story
* Google Volume One
* Googled: The End of the World as We Know It
* How Google Works
* I'm Feeling Lucky
* In the Plex
* The Google Book

Popular culture|

* Google Feud
* Google Me (film)
* "Google Me" (Kim Zolciak song)
* "Google Me" (Teyana Taylor song)
* Is Google Making Us Stupid?
* Proceratium google
* Matt Nathanson: Live at Google
* The Billion Dollar Code
* The Internship
* Where on Google Earth is Carmen Sandiego?

Others|

* "Attention Is All You Need"
* elgooG
* g.co
* .google
* Pimp My Search
* Predictions of the end
* Relationship with Wikipedia
* Sensorvault
* Stanford Digital Library Project

* Italics indicate discontinued products or services.
* Category
* Commons
* Outline
* WikiProject

* v
* t
* e

Differentiable computing

General|

* Differentiable programming
* Information geometry
* Statistical manifold
* Automatic differentiation
* Neuromorphic engineering
* Pattern recognition
* Tensor calculus
* Computational learning theory
* Inductive bias

Concepts|

* Gradient descent
* SGD
* Clustering
* Regression
* Overfitting
* Hallucination
* Adversary
* Attention
* Convolution
* Loss functions
* Backpropagation
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Regularization
* Datasets
* Augmentation
* Diffusion
* Autoregression

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Scientific computing
* Artificial Intelligence
* Language model
* Large language model

Hardware|

* IPU
* TPU
* VPU
* Memristor
* SpiNNaker

Software libraries|

* TensorFlow
* PyTorch
* Keras
* Theano
* JAX
* Flux.jl

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Speech synthesis
* Speech recognition
* Facial recognition
* AlphaFold
* DALL-E
* Midjourney
* Stable Diffusion
* Whisper

|

Verbal|

* Word2vec
* Seq2seq
* BERT
* Gemini
* LaMDA
* Bard
* NMT
* Project Debater
* IBM Watson
* GPT-1
* GPT-2
* GPT-3
* GPT-4
* ChatGPT
* GPT-J
* Chinchilla AI
* PaLM
* BLOOM
* LLaMA

Decisional|

* AlphaGo
* AlphaZero
* Q-learning
* SARSA
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* Auto-GPT
* Robot control

People|

* Yoshua Bengio
* Alex Graves
* Ian Goodfellow
* Stephen Grossberg
* Demis Hassabis
* Geoffrey Hinton
* Yann LeCun
* Fei-Fei Li
* Andrew Ng
* Jürgen Schmidhuber
* David Silver
* Ilya Sutskever

Organizations|

* Anthropic
* EleutherAI
* Google DeepMind
* Hugging Face
* OpenAI
* Meta AI
* Mila
* MIT CSAIL

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network
* Residual neural network
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network

* Portals
* Computer programming
* Technology
* Categories
* Artificial neural networks
* Machine learning

*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template