GloVe

GloVe, coined from Global Vectors, is a model for distributed word representation. The model is an unsupervised learning algorithm for obtaining vector representations of words. This is achieved by mapping words into a meaningful space where the distance between words is related to semantic similarity.[1] Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. As log-bilinear regression model for unsupervised learning of word representations, it combines the features of two model families, namely the global matrix factorization and local context window methods.

It was developed as an open-source project at Stanford University[2] and launched in 2014. It was designed as a competitor to word2vec, and the original paper noted multiple improvements of GloVe over word2vec. As of 2022[update], both approaches are outdated, and Transformer-based models, such as BERT, which add multiple neural-network attention layers on top of a word embedding model similar to Word2vec, have come to be regarded as the state of the art in NLP.[3]

## Contents

* 1 Definition
* 1.1 Word counting
* 1.2 Probabilistic modelling
* 1.3 Logistic regression
* 1.4 Use
* 2 Applications
* 3 See also
* 4 References
* 5 External links

## Definition

> You shall know a word by the company it keeps (Firth, J. R. 1957:11)[4]

The idea of GloVe is to construct, for each word

, two vectors

wi,w~i

, such that the relative positions of the vectors capture part of the statistical regularities of the word

. The statistical regularity is defined as the co-occurrence probabilities. Words that resemble each other in meaning should also resemble each other in co-occurrence probabilities.

### Word counting

Let the vocabulary be V, the set of all possible words (aka "tokens"). Punctuation is either ignored, or treated as vocabulary, and similarly for capitalization and other typographical details.[1]

If two words occur close to each other, then we say that they occur in the context of each other. For example, if the context length is 3, then we say that in the following sentence

> GloVe1, coined2 from3 Global4 Vectors5, is6 a7 model8 for9 distributed10 word11 representation12

the word "model8" is in the context of "word11" but not the context of "representation12".

A word is not in the context of itself, so "model8" is not in the context of the word "model8", although, if a word appears again in the same context, then it does count.

Let Xij be the number of times that the word j appears in the context of the word i over the entire corpus. For example, if the corpus is just "I don't think that that is a problem." we have Xthat,that=2 since the first "that" appears in the second one's context, and vice versa.

Let Xi=∑j∈VXij be the number of words in the context of all instances of word i. By counting, we haveXi=2×(context size)×#(occurrences of word i)(except for words occurring right at the start and end of the corpus)

### Probabilistic modelling

Let Pik:=P(k|i):=XikXibe the co-occurrence probability. That is, if one samples a random occurrence of the word i in the entire document, and a random word within its context, that word is k with probability Pik. Note that Pik≠Pki in general. For example, in a typical modern English corpus, Pado,much is close to one, but Pmuch,ado is close to zero. This is because the word "ado" is almost only used in the context of the archaic phrase "much ado about", but the word "much" occurs in all kinds of contexts.

For example, in a 6 billion token corpus, we have

Table 1 of [1] Probability and Ratio  | k= solid | k= gas | k= water | k= fashion

|
|
|
|

P(k∣ ice ) | 1.9×10−4 | 6.6×10−5 | 3.0×10−3 | 1.7×10−5
P(k∣ steam ) | 2.2×10−5 | 7.8×10−4 | 2.2×10−3 | 1.8×10−5
P(k∣ ice )/P(k∣ steam ) | 8.9 | 8.5×10−2 | 1.36 | 0.96

Inspecting the table, we see that the words "ice" and "steam" are indistinguishable along the "water" (often co-occurring with both) and "fashion" (rarely co-occurring with either), but distinguishable along the "solid" (co-occurring more with ice) and "gas" (co-occurring more with "steam").

The idea is to learn two vectors wi,w~i for each word i, such that we have a multinomial logistic regression:wiTw~j+bi+b~j≈ln⁡Pijand the terms bi,b~j are unimportant parameters.

This means that if the words i,j have similar co-occurrence probabilities (Pik)k∈V≈(Pjk)k∈V, then their vectors should also be similar: wi≈wj.

### Logistic regression

Naively, logistic regression can be run by minimizing the squared loss:L=∑i,j∈V(wiTw~j+bi+b~j−ln⁡Pij)2However, this would be noisy for rare co-occurrences. To fix the issue, the squared loss is weighted so that the loss is slowly ramped-up as the absolute number of co-occurrences Xij increases:L=∑i,j∈Vf(Xij)(wiTw~j+bi+b~j−ln⁡Pij)2wheref(x)={(x/xmax)α if x<xmax1 otherwise and xmax,α are hyperparameters. In the original paper, the authors found that xmax=100,α=3/4 seem to work well in practice.

### Use

Once a model is trained, we have 4 trained parameters for each word: wi,w~i,bi,b~i. The parameters bi,b~i are irrelevant, and only wi,w~i are relevant.

The authors recommended using wi+w~i as the final representation vector for word i, because empirically it worked better than wi or w~i alone.

## Applications

GloVe can be used to find relations between words like synonyms, company-product relations, zip codes and cities, etc. However, the unsupervised learning algorithm is not effective in identifying homographs, i.e., words with the same spelling and different meanings. This is as the unsupervised learning algorithm calculates a single set of vectors for words with the same morphological structure.[5] The algorithm is also used by the SpaCy library to build semantic word embedding features, while computing the top list words that match with distance measures such as cosine similarity and Euclidean distance approach.[6] GloVe was also used as the word representation framework for the online and offline systems designed to detect psychological distress in patient interviews.[7]

## See also

* ELMo
* BERT
* Word2vec
* fastText
* Natural language processing

## References

1. ↑ 1.0 1.1 1.2 Pennington, Jeffrey; Socher, Richard; Manning, Christopher (October 2014). "Glove: Global Vectors for Word Representation". in Moschitti, Alessandro; Pang, Bo; Daelemans, Walter. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics. pp. 1532–1543. doi:10.3115/v1/D14-1162. https://aclanthology.org/D14-1162.
2. ↑ GloVe: Global Vectors for Word Representation (pdf) "We use our insights to construct a new model for word representation which we call GloVe, for Global Vectors, because the global corpus statistics are captured directly by the model."
3. ↑ Von der Mosel, Julian; Trautsch, Alexander; Herbold, Steffen (2022). "On the validity of pre-trained transformers for natural language processing in the software engineering domain". IEEE Transactions on Software Engineering 49 (4): 1487–1507. doi:10.1109/TSE.2022.3178469. ISSN 1939-3520.
4. ↑ Firth, J. R. (1957). Studies in Linguistic Analysis. Wiley-Blackwell. https://cs.brown.edu/courses/csci2952d/readings/lecture1-firth.pdf.
5. ↑ Wenig, Phillip (2019). "Creation of Sentence Embeddings Based on Topical Word Representations: An approach towards universal language understanding". Towards Data Science.
6. ↑ Singh, Mayank; Gupta, P. K.; Tyagi, Vipin; Flusser, Jan; Ören, Tuncer I. (2018). Advances in Computing and Data Sciences: Second International Conference, ICACDS 2018, Dehradun, India, April 20-21, 2018, Revised Selected Papers. Singapore: Springer. pp. 171. ISBN 9789811318122.
7. ↑ Abad, Alberto; Ortega, Alfonso; Teixeira, António; Mateo, Carmen; Hinarejos, Carlos; Perdigão, Fernando; Batista, Fernando; Mamede, Nuno (2016). Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings. Cham: Springer. pp. 165. ISBN 9783319491691.

## External links

* GloVe
* Deeplearning4j GloVe

* v
* t
* e

Natural language processing

General terms|

* Natural language understanding
* Text corpus
* Speech corpus
* Stopwords
* Bag-of-words
* AI-complete
* n-gram (Bigram, Trigram)

Text analysis|

* Text segmentation
* Part-of-speech tagging
* Text chunking
* Compound term processing
* Collocation extraction
* Stemming
* Lemmatisation
* Named-entity recognition
* Coreference resolution
* Sentiment analysis
* Concept mining
* Parsing
* Word-sense disambiguation
* Ontology learning
* Terminology extraction
* Textual entailment
* Truecasing

Automatic summarization|

* Multi-document summarization
* Sentence extraction
* Text simplification

Machine translation|

* Computer-assisted
* Example-based
* Rule-based
* Neural

Automatic identification
and data capture|

* Speech recognition
* Speech synthesis
* Optical character recognition
* Natural language generation

Topic model|

* Pachinko allocation
* Latent Dirichlet allocation
* Latent semantic analysis

Computer-assisted
reviewing|

* Automated essay scoring
* Concordancer
* Grammar checker
* Predictive text
* Spell checker
* Syntax guessing

Natural language
user interface|

* Automated online assistant
* Chatbot
* Interactive fiction
* Question answering
* Voice user interface

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Glossary
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large
* NMT
* Reasoning
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Lethal autonomous weapons (LAWs)
* Generative artificial intelligence (GenAI)
* (Hypothetical: Artificial general intelligence (AGI))
* (Hypothetical: Artificial superintelligence (ASI))
* Agent2Agent protocol

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* GPT Image
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Seedance
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* 5.1
* 5.2
* Claude
* Gemini
* Gemini (language model)
* Gemma
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Christopher D. Manning
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Alex Krizhevsky
* Ilya Sutskever
* Oriol Vinyals
* Quoc V. Le
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* John Schulman
* Mustafa Suleyman
* Jan Leike
* Daniel Kokotajlo
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

Political|

* AI safety (Alignment)
* Ethics of AI
* EU AI Act
* Precautionary principle
* Regulation of AI
* Virtual politician

Social and economic|

* AI boom
* AI bubble
* AI literacy
* AI slop
* AI veganism
* AI winter
* Anthropomorphism
* In architecture
* In education
* In healthcare
* Chatbot psychosis
* Mental health
* In visual art

* Category

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/GloVe. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=GloVe&oldid=3961746"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template