PaLM

Short description: Large language model developed by Google

PaLM

Developer(s)| Google AI
Predecessor| LaMDA
Successor| Gemini
Available in| English
Type| Large language model

PaLM (Pathways Language Model) is a 540 billion parameter transformer-based large language model developed by Google AI.[1] Researchers also trained smaller versions of PaLM, 8 and 62 billion parameter models, to test the effects of model scale.[2]

PaLM is capable of a wide range of tasks, including commonsense reasoning, arithmetic reasoning, joke explanation, code generation, and translation.[2][3][4][5] When combined with chain-of-thought prompting, PaLM achieved significantly better performance on datasets requiring reasoning of multiple steps, such as word problems and logic-based questions.[1][2]

The model was first announced in April 2022 and remained private until March 2023, when Google launched an API for PaLM and several other technologies.[6] The API was initially available to a limited number of developers who joined a waitlist before it was released to the public.[7]

Google and DeepMind developed a version of PaLM 540B called Med-PaLM that is fine-tuned on medical data and outperforms previous models on medical question answering benchmarks.[8][9] Med-PaLM was the first to obtain a passing score on U.S. medical licensing questions, and in addition to answering both multiple choice and open-ended questions accurately, it also provides reasoning and is able to evaluate its own responses.[10]

Google also extended PaLM using a vision transformer to create PaLM-E, a state-of-the-art vision-language model that can be used for robotic manipulation.[11][12] The model can perform tasks in robotics competitively without the need for retraining or fine-tuning.[13]

In May 2023, Google announced PaLM 2 at the annual Google I/O keynote.[14] PaLM 2 is reported to be a 340 billion parameter model trained on 3.6 trillion tokens.[15]

In June 2023, Google announced AudioPaLM for speech-to-speech translation, which uses the PaLM-2 architecture and initialization.[16]

## Training

PaLM is pre-trained on a high-quality corpus of 780 billion tokens that comprise various natural language tasks and use cases. This dataset includes filtered webpages, books, Wikipedia articles, news articles, source code obtained from open source repositories on GitHub, and social media conversations.[1][2] It is based on the dataset used to train Google's LaMDA model.[2] The social media conversation portion of the dataset makes up 50% of the corpus, which aids the model in its conversational capabilities.[2]

PaLM 540B was trained over two TPU v4 Pods with 3,072 TPU v4 chips in each Pod attached to 768 hosts, connected using a combination of model and data parallelism, which is the largest TPU configuration described to date.[2][17] This allowed for efficient training at scale, using 6,144 chips, and marked a record for the highest training efficiency achieved for LLMs at this scale: a hardware FLOPs utilization of 57.8%.[3]

## See also

* LaMDA, PaLM's predecessor
* Gemini, PaLM's successor

## References

1. ↑ 1.0 1.1 1.2 Narang, Sharan; Chowdhery, Aakanksha. "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance" (in en). https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html.
2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 Chowdhery, Aakanksha; Narang, Sharan; Devlin, Jacob; et al. (2022). "PaLM: Scaling Language Modeling with Pathways". arXiv:2204.02311 [cs.CL].
3. ↑ 3.0 3.1 Anadiotis, George (12 April 2022). "Google sets the bar for AI language models with PaLM". https://venturebeat.com/ai/ai-weekly-google-sets-the-bar-for-ai-language-models-with-palm/.
4. ↑ Bastian, Matthias (5 April 2022). "Google PaLM: Giant language AI can explain jokes". https://the-decoder.com/google-palm-giant-language-ai-can-explain-jokes/.
5. ↑ "Google: Why Is No One Talking About PaLM (NASDAQ:GOOG) | Seeking Alpha" (in en). 12 December 2022. https://seekingalpha.com/article/4563975-google-why-is-no-one-talking-about-palm.
6. ↑ Vincent, James (14 March 2023). "Google opens up its AI language model PaLM to challenge OpenAI and GPT-3". https://www.theverge.com/2023/3/14/23639313/google-ai-language-model-palm-api-challenge-openai.
7. ↑ Huffman, Scott; Woodward, Josh. "PaLM API & MakerSuite: an approachable way to start prototyping and building generative AI applications" (in en). https://developers.googleblog.com/2023/03/announcing-palm-api-and-makersuite.html.
8. ↑ Singhal, Karan; Azizi, Shekoofeh; Tu, Tao; et al. (2022). "Large Language Models Encode Clinical Knowledge". arXiv:2212.13138 [cs.CL].
9. ↑ "MedPaLM: New Chatbots Will Soon Be Better Than Waiting For A Doctor". 17 January 2023. https://medicalfuturist.com/medpalm-new-ai-medical-chatbots-will-soon-be-better-than-waiting-for-a-doctor/.
10. ↑ Matias, Yossi; Corrado, Greg (14 March 2023). "Our latest health AI research updates" (in en-us). https://blog.google/technology/health/ai-llm-medpalm-research-thecheckup/.
11. ↑ Driess, Danny; Xia, Fei; Sajjadi, Mehdi S. M.; et al. (2023). "PaLM-E: An Embodied Multimodal Language Model". arXiv:2303.03378 [cs.LG].
12. ↑ Driess, Danny; Florence, Pete. "PaLM-E: An embodied multimodal language model" (in en). https://ai.googleblog.com/2023/03/palm-e-embodied-multimodal-language.html.
13. ↑ Edwards, Benj (7 March 2023). "Google's PaLM-E is a generalist robot brain that takes commands" (in en-us). https://arstechnica.com/information-technology/2023/03/embodied-ai-googles-palm-e-allows-robot-control-with-natural-commands/.
14. ↑ Lardinois, Frederic (May 10, 2023). "Google launches PaLM 2, its next-gen large language model". https://techcrunch.com/2023/05/10/google-launches-palm-2-its-next-gen-large-language-model/.
15. ↑ Elias, Jennifer (16 May 2023). "Google's newest A.I. model uses nearly five times more text data for training than its predecessor". CNBC. https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html.
16. ↑ "AudioPaLM". https://google-research.github.io/seanet/audiopalm/examples/.
17. ↑ "An empirical analysis of compute-optimal large language model training" (in en). https://www.deepmind.com/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training.

* v
* t
* e

Differentiable computing

General|

* Differentiable programming
* Neural Turing machine
* Differentiable neural computer
* Automatic differentiation
* Neuromorphic engineering

Concepts|

* Gradient descent
* Cable theory
* Cluster analysis
* Regression analysis
* Pattern recognition
* Adversarial machine learning
* Computational learning theory

Programming languages|

* Python
* Julia

Application|

* Machine learning
* Artificial neural network
* Scientific computing
* Artificial Intelligence

Hardware|

* TPU
* VPU
* Memristor
* SpiNNaker

Software library|

* TensorFlow
* PyTorch

Implementation|

| Audio-visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Speech synthesis
* Speech recognition
* Facial recognition system

|

Verbal|

* Word2vec
* Transformer
* BERT
* NMT
* Project Debater
* Watson
* GPT-3

Decisional|

* AlphaGo
* Q-learning
* SARSA
* OpenAI Five

People|

* Alex Graves
* Ian Goodfellow
* Yoshua Bengio
* Geoffrey Hinton
* Yann LeCun
* Andrew Ng
* Demis Hassabis

* Portals
* Computer programming
* Technology
* Category
* Artificial neural networks
* Machine learning

* v
* t
* e

Natural language processing

General terms|

* AI-complete
* Bag-of-words
* n-gram
* Bigram
* Trigram
* Computational linguistics
* Natural-language understanding
* Stop words
* Text processing

Text analysis|

* Collocation extraction
* Concept mining
* Coreference resolution
* Deep linguistic processing
* Distant reading
* Information extraction
* Named-entity recognition
* Ontology learning
* Parsing
* Part-of-speech tagging
* Semantic role labeling
* Semantic similarity
* Sentiment analysis
* Terminology extraction
* Text mining
* Textual entailment
* Truecasing
* Word-sense disambiguation
* Word-sense induction

| Text segmentation|

* Compound-term processing
* Lemmatisation
* Lexical analysis
* Text chunking
* Stemming
* Sentence segmentation
* Word segmentation

|

Automatic summarization|

* Multi-document summarization
* Sentence extraction
* Text simplification

Machine translation|

* Computer-assisted
* Example-based
* Rule-based
* Statistical
* Transfer-based
* Neural

Distributional semantics models|

* BERT
* Document-term matrix
* Explicit semantic analysis
* fastText
* GloVe
* Latent semantic analysis
* Word embedding
* Word2vec

Language resources,
datasets and corpora|

| Types and
standards|

* Corpus linguistics
* Lexical resource
* Linguistic Linked Open Data
* Machine-readable dictionary
* Parallel text
* PropBank
* Semantic network
* Simple Knowledge Organization System
* Speech corpus
* Text corpus
* Thesaurus (information retrieval)
* Treebank
* Universal Dependencies

|

Data|

* BabelNet
* Bank of English
* DBpedia
* FrameNet
* Google Ngram Viewer
* UBY
* WordNet

Automatic identification
and data capture|

* Speech recognition
* Speech segmentation
* Speech synthesis
* Natural language generation
* Optical character recognition

Topic model|

* Document classification
* Latent Dirichlet allocation
* Pachinko allocation

Computer-assisted
reviewing|

* Automated essay scoring
* Concordancer
* Grammar checker
* Predictive text
* Spell checker
* Syntax guessing

Natural language
user interface|

* Chatbot
* Interactive fiction
* Question answering
* Virtual assistant
* Voice user interface

Other software|

* Natural Language Toolkit
* spaCy

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/PaLM. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Software:PaLM&oldid=2626960"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template