Highway network

Short description: Type of artificial neural network

This article is about a technique used in machine learning. For other uses, see Highway.

In machine learning, the Highway Network was the first working very deep feedforward neural network with hundreds of layers, much deeper than previous neural networks.[1][2][3] It uses skip connections modulated by learned gating mechanisms to regulate information flow, inspired by long short-term memory (LSTM) recurrent neural networks.[4][5] The advantage of the Highway Network over other deep learning architectures is its ability to overcome or partially prevent the vanishing gradient problem,[6] thus improving its optimization. Gating mechanisms are used to facilitate information flow across the many layers ("information highways").[1][2]

Highway Networks have found use in text sequence labeling and speech recognition tasks.[7][8]

In 2014, the state of the art was training deep neural networks with 20 to 30 layers.[9] Stacking too many layers led to a steep reduction in training accuracy,[10] known as the "degradation" problem.[11] In 2015, two techniques were developed to train such networks: the Highway Network (published in May), and the residual neural network, or ResNet[12] (December). ResNet behaves like an open-gated Highway Net.

## Contents

* 1 Model
* 2 Structure
* 3 Related work
* 4 References

## Model

The model has two gates in addition to the H(WH,x) gate: the transform gate T(WT,x) and the carry gate C(WC,x). The latter two gates are non-linear transfer functions (specifically sigmoid by convention). The function H can be any desired transfer function.

The carry gate is defined as:

C(WC,x)=1−T(WT,x)

while the transform gate is just a gate with a sigmoid transfer function.

## Structure

The structure of a hidden layer in the Highway Network follows the equation:

y=H(x,WH)⋅T(x,WT)+x⋅C(x,WC)=H(x,WH)⋅T(x,WT)+x⋅(1−T(x,WT))

## Related work

Sepp Hochreiter analyzed the vanishing gradient problem in 1991 and attributed to it the reason why deep learning did not work well.[6] To overcome this problem, Long Short-Term Memory (LSTM) recurrent neural networks[4] have residual connections with a weight of 1.0 in every LSTM cell (called the constant error carrousel) to compute yt+1=F(xt)+xt. During backpropagation through time, this becomes the residual formula y=F(x)+x for feedforward neural networks. This enables training very deep recurrent neural networks with a very long time span t. A later LSTM version published in 2000[5] modulates the identity LSTM connections by so-called "forget gates" such that their weights are not fixed to 1.0 but can be learned. In experiments, the forget gates were initialized with positive bias weights,[5] thus being opened, addressing the vanishing gradient problem. As long as the forget gates of the 2000 LSTM are open, it behaves like the 1997 LSTM.

The Highway Network of May 2015[1] applies these principles to feedforward neural networks. It was reported to be "the first very deep feedforward network with hundreds of layers".[13] It is like a 2000 LSTM with forget gates unfolded in time,[5] while the later Residual Nets have no equivalent of forget gates and are like the unfolded original 1997 LSTM.[4] If the skip connections in Highway Networks are "without gates," or if their gates are kept open (activation 1.0), they become Residual Networks.

The residual connection is a special case of the "short-cut connection" or "skip connection" by Rosenblatt (1961)[14] and Lang & Witbrock (1988)[15] which has the form x↦F(x)+Ax. Here the randomly initialized weight matrix A does not have to be the identity mapping. Every residual connection is a skip connection, but almost all skip connections are not residual connections.

The original Highway Network paper[16] not only introduced the basic principle for very deep feedforward networks, but also included experimental results with 20, 50, and 100 layers networks, and mentioned ongoing experiments with up to 900 layers. Networks with 50 or 100 layers had lower training error than their plain network counterparts, but no lower training error than their 20 layers counterpart (on the MNIST dataset, Figure 1 in [16]). No improvement on test accuracy was reported with networks deeper than 19 layers (on the CIFAR-10 dataset; Table 1 in [16]). The ResNet paper,[17] however, provided strong experimental evidence of the benefits of going deeper than 20 layers. It argued that the identity mapping without modulation is crucial and mentioned that modulation in the skip connection can still lead to vanishing signals in forward and backward propagation (Section 3 in [17]). This is also why the forget gates of the 2000 LSTM[18] were initially opened through positive bias weights: as long as the gates are open, it behaves like the 1997 LSTM. Similarly, a Highway Net whose gates are opened through strongly positive bias weights behaves like a ResNet. The skip connections used in modern neural networks (e.g., Transformers) are dominantly identity mappings.

## References

1. ↑ 1.0 1.1 1.2 Srivastava, Rupesh Kumar; Greff, Klaus; Schmidhuber, Jürgen (2 May 2015). "Highway Networks". arXiv:1505.00387 [cs.LG].
2. ↑ 2.0 2.1 Srivastava, Rupesh K; Greff, Klaus; Schmidhuber, Juergen (2015). "Training Very Deep Networks". Advances in Neural Information Processing Systems (Curran Associates, Inc.) 28: 2377–2385. http://papers.nips.cc/paper/5850-training-very-deep-networks.
3. ↑ Schmidhuber, Jürgen (2021). "The most cited neural networks all build on work done in my labs". AI Blog (IDSIA, Switzerland). https://people.idsia.ch/~juergen/most-cited-neural-nets.html.
4. ↑ 4.0 4.1 4.2 Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. https://www.researchgate.net/publication/13853244.
5. ↑ 5.0 5.1 5.2 5.3 Felix A. Gers; Jürgen Schmidhuber; Fred Cummins (2000). "Learning to Forget: Continual Prediction with LSTM". Neural Computation 12 (10): 2451–2471. doi:10.1162/089976600300015015. PMID 11032042.
6. ↑ 6.0 6.1 Hochreiter, Sepp (1991). Untersuchungen zu dynamischen neuronalen Netzen (PDF) (diploma thesis). Technical University Munich, Institute of Computer Science, advisor: J. Schmidhuber.
7. ↑ Liu, Liyuan; Shang, Jingbo; Xu, Frank F.; Ren, Xiang; Gui, Huan; Peng, Jian; Han, Jiawei (12 September 2017). "Empower Sequence Labeling with Task-Aware Neural Language Model". arXiv:1709.04109 [cs.CL].
8. ↑ Kurata, Gakuto; Ramabhadran, Bhuvana; Saon, George; Sethy, Abhinav (19 September 2017). "Language Modeling with Highway LSTM". arXiv:1709.06436 [cs.CL].
9. ↑ Simonyan, Karen; Zisserman, Andrew (2015-04-10), Very Deep Convolutional Networks for Large-Scale Image Recognition
10. ↑ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification". arXiv:1502.01852 [cs.CV].
11. ↑ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (10 Dec 2015). "Deep Residual Learning for Image Recognition".
12. ↑ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016). "Deep Residual Learning for Image Recognition". 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. pp. 770–778. doi:10.1109/CVPR.2016.90. ISBN 978-1-4673-8851-1.
13. ↑ Schmidhuber, Jürgen (2015). "Highway Networks (May 2015): First Working Really Deep Feedforward Neural Networks With Over 100 Layers". https://people.idsia.ch/~juergen/highway-networks.html.
14. ↑ Rosenblatt, Frank (1961). Principles of neurodynamics. perceptrons and the theory of brain mechanisms. https://safari.ethz.ch/digitaltechnik/spring2018/lib/exe/fetch.php?media=neurodynamics1962rosenblatt.pdf#page=327.
15. ↑ Lang, Kevin; Witbrock, Michael (1988). "Learning to tell two spirals apart". Proceedings of the 1988 Connectionist Models Summer School: 52–59. https://gwern.net/doc/ai/nn/fully-connected/1988-lang.pdf.
16. ↑ 16.0 16.1 16.2 Srivastava, Rupesh Kumar; Greff, Klaus; Schmidhuber, Jürgen (3 May 2015). "Highway Networks". arXiv:1505.00387 [cs.LG].
17. ↑ 17.0 17.1 He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Identity Mappings in Deep Residual Networks". arXiv:1603.05027 [cs.CV].
18. ↑ Felix A. Gers; Jürgen Schmidhuber; Fred Cummins (2000). "Learning to Forget: Continual Prediction with LSTM". Neural Computation 12 (10): 2451–2471. doi:10.1162/089976600300015015. PMID 11032042.

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Glossary
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large
* NMT
* Reasoning
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Lethal autonomous weapons (LAWs)
* Generative artificial intelligence (GenAI)
* (Hypothetical: Artificial general intelligence (AGI))
* (Hypothetical: Artificial superintelligence (ASI))
* Agent2Agent protocol

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* GPT Image
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Seedance
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* 5.1
* 5.2
* Claude
* Gemini
* Gemini (language model)
* Gemma
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Christopher D. Manning
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Alex Krizhevsky
* Ilya Sutskever
* Oriol Vinyals
* Quoc V. Le
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* John Schulman
* Mustafa Suleyman
* Jan Leike
* Daniel Kokotajlo
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

Political|

* AI safety (Alignment)
* Ethics of AI
* EU AI Act
* Precautionary principle
* Regulation of AI
* Virtual politician

Social and economic|

* AI boom
* AI bubble
* AI literacy
* AI slop
* AI veganism
* AI winter
* Anthropomorphism
* In architecture
* In education
* In healthcare
* Chatbot psychosis
* Mental health
* In visual art

* Category

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Highway network. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Highway_network&oldid=4151870"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template