Differentiable programming

Short description: Programming paradigm

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

Differentiable programming is a programming paradigm in which a numeric computer program can be differentiated throughout via automatic differentiation.[1][2][3][4][5] This allows for gradient-based optimization of parameters in the program, often via gradient descent, as well as other learning approaches that are based on higher-order derivative information. Differentiable programming has found use in a wide variety of areas, particularly scientific computing and machine learning.[5] One of the early proposals to adopt such a framework in a systematic fashion to improve upon learning algorithms was made by the Advanced Concepts Team at the European Space Agency in early 2016.[6]

## Contents

* 1 Approaches
* 2 Applications
* 3 Multidisciplinary application
* 4 See also
* 5 Notes
* 6 References

## Approaches

Most differentiable programming frameworks work by constructing a graph containing the control flow and data structures in the program.[7] Attempts generally fall into two groups:

* Static, compiled graph-based approaches such as TensorFlow,[note 1] Theano, and MXNet. They tend to allow for good compiler optimization and easier scaling to large systems, but their static nature limits interactivity and the types of programs that can be created easily (e.g. those involving loops or recursion), as well as making it harder for users to reason effectively about their programs.[7] A proof-of-concept compiler toolchain called Myia uses a subset of Python as a front end and supports higher-order functions, recursion, and higher-order derivatives.[8][9][10]
* Operator overloading, dynamic graph-based approaches such as PyTorch, NumPy's autograd package, and Pyaudi. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to interpreter overhead (particularly when composing many small operations), poorer scalability, and reduced benefit from compiler optimization.[9][10]

The use of just-in-time compilation has emerged recently[when?] as a possible solution to overcome some of the bottlenecks of interpreted languages. The C++ heyoka and Python package heyoka.py make large use of this technique to offer advanced differentiable programming capabilities (also at high orders). A package for the Julia programming language - Zygote.jl - works directly on Julia's intermediate representation (IR).[7][11][5]

A limitation of earlier approaches is that they are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs. Newer approaches resolve this issue by constructing the graph from the language's syntax or IR, allowing arbitrary code to be differentiated.[7][9]

## Applications

Differentiable programming has been applied in areas such as combining deep learning with physics engines in robotics,[12] solving electronic-structure problems with differentiable density functional theory,[13] differentiable ray tracing,[14] differentiable imaging,[15][16] image processing,[17] and probabilistic programming.[5]

## Multidisciplinary application

Differentiable programming is making significant strides in various fields beyond its traditional applications. In healthcare and life sciences, for example, it is being used for deep learning in biophysics-based modelling of molecular mechanisms, in areas such as protein structure prediction and drug discovery. These applications demonstrate the potential of differentiable programming in contributing to significant advancements in understanding complex biological systems and improving healthcare solutions.[18]

## See also

* Differentiable function
* Machine learning

## Notes

1. ↑ TensorFlow 1 uses the static graph approach, whereas TensorFlow 2 uses the dynamic graph approach by default.

## References

1. ↑ Izzo, Dario; Biscani, Francesco; Mereta, Alessio (2017). "Differentiable Genetic Programming". Genetic Programming. Lecture Notes in Computer Science. 10196. pp. 35–51. doi:10.1007/978-3-319-55696-3_3. ISBN 978-3-319-55695-6.
2. ↑ Baydin, Atilim Gunes; Pearlmutter, Barak A.; Radul, Alexey Andreyevich; Siskind, Jeffrey Mark (2018). "Automatic Differentiation in Machine Learning: a Survey". Journal of Machine Learning Research 18 (153): 1–43. https://jmlr.org/papers/v18/17-468.html.
3. ↑ Wang, Fei; Decker, James; Wu, Xilun; Essertel, Gregory; Rompf, Tiark (2018). "Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming". in Bengio, S.; Wallach, H.; Larochelle, H. et al.. NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates. pp. 10201–10212. https://papers.nips.cc/paper/8221-backpropagation-with-callbacks-foundations-for-efficient-and-expressive-differentiable-programming.pdf.
4. ↑ Innes, Mike (2018). "On Machine Learning and Programming Languages". SysML Conference 2018. http://www.sysml.cc/doc/2018/37.pdf. Retrieved 2019-07-04.
5. ↑ 5.0 5.1 5.2 5.3 Innes, Mike; Edelman, Alan; Fischer, Keno; Rackauckas, Chris; Saba, Elliot; Viral B Shah; Tebbutt, Will (2019). "A Differentiable Programming System to Bridge Machine Learning and Scientific Computing". arXiv:1907.07587 [cs.PL].
6. ↑ "Differential Intelligence". October 2016. https://www.esa.int/gsp/ACT/projects/differential_intelligence/.
7. ↑ 7.0 7.1 7.2 7.3 Innes, Michael; Saba, Elliot; Fischer, Keno; Gandhi, Dhairya; Marco Concetto Rudilosso; Neethu Mariya Joy; Karmali, Tejan; Pal, Avik; Shah, Viral (2018). "Fashionable Modelling with Flux". arXiv:1811.01457 [cs.PL].
8. ↑ Merriënboer, Bart van; Breuleux, Olivier; Bergeron, Arnaud; Lamblin, Pascal (3 December 2018). "Automatic differentiation in ML: where we are and where we should be going". NIPS'18. 31. pp. 8771–8781. https://papers.nips.cc/paper/2018/hash/770f8e448d07586afbf77bb59f698587-Abstract.html.
9. ↑ 9.0 9.1 9.2 Breuleux, O.; van Merriënboer, B. (2017). "Automatic Differentiation in Myia". https://www.sysml.cc/doc/2018/39.pdf.
10. ↑ 10.0 10.1 "TensorFlow: Static Graphs". Tutorials: Learning PyTorch. PyTorch.org. https://pytorch.org/tutorials/beginner/examples_autograd/tf_two_layer_net.html.
11. ↑ Innes, Michael (2018). "Don't Unroll Adjoint: Differentiating SSA-Form Programs". arXiv:1810.07951 [cs.PL].
12. ↑ Degrave, Jonas; Hermans, Michiel; Dambre, Joni; wyffels, Francis (2016). "A Differentiable Physics Engine for Deep Learning in Robotics". arXiv:1611.01652 [cs.NE].
13. ↑ Li, Li; Hoyer, Stephan; Pederson, Ryan; Sun, Ruoxi; Cubuk, Ekin D.; Riley, Patrick; Burke, Kieron (2021). "Kohn-Sham Equations as Regularizer: Building Prior Knowledge into Machine-Learned Physics". Physical Review Letters 126 (3). doi:10.1103/PhysRevLett.126.036401. PMID 33543980. Bibcode: 2021PhRvL.126c6401L.
14. ↑ Li, Tzu-Mao; Aittala, Miika; Durand, Frédo; Lehtinen, Jaakko (2018). "Differentiable Monte Carlo Ray Tracing through Edge Sampling". ACM Transactions on Graphics 37 (6): 222:1–11. doi:10.1145/3272127.3275109. https://people.csail.mit.edu/tzumao/diffrt/.
15. ↑ Chen, Ni; Cao, Liangcai; Poon, Ting-Chung; Lee, Byoungho; Lam, Edmund Y. (2023). "Differentiable Imaging: A New Tool for Computational Optical Imaging" (in en). Advanced Physics Research 2 (6). doi:10.1002/apxr.202200118. ISSN 2751-1200.
16. ↑ Chen, Ni; Brady, David J.; Lam, Edmund Y. (2025-07-04). "Differentiable Imaging: Progress, Challenges, and Outlook" (in en). Advanced Devices & Instrumentation 6. doi:10.34133/adi.0117. ISSN 2767-9713. Bibcode: 2025AdDI....6..117C.
17. ↑ Li, Tzu-Mao; Gharbi, Michaël; Adams, Andrew; Durand, Frédo; Ragan-Kelley, Jonathan (August 2018). "Differentiable Programming for Image Processing and Deep Learning in Halide". ACM Transactions on Graphics 37 (4): 139:1–13. doi:10.1145/3197517.3201383. https://cseweb.ucsd.edu/~tzli/gradient_halide.
18. ↑ AlQuraishi, Mohammed; Sorger, Peter K. (October 2021). "Differentiable biology: using deep learning for biophysics-based and data-driven modeling of molecular mechanisms". Nature Methods 18 (10): 1169–1180. doi:10.1038/s41592-021-01283-4. PMID 34608321.

* v
* t
* e

Differentiable computing

General|

* Differentiable programming
* Neural Turing machine
* Differentiable neural computer
* Automatic differentiation
* Neuromorphic engineering

Concepts|

* Gradient descent
* Cable theory
* Cluster analysis
* Regression analysis
* Pattern recognition
* Adversarial machine learning
* Computational learning theory

Programming languages|

* Python
* Julia

Application|

* Machine learning
* Artificial neural network
* Scientific computing
* Artificial Intelligence

Hardware|

* TPU
* VPU
* Memristor
* SpiNNaker

Software library|

* TensorFlow
* PyTorch

Implementation|

| Audio-visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Speech synthesis
* Speech recognition
* Facial recognition system

|

Verbal|

* Word2vec
* Transformer
* BERT
* NMT
* Project Debater
* Watson
* GPT-3

Decisional|

* AlphaGo
* Q-learning
* SARSA
* OpenAI Five

People|

* Alex Graves
* Ian Goodfellow
* Yoshua Bengio
* Geoffrey Hinton
* Yann LeCun
* Andrew Ng
* Demis Hassabis

* Portals
* Computer programming
* Technology
* Category
* Artificial neural networks
* Machine learning

* v
* t
* e

Programming paradigms

Imperative|

| Structured|

* Jackson structures
* Block-structured
* Modular
* Non-structured
* Procedural
* Programming in the large and in the small
* Design by contract
* Invariant-based
* Nested function

|

Object-oriented|

* Class-based, Prototype-based, Object-based
* Agent
* Immutable object
* Persistent
* Uniform function call syntax

Declarative|

| Functional|

* Recursive
* Anonymous function (Partial application)
* Higher-order
* Purely functional
* Total
* Strict
* GADTs
* Dependent types
* Functional logic
* Point-free style
* Expression-oriented
* Applicative, Concatenative
* Function-level, Value-level
* Monad

|

Dataflow|

* Flow-based
* Reactive (Functional reactive)
* Signals
* Streams
* Synchronous

Logic|

* Abductive logic
* Answer set
* Constraint (Constraint logic)
* Inductive logic
* Nondeterministic
* Ontology
* Probabilistic logic
* Query

Domain-
specific
language
(DSL)|

* Algebraic modeling
* Array
* Automata-based (Action)
* Command (Spacecraft)
* Differentiable
* End-user
* Grammar-oriented
* Interface description
* Language-oriented
* List comprehension
* Low-code
* Modeling
* Natural language
* Non-English-based
* Page description
* Pipes and filters
* Probabilistic
* Quantum
* Scientific
* Scripting
* Set-theoretic
* Simulation
* Stack-based
* System
* Tactile
* Templating
* Transformation (Graph rewriting, Production, Pattern)
* Visual

Concurrent,
parallel|

* Actor-based
* Automatic mutual exclusion
* Choreographic programming
* Concurrent logic (Concurrent constraint logic)
* Concurrent OO
* Macroprogramming
* Multitier programming
* Organic computing
* Parallel programming models
* Partitioned global address space
* Process-oriented
* Relativistic programming
* Service-oriented
* Structured concurrency

Metaprogramming|

* Attribute-oriented
* Automatic (Inductive)
* Dynamic
* Extensible
* Generic
* Homoiconicity
* Interactive
* Macro (Hygienic)
* Metalinguistic abstraction
* Multi-stage
* Program synthesis (Bayesian, by demonstration, by example, vibe coding)
* Reflective
* Self-modifying code
* Symbolic
* Template

Separation
of concerns|

* Aspects
* Components
* Data-driven
* Data-oriented
* Event-driven
* Features
* Literate
* Roles
* Subjects

Comparisons/Lists|

Comparison (multi-paradigm, object-oriented, functional), List (OO, by type)

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Differentiable programming. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Differentiable_programming&oldid=4569521"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template