The following outline is provided as an overview of, and topical guide to, deep learning:
Deep learning is a subfield of machine learning and artificial intelligence based on artificial neural networks with multiple processing layers. It emphasizes representation learning and is widely used in areas such as computer vision, natural language processing, speech recognition, recommender systems, robotics, and generative artificial intelligence.123
Ways to categorize deep learning
- A field of study
- A branch of artificial intelligence
- A subfield of machine learning
- A subfield of computer science
- A form of representation learning
- A class of methods based on artificial neural networks
- An approach used in computational statistics
History
Precursors
Milestones
- LeNet
- Long short-term memory
- Deep belief network
- AlexNet
- Sequence to sequence learning
- Generative adversarial network
- Residual neural network
- Transformer
- BERT
- Generative pre-trained transformer
- Diffusion model
Related histories
Core concepts
Learning settings
- Supervised learning
- Unsupervised learning
- Self-supervised learning
- Semi-supervised learning
- Reinforcement learning
- Transfer learning
- Multitask learning
- Multimodal learning
- Online machine learning
- Continual learning
Common tasks
- Image classification
- Object detection
- Image segmentation
- Automatic speech recognition
- Neural machine translation
- Question answering
- Automatic summarization
- Text-to-image model
- Protein structure prediction
Architectures
Feedforward and convolutional architectures
- Feedforward neural network
- Multilayer perceptron
- Convolutional neural network
- Radial basis function network
- Residual neural network
- U-Net
Recurrent and sequence architectures
- Recurrent neural network
- Long short-term memory
- Gated recurrent unit
- Sequence to sequence learning
- Recursive neural network
Representation-learning architectures
- Autoencoder
- Denoising autoencoder
- Sparse autoencoder
- Variational autoencoder
- Restricted Boltzmann machine
- Deep belief network
Attention and transformer architectures
Generative and probabilistic architectures
- Autoregressive model
- Diffusion model
- Energy-based model
- Generative adversarial network
- Mixture of experts
Graph and memory architectures
- Graph neural network
- Graph convolutional network
- Siamese network
- Neural Turing machine
- Memory network4
- Echo state network
- Capsule neural network
Neural network components and techniques
- Artificial neuron
- Activation function
- Embedding
- Convolution
- Pooling layer
- Attention
- Batch normalization
- Layer normalization
- Residual connections
Training and optimization
- Backpropagation
- Gradient descent
- Stochastic gradient descent
- Adam optimization5
- Learning rate
- Loss function
- Regularization
- Batch normalization
- Data augmentation
- Transfer learning
- Knowledge distillation
- Ensemble learning
- Curriculum learning
Datasets and benchmarks
- CIFAR-10
- ImageNet
- MNIST database
- Common Objects in Context (COCO)6
- General Language Understanding Evaluation (GLUE) benchmark7
- LibriSpeech89
- SQuAD1011
Applications
Computer vision
- Computer vision
- Facial recognition system
- Image classification
- Image segmentation
- Medical imaging
- Object detection
- Optical character recognition
Natural language processing
- Automatic summarization
- Chatbot
- Information retrieval
- Large language model
- Natural language processing
- Neural machine translation
- Question answering
- Sentiment analysis
Speech and audio
Science and medicine
Robotics and control
Recommendation, search, and forecasting
Generative artificial intelligence
- Deepfake
- Generative artificial intelligence
- Large language model
- Speech synthesis
- Text-to-image model
Computer graphics and video games
- Deep Learning Anti-Aliasing (DLAA)
- Deep Learning Super Sampling (DLSS)
Hardware
- AMD Instinct
- AMD XDNA
- Application-specific integrated circuit
- Deep learning processor, Neural processing unit (NPU), or Neural Engine
- Field-programmable gate array
- General-purpose computing on graphics processing units (GPGPU)
- Graphics processing unit
- NVIDIA Deep Learning Accelerator (NVDLA)
- Tensor processing unit
- Vision processing unit
- Wafer-scale integration12
Supporting software platforms
Software
Open-source frameworks and libraries
Neural network software
Platforms, tools, and deployment
Algorithms for deep learning and neural networks
- Backpropagation
- Conjugate gradient method
- Generalized Hebbian algorithm
- Gradient descent
- Levenberg–Marquardt algorithm
- Perceptron
- Quasi-Newton method
- Wake-sleep algorithm17
Methods and related topics
Representation and metric learning
Generative modeling
- Autoregressive model
- Diffusion model
- Generative adversarial network
- Generative model
- Variational inference
Efficient and scalable deep learning
Reliability, safety, and interpretability
- Adversarial machine learning
- AI alignment
- Algorithmic bias
- Catastrophic forgetting
- Differential privacy
- Explainable artificial intelligence
- Federated learning
- Hallucination (artificial intelligence)
Conferences and workshops
- Annual Meeting of the Association for Computational Linguistics
- Conference on Computer Vision and Pattern Recognition
- Conference on Neural Information Processing Systems
- International Conference on Computer Vision
- International Conference on Learning Representations
- International Conference on Machine Learning
Organizations
Research laboratories and institutions
- Allen Institute for AI
- Alberta Machine Intelligence Institute
- European Laboratory for Learning and Intelligent Systems
- Google DeepMind
- Meta AI
- Mila
- Microsoft Research
- Vector Institute
Companies
Publications
Books
- Deep Learning18 – Ian Goodfellow and Yoshua Bengio
- Neural Networks and Deep Learning19 – Michael Nielsen
- Perceptrons – Marvin Minsky and Seymour Papert
Journals
Influential persons
- Alex Graves
- Alex Krizhevsky
- Andrew Ng
- Andrej Karpathy
- Ashish Vaswani
- Christopher Bishop
- Demis Hassabis
- Fei-Fei Li
- Geoffrey Hinton
- Ian Goodfellow
- Ilya Sutskever
- John Hopfield
- Jürgen Schmidhuber
- Noam Shazeer
- Oriol Vinyals
- Paul Werbos
- Quoc V. Le
- Ruslan Salakhutdinov
- Sepp Hochreiter
- Seppo Linnainmaa
- Terry Sejnowski
- Yann LeCun
- Yoshua Bengio
See also
See also
- Artificial intelligence
- Artificial neural network
- Generative artificial intelligence
- Glossary of artificial intelligence
- Lists of open-source artificial intelligence software
- Machine learning
- Neural network software
- Outline of artificial intelligence
- Outline of computer vision
- Outline of machine learning
- Outline of robotics
References
References
- LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey (2015-05-27). "Deep learning". Nature. 521 (7553): 436–444. Bibcode:2015Natur.521..436L. doi:10.1038/nature14539. PMID 26017442.
- Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. The MIT Press. ISBN 978-0-262-03561-3.
- Schmidhuber, Jürgen (January 2015). "Deep learning in neural networks: An overview". Neural Networks. 61: 85–117. arXiv:1404.7828. Bibcode:2015NN.....61...85S. doi:10.1016/j.neunet.2014.09.003. PMID 25462637.
- Biggs, David; Nuttall, Andrew (2015). Neural Memory Networks (PDF) (Report). CS229 Final Report. Stanford University. Retrieved 17 April 2026.
- Akash Ajagekar (2021). "Adam". Cornell University Computational Optimization Open Textbook – Optimization Wiki. Retrieved 17 April 2026.
- "COCO: Common Objects in Context". COCO: Common Objects in Context. Retrieved 17 April 2026.
- "GLUE Benchmark". GLUE Benchmark. Retrieved 17 April 2026.
- "LibriSpeech ASR corpus". Open Speech and Language Resources. Retrieved 17 April 2026.
- "LibriSpeech-Long". GitHub. Google DeepMind. 2024. Retrieved 17 April 2026.
- "The Stanford Question Answering Dataset". SQuAD. Retrieved 17 April 2026.
- "Stanford Question Answering Dataset". Kaggle. Retrieved 17 April 2026.
- Moore, Samuel K. (1 January 2020). "Cerebras's Giant Chip Will Smash Deep Learning's Speed Barrier". IEEE Spectrum. Retrieved 17 April 2026.
- Li, Ming; Bi, Ziqian; Wang, Tianyang; Wen, Yizhu; Niu, Qian; Song, Xinyuan; Jiang, Zekun; Liu, Junyu; Peng, Benji; Zhang, Sen; Pan, Xuanhe; Xu, Jiawei; Wang, Jinlang; Chen, Keyu; Caitlyn Heqi Yin; Feng, Pohsun; Liu, Ming (2024-10-08). "Deep Learning and Machine Learning with GPGPU and CUDA: Unlocking the Power of Parallel Computing". arXiv:2410.05686 [cs.DC].
- "Accelerated PyTorch training on Mac". Apple Developer. Apple. Retrieved 17 April 2026.
- "GitHub - tsawler/go-metal: A high-performance deep learning library for Go that leverages Apple's Metal for GPU acceleration on Apple Silicon". GitHub. Retrieved 17 April 2026.
- "Accelerating the Machine Learning Lifecycle with MLflow". GitHub.
- Quesada, Alberto (28 October 2019). "5 algorithms to train a neural network". Neural Designer Blog. Artelnics. Retrieved 20 April 2026.
- Janishar Ali. "MIT Deep Learning Book (beautiful and flawless PDF version)". GitHub. Retrieved 17 April 2026.
- Nielsen, Michael (2015). "Neural Networks and Deep Learning". Neural Networks and Deep Learning. Determination Press. Retrieved 17 April 2026.