APSIPA Transactions on Signal and Information Processing > Vol 3 > Issue 1

A tutorial survey of architectures, algorithms, and applications for deep learning

Li Deng, Microsoft Research, USA, deng@microsoft.com
 
Suggested Citation
Li Deng (2014), "A tutorial survey of architectures, algorithms, and applications for deep learning", APSIPA Transactions on Signal and Information Processing: Vol. 3: No. 1, e2. http://dx.doi.org/10.1017/atsip.2013.9

Publication Date: 22 Jan 2014
© 2014 Li Deng
 
Subjects
 
Keywords
Deep learningAlgorithmsInformation processing
 

Share

Open Access

This is published under the terms of the Creative Commons Attribution licence.

Downloaded: 26932 times

In this article:
I. INTRODUCTION 
II. A BRIEF HISTORICAL ACCOUNT OF DEEP LEARNING 
III. THREE BROAD CLASSES OF DEEP ARCHITECTURES: AN OVERVIEW 
IV. GENERATIVE ARCHITECTURE: DEEP AUTOENCODER 
V. HYBRID ARCHITECTURE: DNN PRETRAINED WITH DBN 
VI. DISCRIMINATIVE ARCHITECTURES: DSN AND RECURRENT NETWORK 
VII. APPLICATIONS OF DEEP LEARNING TO SIGNAL AND INFORMATION PROCESSING 
VIII. SUMMARY AND DISCUSSIONS 

Abstract

In this invited paper, my overview material on the same topic as presented in the plenary overview session of APSIPA-2011 and the tutorial material presented in the same conference [1] are expanded and updated to include more recent developments in deep learning. The previous and the updated materials cover both theory and applications, and analyze its future directions. The goal of this tutorial survey is to introduce the emerging area of deep learning or hierarchical learning to the APSIPA community. Deep learning refers to a class of machine learning techniques, developed largely since 2006, where many stages of non-linear information processing in hierarchical architectures are exploited for pattern classification and for feature learning. In the more recent literature, it is also connected to representation learning, which involves a hierarchy of features or concepts where higher-level concepts are defined from lower-level ones and where the same lower-level concepts help to define higher-level ones. In this tutorial survey, a brief history of deep learning research is discussed first. Then, a classificatory scheme is developed to analyze and summarize major work reported in the recent deep learning literature. Using this scheme, I provide a taxonomy-oriented survey on the existing deep architectures and algorithms in the literature, and categorize them into three classes: generative, discriminative, and hybrid. Three representative deep architectures – deep autoencoders, deep stacking networks with their generalization to the temporal domain (recurrent networks), and deep neural networks (pretrained with deep belief networks) – one in each of the three classes, are presented in more detail. Next, selected applications of deep learning are reviewed in broad areas of signal and information processing including audio/speech, image/vision, multimodality, language modeling, natural language processing, and information retrieval. Finally, future directions of deep learning are discussed and analyzed.

DOI:10.1017/atsip.2013.9