APSIPA Transactions on Signal and Information Processing > Vol 5 > Issue 1

Deep neural networks – a developmental perspective

Biing Hwang Juang, Georgia Institute of Technology, USA, juang@gatech.edu
 
Suggested Citation
Biing Hwang Juang (2016), "Deep neural networks – a developmental perspective", APSIPA Transactions on Signal and Information Processing: Vol. 5: No. 1, e7. http://dx.doi.org/10.1017/ATSIP.2016.9

Publication Date: 01 Apr 2016
© 2016 Biing Hwang Juang
 
Subjects
 
Keywords
Deep neural networksPattern recognitionMachine learningRestrictive Boltzmann machineFeed-forward networks
 

Share

Open Access

This is published under the terms of the Creative Commons Attribution licence.

Downloaded: 1707 times

In this article:
I. INTRODUCTION 
II. STATISTICAL PATTERN RECOGNITION 
III. ARTIFICIAL NEURAL NETWORKS 
IV. MULTIVARIATE DISTRIBUTION AND MRF 
V. WHAT DOES AN RBM DO? 
VI. LATENT VARIABLE MODEL AND MIXTURE MODEL 
VII. DEEP BELIEF NETWORK 
VIII. MANIFOLDS AND PROBABILITY SPACE 
IX. WHAT SOME EXPERIMENTAL RESULTS MAY TELL US 
X. OVERALL REMARKS & SUMMARY 

Abstract

There is a recent surge in research activities around “deep neural networks” (DNN). While the notion of neural networks have enjoyed cycles of enthusiasm, which may continue its ebb and flow, concrete advances now abound. Significant performance improvements have been shown in a number of pattern recognition tasks. As a technical topic, DNN is important in classes and tutorial articles and related learning resources are available. Streams of questions, nonetheless, never subside from students or researchers and there appears to be a frustrating tendency among the learners to treat DNN simply as a black box. This is an awkward and alarming situation in education. This paper thus has the intent to help the reader to properly understand DNN, not just its mechanism (what and how) but its motivation and justification (why). It is written from a developmental perspective with a comprehensive view, from the very basic but oft-forgotten principle of statistical pattern recognition and decision theory, through the problem stages that may be encountered during system design, to key ideas that led to the new advance. This paper can serve as a learning guide with historical reviews and important references, helpful in reaching an insightful understanding of the subject.

DOI:10.1017/ATSIP.2016.9