Foundations and Trends® in Communications and Information Theory > Vol 1 > Issue 4

Information Theory and Statistics: A Tutorial

  • Csiszár, I. 1
  • Shields, P.C. 2

[1]Csiszár, I., Rényi Institute of Mathematics, Hungarian Academy of Sciences, Hungary, csiszar@renyi.hu [2]Shields, P.C., University of Toledo, USA, paul.shields@utoledo.edu

Published: 15 December 2004 © 2004 I. Csiszár and P.C. Shields

Short description

Information Theory and Statistics: A Tutorial, is an excellent introductory text to this highly-important topic in mathematics, computer science and electrical engineering. It provides both students and researchers with an invaluable resource to quickly get up to speed in the field.

Keywords

Information theory and statistics

No full text access

Table of contents

Preface
Section 1 Preliminaries
Section 2 Large deviations, hypothesis testing
Section 3 I-projections
Section 4 f-Divergence and contingency tables
Section 5 Iterative algorithms
Section 6 Universal coding
Section 7 Redundancy bounds
Section 8 Redundancy and the MDL principle
Appendix A Summary of process concepts
Historical Notes
References

Foundations and Trends® in Communications and Information Theory

(Vol 1, Issue 4, 2004, pp 417-528)

DOI: 10.1561/0100000004

Abstract

This tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The information measure known as information divergence or Kullback-Leibler distance or relative entropy plays a key role, often with a geometric flavor as an analogue of squared Euclidean distance, as in the concepts of I-projection, I-radius and I-centroid. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an "information geometry" background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory.

Buy e-Book
100.00 USD
(Full text PDF)
Buy Book
40.00 USD
(Physical copy)

Table of contents

1 Preliminaries
2 Large deviations, hypothesis testing
3 I-projections
4 f-Divergence and contingency tables
5 Iterative algorithms
6 Universal coding
7 Redundancy bounds
8 Redundancy and the MDL principle
Appendix A Summary of process concepts Historical notes
Cover image for Information Theory and Statistics

Information Theory and Statistics

124 pages

DOI: 10.1561/9781933019543

E-ISBN: 978-1-933019-54-3

ISBN: 978-1-933019-05-5

Description

Information Theory and Statistics: A Tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an "information geometry" background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory. The tutorial does not assume the reader has an in-depth knowledge of Information Theory or statistics. As such, Information Theory and Statistics: A Tutorial, is an excellent introductory text to this highly-important topic in mathematics, computer science and electrical engineering. It provides both students and researchers with an invaluable resource to quickly get up to speed in the field.