Foundations and Trends® in Communications and Information Theory > Vol 6 > Issue 1–2

Statistical Physics and Information Theory

By Neri Merhav, Department of Electrical Engineering, Technion — Israel Institute of Technology, Israel, merhav@ee.technion.ac.il

 
Suggested Citation
Neri Merhav (2010), "Statistical Physics and Information Theory", Foundations and Trends® in Communications and Information Theory: Vol. 6: No. 1–2, pp 1-212. http://dx.doi.org/10.1561/0100000052

Publication Date: 13 Dec 2010
© 2010 N. Merhav
 
Subjects
Information theory and statistics
 

Free Preview:

Download extract

Share

Download article
In this article:
1. Introduction 
2. Basic Background in Statistical Physics 
3. Physical Interpretations of Information Measures 
4. Analysis Tools and Asymptotic Methods 
5. Interacting Particles and Phase Transitions 
6. The Random Energy Model and Random Coding 
7. Extensions of the REM 
8. Summary and Outlook 
Acknowledgments 
References 

Abstract

This monograph is based on lecture notes of a graduate course, which focuses on the relations between information theory and statistical physics. The course was delivered at the Technion during the Spring of 2010 for the first time, and its target audience consists of EE graduate students in the area of communications and information theory, as well as graduate students in Physics who have basic background in information theory. Strong emphasis is given to the analogy and parallelism between information theory and statistical physics, as well as to the insights, the analysis tools and techniques that can be borrowed from statistical physics and 'imported' to certain problem areas in information theory. This is a research trend that has been very active in the last few decades, and the hope is that by exposing the students to the meeting points between these two disciplines, their background and perspective may be expanded and enhanced. This monograph is substantially revised and expanded relative to an earlier version posted in arXiv (1006.1565v1 [cs.iT]).

DOI:10.1561/0100000052
ISBN: 978-1-60198-406-7
236 pp. $99.00
Buy book (pb)
 
ISBN: 978-1-60198-407-4
236 pp. $150.00
Buy E-book (.pdf)
Table of contents:
1. Introduction
2. Basic Background in Statistical Physics
3. Physical Interpretations of Information Measures
4. Analysis Tools and Asymptotic Methods
5. Interacting Particles and Phase Transitions
6. The Random Energy Model and Random Coding
7. Extensions of the REM
8. Summary and Outlook
Acknowledgments
References

Statistical Physics and Information Theory

Statistical Physics and Information Theory focuses on some of the relationships and the interplay between information theory and statistical physics - a branch of physics that deals with many-particle systems using probabilistic and statistical methods in the microscopic level. The author interlaces the physics and the information-theoretic subjects with each other, rather than giving them in two continuous, separate parts. This makes the relations between information theory and statistical physics more apparent. He also shows that, not only are the relations between information theory and statistical physics interesting academically in their own right, but moreover, they prove useful and beneficial in that they provide information-theorists with new insights and mathematical tools to deal with information-theoretic problems. These mathematical tools sometimes prove a lot more efficient than traditional tools used in information theory, and they may give either simpler expressions for performance analysis, or improved bounds, or both. The author provides examples of the techniques and insights. One example is the use of integrals in the complex plane and the saddle-point method. Another example is the analysis technique of error exponents, which stems from the random energy model, along with its insights about phase transitions.

Statistical Physics and Information Theory highlights to the reader techniques that have been been used in one branch of science which can be applied effectively in another. The point is that it is not the physics itself that may be useful, it is the way in which physicists use mathematical tools. This will bring new insights to all students and researchers in the field of information theory and communications.

 
CIT-052