Information Theory – USU ECE7680

This 2006 Utah State University OpenCourseWare graduate course explores the fundamental limits of the representation and transmission of information. It focuses on the definition and implications of (information) entropy, the source coding theorem, and the channel coding theorem. These concepts provide a vital background for researchers in the areas of data compression, signal processing, controls, and pattern recognition.

This class is highly mathematical. The direct applications are, in a sense, only recently emerging, despite the nearly 50 year history. A firm determination and a fair degree of mathematical maturity will be required by students hoping to do well in the class. After all, how many engineering classes have you had which have the word "theory" in the title?

Course Organization

This course constitutes 14 lectures, originally offered twice per week. HTML and PDF versions of lecture notes are available for all of the lectures. No audio or video is associated with this course. Some sessions have a homework assignment, including reading, associated with them. The homework actually comes from the Spring 2000 class and not the 2006 class. Since the 2000 class had a different lecture layout, best judgement was made placing the homework assignments into the 2006 layout seen here.

A final class project is also associated with this course and described in the last session.

Further Reading

The following materials are listed as textbooks and supplementary reading for the course. The OCW often but not always makes it clear what chapters from which books were assigned for each session. In cases where it's not clear, use your own judgement in selecting chapters/sections for each session.


Cover, Thomas M. and Thomas, Joy A. Elements of Information Theory. 2nd edition. John Wiley & Sons, 2012.

Mackay, David J.C. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2004.

Supplementary reading

Lucky, Robert W. Silicon Dreams: Information, Man, and Machine. St. Martin's Press, 1991.

Moon, Todd K. and Stirling, Wynn C. Mathematical Methods and Algorithms for Signal Processing. Prentice Hall, 2000.

Proakis, John G. and Salehi, Masoud. Communications Systems Engineering. Prentice Hall, 1994.

You must be logged in to post a comment Login