Entropy, Relative Entropy and Mutual Information

• 5 October 2001
• book chapter
• Published by Wiley
• p. 12-49
• https://doi.org/10.1002/0471200611.ch2

Abstract

Keywords

• ENTROPY
• JOINT ENTROPY
• CONDITIONAL ENTROPY
• RELATIVE ENTROPY
• MUTUAL INFORMATION
• CHAIN RULES
• JENSEN'S INEQUALITY THEOREM
• SECOND LAW OF THERMODYNAMICS
• DATA PROCESSING INEQUALITY
• FANO'S INEQUALITY
• PROBLEMS