Diapositive 1

1
Information and Entropy
2
Shannon information entropy on discrete variables

• Consider W discrete events with probabilities pi
  such that ?i1W pi1.
• Shannons(1) measure of the amount of choice for
  the pi is
• H -k ?i1W pilog pi , where k is a positive
  constant
• If pi 1/W and kBoltzmanns constant, then H
  -k W/W log 1/W k log W, which is the entropy of
  a system with W microscopic configurations
• Hence (using k1), H-?i1M pilog pi is the
  Shannons information entropy
• Example
• Schneider(2) notes that H is a measure of
  entropy/disorder/incertitude. It is a measure of
  information in Shannons sense only if
  considering it as the information gained by
  complete incertitude removal (i.e. noiseless
  channel)
• (1) C. E. Shannon. A mathematical theory of
  communication. Bell Sys. Tech. J., 1948
• (2) T. D. Schneider, Information Theory Primer,
  last updated Jan 6, 2003

Second law of thermodynamic The entropy of a
system increases until it reaches equilibrium
within the constraints imposed on it.
3
Information Entropy on continuous variables

• Information about a random variable xmap taking
  continuous values arises from the exclusion its
  possible alternatives (realizations)
• Hence a measure of information for continuous
  valued xmap is
• Info(xmap) -log f (xmap)
• The expected information is then H(xmap)
  -?dcmap f (cmap) log f (cmap)
• By noting the similarity with H-?i1M pilog pi
  for discrete variables, we see that
• H(xmap) -?dcmap f (cmap) log f (cmap) is
  Shannons information entropy associated with the
  PDF f (cmap) for continuous variables xmap

4
Maximizing entropy given knowledge constraints

• Example 1 Given knowledge that two blue toys
  are in the corner of a room, consider the
  following two arrangements
• Example 2 Given knowledge that the PDF has mean
  m0 and variance s21, consider the following
  uniform and Gaussian PDFs
• Hence, the prior stage of BME aims at
  informativeness by using all but no more general
  knowledge than is available, i.e. we will seek to
  maximize information entropy given constraints
  expressing general knowledge.

Out of these two arrangements, arrangement (a)
maximizes entropy given the knowledge constraint,
hence given our knowledge, it is the most likely
toy arrangement (would kids produce (b)?)

• (b)

• (a)

Out of these two PDFs, the Gaussian PDF maximizes
information entropy given the knowledge
constraint that m0 and s21
Uniform s21, H 1.24 Gaussian s21, H 1.42