Information Theory

1
Information Theory
Claude Shannon

• John A. Cagle, Ph.D.
• Communication
• California State University, Fresno

2
(No Transcript)
3
(No Transcript)
4
Shannon Weaver (1947)
Norbert Weiners Cybernetics added the notion of
feedback to this communication model.
5
Information Theory

• In a perfect communication system, the sender and
  receiver have identical knowledge of the code.
• All possible messages are known in advance.
• The source makes a choice to send a message from
  the set of possible messages.
• The receiver needs to know what choice the sender
  made.

6
Information is not content

• A message has information if it reduces the
  uncertainty about what choice the sender made.
• If the choice is already known to the receiver,
  the message is redundant.
• Information is not content in information theory.

7
Information is not meaning

• Information is different from the content and
  meaning of messages.
• Information is not the interpretation of
  information.
• Peter Drucker wrote of the difference between
  informating and communicating in an organization.

8
Entropy

• Entropy is randomness, chaos, the lack of
  organization and predictability.
• Entropy is uncertainty.
• Information reduces entropy in a communication
  system.
• Entropy is variable in most situations.

9
Measurement of Information

• The smallest unit of information is a bit
• Eight bits one byte
• Four bytes one word
• These terms are still at the core of computer
  science
• E.g., 32-bit word processors in the CPU

10
Measurement of Information

• I - log2 piis the formula for measuring the
  information value of each message sent against
  the probability of that message in the field of
  all the messages that could be sent.

11
Measurement of Information

• H - ? pi log2 piis the formula for measuring
  the amount of information of all the messages
  that could be sent in a communication system.

12
Choices

• Signal we make choices about which signal to
  send (sounds, letters, etc.)
• Semantics we make choices in a given situation
  about which meaning to send.
• Lexical choice
• Meaning
• Pragmatics we make choices in a given situation
  about which behaviors to enact.

13
Communication behavior is a specific form of
molar behavior _____1_____ occurs in a situation
or field possessing specified properties,
____2_____ part of which are in interdependent
relationship with each _____3_____. A theory of
such behavior is concerned with forces,
_____4_____, social, and physical, which
determine the course of this _____5_____ and its
outcomes in relation to the culture in
_____6_____ it occurs.
14
(No Transcript)
15
Information theory analysis
choices freq rel freq I
one 10 0.33 1.584963 -0.53
two 6 0.20 2.321928 -0.46
three 3 0.10 3.321928 -0.33
four 3 0.10 3.321928 -0.33
five 1 0.03 4.906891 -0.16
six 1 0.03 4.906891 -0.16
seven 1 0.03 4.906891 -0.16
eight 1 0.03 4.906891 -0.16
nine 1 0.03 4.906891 -0.16
ten 1 0.03 4.906891 -0.16
eleven 1 0.03 4.906891 -0.16
twelve 1 0.03 4.906891 -0.16
30
1.00 H 2.97
16
Ring a-round the roses, A pocket full of
posies, _____1_____! Ashes! We all fall
down! Three blind mice, See how they run! They
all _____2_____ after a farmer's wife, Who cut
off their tails _____3_____ a carving knife. Did
you ever see such a ____4______ in your life, As
three blind mice?
17
Brian está en el aeropuerto de Barajas en Madrid.
_____1_____ y otros estudiantes del drupo
esperan la llegada del _____2_____ para ir a
Leób. Deben esperar una hora. Qué ____3______
hacer?
1 Alice 2 vuelo 3 deciden
18
Samuel Becker (1968)
We construct messages which "are, in effect,
overlayed to form the large and complex
communication environment or 'mosaic' in which
each of us exists. This mosaic consists of an
immense number of fragments or bits of
information on an immense number of topics. . . .
These bits are scattered over time and space and
modes of communication. Each individual must
grasp from this mosaic those bits which serve his
needs, must group them into message sets which
are relevant for him at any given time, and
within each message set must organize the bits
and close the gaps between them in order to
arrive at a coherent picture of the world to
which he can respond."
19
(No Transcript)