ECE 101 An Introduction to Information Technology Information Theory

1
ECE 101 An Introduction to Information
TechnologyInformation Theory
2
Information Path
Source of Information
Information Display
Digital Sensor
Information Receiver and Processor
Information Processor Transmitter
Transmission Medium
3
Information Theory

• Source generates information by producing data
  units called symbols
• Measurement of information present
• measure randomness (value of information)
• do this mathematically using probability
• amount of information present is measure of
  entropy

4
Probability

• Study of random outcomes
• The experiment
• The outcome
• PXi probability of an a particular outcome
  (Xi)
• 0 lt PXi lt 1
• where N number of different outcomes

5
Measuring Information

• Symbol - data units of information
• Entropy
• average amount of energy that a source produces,
  measured in bits/symbol

6
Logarithms Base 2

• In information theory we need logs to the base 2,
  not 10 (log10 N x or 10x N) (logs are
  exponents)
• log2 N x or 2x N
• 20 1 log2 1 0
• 21 2 log2 2 1
• 22 4 log2 4 2
• 23 8 log2 8 3
• 24 16 log2 16 4
• 25 32 log2 32 5

7
Logarithms Base a then a2

• Conversion of bases in general
• loga N x or ax N
• So log2 N x or 2x N
• loga N (log10 N)/ (log10 a)
• If a 2, then use log10 2 .301
• log2 N 3.32 (log10 N)
• loga MN (loga M) (loga N)
• loga M/N (loga M) - (loga N)
• loga Nm m(loga N)

8
Measuring Information

• Symbol - data units of information
• Entropy
• average amount of energy that a source produces,
  measured in bits/symbol

9
Effective Probability and Entropy

• Measurement of entropy when probability is not
  known
• estimate probability when it is not known
• effective probability PeXi NXi/N

10
Simulating Randomness by Computer

• Information is an unexpected quality
• Model it an an experiment that produces random
  outcomes
• Common method pseudo-random number generator
  (PRNG)
• PRNG uses Modular Arithmetic

11
Modular Arithmetic

• Bmod(N) modulo-N value of integer B
• Divide B by N B/N I R/N
• where I is integer quotient and R is remainder
• 0 ? R ? (N-1)
• Bmod(N) R B - (I ? N)
• or R (B/N - I) ? N, where B/N I.xxx

12
Pseudo-Random Number Generator

• Create a random number from a sequence X1, X 2,
  X3 , , Xn, where Xn is the nth integer in
  the sequence
• Find Xn A ? Xn-1 Bmod(N) where
• A is an arbitrary multiplier of Xn-1
• N is the base of the modulus
• B prevents the sequence from degenerating into a
  set of zeroes
• to get started we need an arbitrary X0, or seed

13
Arbitrary Range for Pseudo-Random Numbers

• Desire range other than an integer number then