Lecture 2 Probability Review and Random Process

1
Lecture 2Probability Review and Random Process
2
Review of last lecture

• The point worth noting are
• The source coding algorithm plays an important
  role in higher code rate (compressing data)
• The channel encoder introduce redundancy in data
• The modulation scheme plays important role in
  deciding the data rate and immunity of signal
  towards the errors introduced by the channel
• Channel can introduce many types of errors due to
  thermal noise etc.
• The demodulator and decoder should provide high
  Bit Error Rate (BER).

3
ReviewLayering of Source Coding

• Source coding includes
• Sampling
• Quantization
• Symbols to bits
• Compression
• Decoding includes
• Decompression
• Bits to symbols
• Symbols to sequence of numbers
• Sequence to waveform (Reconstruction)

4
ReviewLayering of Source Coding
5
ReviewLayering of Channel Coding

• Channel Coding is divided into
• Discrete encoder\Decoder
• Used to correct channel Errors
• Modulation\Demodulation
• Used to map bits to waveform for transmission

6
ReviewLayering of Channel Coding
7
ReviewResources of a Communication System

• Transmitted Power
• Average power of the transmitted signal
• Bandwidth (spectrum)
• Band of frequencies allocated for the signal
• Type of Communication system
• Power limited System
• Space communication links
• Band limited Systems
• Telephone systems

8
ReviewDigital communication system

• Important features of a DCS
• Transmitter sends a waveform from a finite set of
  possible waveforms during a limited time
• Channel distorts, attenuates the transmitted
  signal and adds noise to it.
• Receiver decides which waveform was transmitted
  from the noisy received signal
• Probability of erroneous decision is an important
  measure for the system performance

9
Review of Probability
10
Sample Space and Probability

• Random experiment its outcome, for some reason,
  cannot be predicted with certainty.
• Examples throwing a die, flipping a coin and
  drawing a card from a deck.
• Sample space the set of all possible outcomes,
  denoted by S. Outcomes are denoted by Es and
  each E lies in S, i.e., E ? S.
• A sample space can be discrete or continuous.
• Events are subsets of the sample space for which
  measures of their occurrences, called
  probabilities, can be defined or determined.

11
Three Axioms of Probability

• For a discrete sample space S, define a
  probability measure P on as a set function that
  assigns nonnegative values to all events, denoted
  by E, in such that the following conditions are
  satisfied
• Axiom 1 0 P(E) 1 for all E ? S
• Axiom 2 P(S) 1 (when an experiment is
  conducted there has to be an outcome).
• Axiom 3 For mutually exclusive events E1, E2,
  E3,. . . we have

12
Conditional Probability

• We observe or are told that event E1 has occurred
  but are actually interested in event E2
  Knowledge that of E1 has occurred changes the
  probability of E2 occurring.
• If it was P(E2) before, it now becomes P(E2E1),
  the probability of E2 occurring given that event
  E1 has occurred.
• This conditional probability is given by
• If P(E2E1) P(E2), or P(E2 n E1) P(E1)P(E2),
  then E1 and E2 are said to be statistically
  independent.
• Bayes rule
• P(E2E1) P(E1E2)P(E2)/P(E1)

13
Mathematical Model for Signals

• Mathematical models for representing signals
• Deterministic
• Stochastic
• Deterministic signal No uncertainty with respect
  to the signal value at any time.
• Deterministic signals or waveforms are modeled by
  explicit mathematical expressions, such as
• x(t) 5 cos(10t).
• Inappropriate for real-world problems???
• Stochastic/Random signal Some degree of
  uncertainty in signal values before it actually
  occurs.
• For a random waveform it is not possible to write
  such an explicit expression.
• Random waveform/ random process, may exhibit
  certain regularities that can be described in
  terms of probabilities and statistical averages.
• e.g. thermal noise in electronic circuits due to
  the random movement of electrons

14
Energy and Power Signals

• The performance of a communication system depends
  on the received signal energy higher energy
  signals are detected more reliably (with fewer
  errors) than are lower energy signals.
• An electrical signal can be represented as a
  voltage v(t) or a current i(t) with instantaneous
  power p(t) across a resistor defined by
• OR

15
Energy and Power Signals

• In communication systems, power is often
  normalized by assuming R to be 1.
• The normalization convention allows us to express
  the instantaneous power as
• where x(t) is either a voltage or a current
  signal.
• The energy dissipated during the time interval
  (-T/2, T/2) by a real signal with instantaneous
  power expressed by Equation (1.4) can then be
  written as
• The average power dissipated by the signal during
  the interval is

16
Energy and Power Signals

• We classify x(t) as an energy signal if, and only
  if, it has nonzero but finite energy (0 lt Ex lt 8)
  for all time, where
• An energy signal has finite energy but zero
  average power
• Signals that are both deterministic and
  non-periodic are termed as Energy Signals

17
Energy and Power Signals

• Power is the rate at which the energy is
  delivered
• We classify x(t) as an power signal if, and only
  if, it has nonzero but finite energy (0 lt Px lt 8)
  for all time, where
• A power signal has finite power but infinite
  energy
• Signals that are random or periodic termed as
  Power Signals

18
Random Variable

• Functions whose domain is a sample space and
  whose range is a some set of real numbers is
  called random variables.
• Type of RVs
• Discrete
• E.g. outcomes of flipping a coin etc
• Continuous
• E.g. amplitude of a noise voltage at a particular
  instant of time

19
Random Variables

• Random Variables
• All useful signals are random, i.e. the receiver
  does not know a priori what wave form is going to
  be sent by the transmitter
• Let a random variable X(A) represent the
  functional relationship between a random event A
  and a real number.
• The distribution function Fx(x) of the random
  variable X is given by

20
Random Variable

• A random variable is a mapping from the sample
  space to the set of real numbers.
• We shall denote random variables by boldface,
  i.e., x, y, etc., while individual or specific
  values of the mapping x are denoted by x(w).

21
Random process

• A random process is a collection of time
  functions, or signals, corresponding to various
  outcomes of a random experiment. For each
  outcome, there exists a deterministic function,
  which is called a sample function or a
  realization.

Random variables
Sample functions or realizations (deterministic
function)
22
Random Process

• A mapping from a sample space to a set of time
  functions.

23
Random Process contd

• Ensemble The set of possible time functions that
  one sees.
• Denote this set by x(t), where the time functions
  x1(t, w1), x2(t, w2), x3(t, w3), . . . are
  specific members of the ensemble.
• At any time instant, t tk, we have random
  variable x(tk).
• At any two time instants, say t1 and t2, we have
  two different random variables x(t1) and x(t2).
• Any realationship b/w any two random variables is
  called Joint PDF

24
Classification of Random Processes

• Based on whether its statistics change with time
  the process is non-stationary or stationary.
• Different levels of stationary
• Strictly stationary the joint pdf of any order
  is independent of a shift in time.
• Nth-order stationary the joint pdf does not
  depend on the time shift, but depends on time
  spacing

25
Cumulative Distribution Function (cdf)

• cdf gives a complete description of the random
  variable. It is defined as
• FX(x) P(E ? S X(E) x) P(X x).
• The cdf has the following properties
• 0 FX(x) 1 (this follows from Axiom 1 of the
  probability measure).
• Fx(x) is non-decreasing Fx(x1) Fx(x2) if x1
  x2 (this is because event x(E) x1 is contained
  in event x(E) x2).
• Fx(-8) 0 and Fx(8) 1 (x(E) -8 is the empty
  set, hence an impossible event, while x(E) 8 is
  the whole sample space, i.e., a certain event).
• P(a lt x b) Fx(b) - Fx(a).

26
Probability Density Function

• The pdf is defined as the derivative of the cdf
• fx(x) d/dx Fx(x)
• It follows that
• Note that, for all i, one has pi 0 and ?pi 1.

27
Cumulative Joint PDF Joint PDF

• Often encountered when dealing with combined
  experiments or repeated trials of a single
  experiment.
• Multiple random variables are basically
  multidimensional functions defined on a sample
  space of a combined experiment.
• Experiment 1
• S1 x1, x2, ,xm
• Experiment 2
• S2 y1, y2 , , yn
• If we take any one element from S1 and S2
• 0 lt P(xi, yj) lt 1 (Joint Probability of two or
  more outcomes)
• Marginal probabilty distributions
• Sum all j P(xi, yj) P(xi)
• Sum all i P(xi, yj) P(yi)

28
Expectation of Random Variables(Statistical
averages)

• Statistical averages, or moments, play an
  important role in the characterization of the
  random variable.
• The first moment of the probability distribution
  of a random variable X is called mean value mx or
  expected value of a random variable X
• The second moment of a probability distribution
  is mean-square value of X
• Central moments are the moments of the difference
  between X and mx, and second central moment is
  the variance of x.
• Variance is equal to the difference between the
  mean-square value and the square of the mean

29
Contd

• The variance provides a measure of the variables
  randomness.
• The mean and variance of a random variable give a
  partial description of its pdf.

30
Time Averaging and Ergodicity

• A process where any member of the ensemble
  exhibits the same statistical behavior as that of
  the whole ensemble.
• For an ergodic process To measure various
  statistical averages, it is sufficient to look at
  only one realization of the process and find the
  corresponding time average.
• For a process to be ergodic it must be
  stationary. The converse is not true.

31
Gaussian (or Normal) Random Variable (Process)

• A continuous random variable whose pdf is
• µ and are parameters. Usually denoted as
• N(µ, ) .
• Most important and frequently encountered random
  variable in communications.

32
Central Limit Theorem

• CLT provides justification for using Gaussian
  Process as a model based if
• The random variables are statistically
  independent
• The random variables have probability with same
  mean and variance

33
CLT

• The central limit theorem states that
• The probability distribution of Vn approaches a
  normalized Gaussian Distribution N(0, 1) in the
  limit as the number of random variables approach
  infinity
• At times when N is finite it may provide a poor
  approximation of for the actual probability
  distribution

34
Autocorrelation

• Autocorrelation of Energy Signals
• Correlation is a matching process
  autocorrelation refers to the matching of a
  signal with a delayed version of itself
• The autocorrelation function of a real-valued
  energy signal x(t) is defined as
• The autocorrelation function Rx(?) provides a
  measure of how closely the signal matches a copy
  of itself as the copy is shifted ? units in time.
• Rx(?) is not a function of time it is only a
  function of the time difference ? between the
  waveform and its shifted copy.

35
Autocorrelation

• symmetrical in ? about zero
• maximum value occurs at the origin
• autocorrelation and ESD form a Fourier transform
  pair, as designated by the double-headed arrows
• value at the origin is equal to the energy of the
  signal

36
AUTOCORRELATION OF A PERIODIC (POWER) SIGNAL

• The autocorrelation function of a real-valued
  power signal x(t) is defined as
• When the power signal x(t) is periodic with
  period T0, the autocorrelation function can be
  expressed as

37
Autocorrelation of power signals
The autocorrelation function of a real-valued
periodic signal has properties similar to those
of an energy signal

• symmetrical in ? about zero
• maximum value occurs at the origin
• autocorrelation and PSD form a Fourier transform
  pair, as designated by the double-headed arrows
• value at the origin is equal to the average power
  of the signal

38
(No Transcript)
39
(No Transcript)
40
Spectral Density
41
SPECTRAL DENSITY

• The spectral density of a signal characterizes
  the distribution of the signals energy or power,
  in the frequency domain
• This concept is particularly important when
  considering filtering in communication systems
  while evaluating the signal and noise at the
  filter output.
• The energy spectral density (ESD) or the power
  spectral density (PSD) is used in the evaluation.
• Need to determine how the average power or energy
  of the process is distributed in frequency.

42
Spectral Density

• Taking the Fourier transform of the random
  process does not work

43
ENERGY SPECTRAL DENSITY

• Energy spectral density describes the energy per
  unit bandwidth measured in joules/hertz
• Represented as ?x(t), the squared magnitude
  spectrum
• ?x(t) x(f)2
• According to Parsevals Relation
• Therefore
• The Energy spectral density is symmetrical in
  frequency about origin and total energy of the
  signal x(t) can be expressed as

44
Power Spectral Density

• The power spectral density (PSD) function Gx(f)
  of the periodic signal x(t) is a real, even ad
  nonnegative function of frequency that gives the
  distribution of the power of x(t) in the
  frequency domain.
• PSD is represented as (Fourier Series)
• PSD of non-periodic signals
• Whereas the average power of a periodic signal
  x(t) is represented as

45
Noise
46
Noise in the Communication System

• The term noise refers to unwanted electrical
  signals that are always present in electrical
  systems e.g. spark-plug ignition noise,
  switching transients and other electro-magnetic
  signals or atmosphere the sun and other galactic
  sources
• Can describe thermal noise as zero-mean Gaussian
  random process
• A Gaussian process n(t) is a random function
  whose value n at any arbitrary time t is
  statistically characterized by the Gaussian
  probability density function

47
WHITE NOISE

• The primary spectral characteristic of thermal
  noise is that its power spectral density is the
  same for all frequencies of interest in most
  communication systems
• A thermal noise source emanates an equal amount
  of noise power per unit bandwidth at all
  frequenciesfrom dc to about 1012 Hz.
• Power spectral density G(f)
• Autocorrelation function of white noise is
• The average power P of white noise if infinite

48
White Noise
49
White Noise

• Since Rw( T) 0 for T 0, any two different
  samples of white noise, no matter how close in
  time they are taken, are uncorrelated.
• Since the noise samples of white noise are
  uncorrelated, if the noise is both white and
  Gaussian (for example, thermal noise) then the
  noise samples are also independent.

50
Additive White Gaussian Noise (AWGN)

• The effect on the detection process of a channel
  with Additive White Gaussian Noise (AWGN) is that
  the noise affects each transmitted symbol
  independently
• Such a channel is called a memoryless channel
• The term additive means that the noise is
  simply superimposed or added to the signalthat
  there are no multiplicative mechanisms at work

51
Random Processes and Linear Systems

• If a random process forms the input to a
  time-invariant linear system, the output will
  also be a random process

52
Distortion less Transmission

• Remember linear and non-linear group delays in
  DSP

53
DISTORTION LESS TRANSMISSION

• What is required of a network for it to behave
  like an ideal transmission line?
• The output signal from an ideal transmission line
  may have some time delay and different amplitude
  as compared with the input
• It must have no distortionit must have the same
  shape as the input
• For idea distortion less transmission

54
Ideal Distortion Less Transmission

• The overall system response must have a constant
  magnitude response
• The phase shift must be linear with frequency
• All of the signals frequency components must
  also arrive with identical time delay in order to
  add up correctly
• The time delay t0 is related to the phase shift
  and the radian frequency ? 2?f by
• A characteristic often used to measure delay
  distortion of a signal is called envelope delay
  or group delay, which is defined as

55
BANDWIDTH OF DIGITAL DATA

• Baseband signals
• Signals containing frequencies ranging from 0 to
  some frequency fs
• Bandpass or Passband Signals
• Signals containing frequencies ranging from fs1
  to some frequency fs2

56
Note

• Chapters/Topics from different books

• Topics to be covered on your own

• Chapter 1 from Bernard Sklar
• Chapter 1 from Simon Haykin
• Appendix 1 from Digital Communication, Simon
  Haykin for Probability

• Periodic, Non-periodic Signals
• Analog and Digital Signals
• Ideal Filters
• Realizable filters

57
References

• Bernard Sklar
• University of Saskatchewan
• Communication System, Simon Haykin
• MIT open source lectures (Robert Gallager)