Chapter 1: Discrete and Continuous Probability Distributions

1
Chapter 1 Discrete and Continuous Probability
Distributions

• 1.1 Probability Theory
• 1.2 Discrete Probability Distribution
• 1.3 Continuous Probability Distribution

2
1.1 Probability Theory

• 1.1.1 Basic Probability Theory
• 1.1.2 Set Theory Operations
• 1.1.3 Counting Techniques
• 1.1.4 Addition, Multiplication
• 1.1.5 Conditional Probability
• 1.1.6 Independence
• 1.1.7 Total Probability Rule Bayes Theorem

3
Basics of Probability Theory

• Probability refers to the study of randomness and
  uncertainty.
• In other words, probability is a numerical
  measure of chance for the occurrence of an event.

4

• Example 1
• An experiment consists of tossing three coins.
  Find the sample space if
• We are interested in the observed face of each
  coin,
• We are interested in the total number of heads
  obtained.
• Solution
• S HHH, HHT, HTH, THH, HTT, THT, TTH, TTT
• S 0, 1, 2, 3

5
Frequentist Interpretation

• Intuitively, the probability of an event is the
  proportion of times the event should occur when
  the experiment is run a large number of times.
• Suppose S is a sample space in which all
  outcomes are assumed to be equally likely, and E
  is an event. Then the probability of E,
  denoted by P(E), is
• Probability is quantified as a percentage or a
  proportion of the total possible outcomes.

6
Subjective Interpretation

• There is yet another definition of probability,
  called as the subjective probability, which is
  the foundation of Bayesian Econometrics.
• Under the subjective or degrees of belief"
  definition of probability, you can ask question
  such as
• What is the probability that there will be a
  stock market boom in 2005?
• What is the probability that Brazil will win the
  World Cup?

7
Set Operations on Events
8
(No Transcript)
9

• Example 2
• For the experiment in which the number of pumps
  in use at a single six-pump gas station is
  observed. Let A 0, 1, 2, 3, 4, B 3, 4, 5,
  6, and C 1, 3, 5. Then find
• (a) A?B (b) A?B (c) A?C (d) (A?C)
• Solutions
• (a) A?B 0, 1, 2, 3, 4 ? 3, 4, 5, 6 3, 4
• (b) A?B 0,1,2,3,4 ? 3, 4, 5, 6 0, 1, 2,
  3, 4, 5, 6 S
• (c) A?C 0, 1, 2, 3, 4? 1, 3, 5 0, 1, 2,
  3, 4, 5
• (d) (A?C) S (A?C) 6

10
Counting Techniques

• Product Rule
• If an operation can be described as a sequence of
  k steps, and
• if the number of ways of completing step 1 is
  n1, and
• if the number of completing step two is n2 for
  each way of completing step 1, and so forth,
• the total number of ways of completing the
  operation is n1 x n2 x n3 x .. X nk

11

• Permutation
• Any ordered sequence of k objects taken from a
  set of n distinct objects is called a permutation
  of size k of the objects.
• The number of permutation of size k that can be
  constructed from n objects is denoted by
• Combination
• Given a set of n distinct objects, any unordered
  subset of size k of the objects is called a
  combination.
• The number of combination of size k that can be
  constructed from n objects is denoted by

12

• Example 3
• A restaurant has four kinds of soups, eight
  kinds of main course, five kinds of dessert, and
  six kinds of drinks. If a customer selects one
  item from each category, how many different
  outcomes are possible?
• Solution
• 4 x 8 x 5 x 6 960

13

• Example 4
• There are ten teaching assistants available for
  grading papers in a particular course. The exam
  consists of four questions, and the professor
  wishes to select different assistant to grade
  each question (one assistant per question). In
  how may ways can assistant be chosen to grade the
  exam?
• Solution
• Recall that , where n
  10 and k 4
• thus number of permutations is

14

• Example 5
• A printed circuit board has 8 different
  locations in which a component can be placed. If
  5 identical component are to be placed on the
  board, how many different designs are possible?
• Solution
• Each design is a subset of the 8 locations that
  are to contain the components. The number of
  possible designs is
• 56

15
Addition and Multiplication Rule

• Addition Rules
• Multiplication Rule

If A and B are mutually exclusive
16

• Example 6
• Assume that the engine component of a spacecraft
  consists of two engines in parallel. If the main
  engine is 95 reliable, the backup is 80
  reliable, and the engine component as a whole is
  99 reliable, what is the probability that
• Both engines will be operable i.e P(M ? B)?
• The main engine will fail but the backup will be
  operable? i.e P(M ?B)
• The engine component will fail i.e P(M ? B) ?

17

• Solution
• Engine reliability
• main P(M) 0.95
• backup P(B) 0.80
• whole P(M ? B) 0.99
• Both engines operable P(M ? B)
• Recall P(M ? B) P(M)P(B) ?- P(M ? B)
• 0.95 0.80 0.99 0.76
• Main engine fail but backup operable P(M ?B)
• P(M?B) P(B) P(M?B)
• 0.80 0.76 0.04
• Engine component fails
• P(M ? B) 1 P(M ? B)
• P(M ? B) 0.01

18
Conditional Probability
19

• Example 7
• In studying the causes of power failures, these
  data have been gathered. 5 are due to
  transformer damage, 80 are due to line damage
  and 1 involve both problems. Based on these
  percentages, what is the probability that a given
  power failure involves
• line damage given that there is transformer
  damage, P(LT)
• transformer damage given that there is line
  damage, P(TL)
• transformer damage but not line damage, P(T?L)
• transformer damage given that there is no line
  damage, P(TL)

20

• Transformer damage P(T) 0.05,
• Line damage P(L) 0.80
• Transformer and Line damage P(T ? L) 0.01
• i)
• ii)
• iii)
• iv)

21

• Example 8
• Table above show an example of 400 parts
  classified by surface flaws and as defective.
  Determine the following probabilities
• Part is defective given that there is surface
  flaw
• Part is defective given that there is no surface
  flaw
• Part has surface flaw given that it is defective
• Part has no surface flaw and it is not defective

22
(No Transcript)
23
Independence

• Independent Events
• Two events are independent if the occurrence of
  one does not affect the probability of the other
  occurring.
• Dependent Events
• Two events are dependent if the first event
  affects the outcome or occurrence of the second
  event in a way the probability is changed.
• Two events A and B are called independent events
  if
• 1) P(AB) P(A)
• 2) P(BA) P(B)
• 3)
• Otherwise, A and B are dependent.

24

• Example 9
• It is known that 30 of a certain companys
  washing machines require service while under
  warranty, whereas only 10 of its dryers need
  such service. If someone purchases both a washer
  and a dryer made by this company, what is the
  probability that both machines need warranty
  service?
• Solution
• P(A) 0.3, P(B) 0.1. Assuming the two
  machines function independently, the probability
  is

25
Total Probability Rule
26
Bayes Theorem
27

• Example 10
• A certain disease occurs in mild or severe form
  three-quarter of patients have the mild form. A
  new drug is available. The probability that a
  mild case of the disease responds to the drug is
  0.9, and the probability that a severe case
  responds is 0.5.
• What is the probability that a randomly chosen
  case will respond to the drug, P(R)?
• You are told that a certain patient has responded
  to the drug. What is the probability that the
  patient has the mild form of disease, P(MR)?

28

• Solution
• Mild cases P(M) 0.75 Severe cases P(S)
  0.25
• Response P(RM) 0.9 P(RS) 0.5
• P(R) P(R ? M) P(R?S)
• P(R) P(R ? M) P(R?S) 0.8.
• ,

29

• Example 11
• In a bolt factory, machine 1, 2 and 3
  respectively produce 20, 30 and 50 of the
  total output. Of their respective outputs, 5, 3
  and 2 are defective. A bolt is selected at
  random
• What is the probability that it is defective,
  P(D)?
• Given that it is defective, what is the
  probability that it was made by machine 1?
• Solution
• Let machine 1, 2, 3 be A, B, C respectively.
• Denote the probabilities P(A)0.2, P(B)0.3,
  P(C) 0.5.
• Defective Output
• P(DA)0.05, P(DB) 0.03, P(DC)0.02

30

• Defective bolt P(D) P(D?A) P(D?B) P(D?C)
• Recall P(D?A) P(DA)P(A) and likewise
• P(D) P(DA)P(A) P(DB)P(B) P(DC)P(C)
• 0.01 0.009 0.01 0.029 .
• P(AD)
• Recall

31
1.2 Discrete Random Variable and Probability
Distribution

• 1.2.1 Discrete Random Variables
• 1.2.2 Discrete Probability Distribution
• 1.2.3 Expected Values of Discrete Random
  Variables
• 1.2.4 Binomial Distribution
• 1.2.5 Negative Binomial Distribution
• 1.2.6 Poisson Distribution

32
Discrete Random Variables

• X is discrete random variable if it takes
  discrete values x1, x2, , xn, with respective
  probabilities associated with these values are
  p1, p2, , pn, where
• P(X x1) p1
• P(X x2) p2
• P(X xn) pn
• Furthermore, p1 p2 ... pn 1

33

• Example 12
• Observe the outcome of tossing a die

34
Discrete Probability Distribution

• The probability distribution of a discrete random
  variable lists all the possible values that the
  random variable can assume and their
  corresponding probabilities.
• The probability distribution or probability mass
  function (pmf) for the discrete random variable
  is defined for every number x by f(x)P(X x)
  P
• The following conditions are required for any pmf
• (positivity)
• ? f(xi) 1 (normalization)

35

• The cumulative distribution function (cdf) F(x)
  of a discrete random variable X with probability
  mass function f(x) is defined by
• F(x)
• ,

36

• Example 13
• From a box containing four 10 cents and two 5
  cents, 3 coins are selected at random without
  replacement. Find
• the probability distribution for the sum X, of
  the 3 coins.
• the cumulative mass function for the sum X, of
  the 3 coins
• Solution
• Let F Five cent, T Ten cent, X sum of 3
  coins
• Possible outcome FFT, FTF, TFF, FTT, TFT, TTF,
  TTT
• Possible sum, X 20, 25, 30 random variable

37

• pmf
• cdf

38
Expected Values of Discrete Random Variables

• Expected Value of X
• Let X be a discrete random variable with set of
  possible values and probability mass function
  p(x).
• The expected vale or mean value of X denoted by
  E(X) or ?X is
• Expected Value of a Function h(X)
• If the random variable has a set of possible
  values n and pmf p(x), then the expected value of
  any function h(X), is given by

39

• Laws of Expectation (Properties of Mean, E(X))
• E(a) a where a is any constant
• E(aX) aE(X)
• E(aX b) aE(X) b
• E(X ? Y) E(X) ? E(Y)
• Variance of a Random Variable
• Let X be a random variable with probability
  distribution f(x) and mean ?. The variance of X,
  denote by V(X) or , or just , is

40

• Example 14
• There is a chance that a bit transmitted through
  a digital transmission channel is received in
  error. Let X equal the number of bits in error in
  the next four bits transmitted. The possible
  values for X are 0, 1, 2, 3, 4. Based on a
  model for the errors that is presented in the
  following section, probabilities for these values
  will be determined. Suppose the probabilities
  are
• P(X 0) 0.6561 P(X 1) 0.2916 P(X 2)
  0.0486
• P(X 3) 0.0036 P(X 4) 0.0001
• The mean value of X is
• E(X) ?X
• 0(0.6561) 1(0.2916) 2(0.0486)
  3(0.0036) 4(0.0001)
• 0.4

41

• The variance of X is

42

• The function of Eh(X)
• What is the expected value of the square of the
  number of bits in error?
• h(X) X2
• Remember
• Eh(X) 02 x 0.6561 12 x 0.2916 22 x
  0.0486
• 32 x 0.0036 42 x 0.0001
• 0.52

43
Binomial Distribution

• Binomial distribution is a commonly used discrete
  probability distribution since many statistical
  problems are deal with the situations referred to
  as repeated trial.
• Those experiment that possess the following
  properties are called binomial experiment
• Properties of a Binomial Experiment
• The experiment consists of a sequence of n
  identical trials.
• Two outcomes, success and failure, are possible
  on each trial.
• The probability of a success, denoted by p, does
  not change from trial to trial.
• The trials are independent.

44

• Our interest is in the number of successes
  occurring in the n trials. Let x denote the
  number of successes occurring in the n trials.
• Number of experimental outcomes providing exactly
  x successes in n trials
• where n! n(n 1)(n 2) . . . (2)(1)
• 0! 1
• Probability of a particular sequence of trial
  outcomes with x successes in n trials is

45

• Probability mass function
• Probability mass function of binomial random
  variable X depends on the two parameters n(number
  of trials) and p (probability of success),
  denoted as
• where n number of trials
• x number of success among n trials
• p probability of success in any one trial
• q probability of failure in any one
  trial.

46

• P(Xx) f(x) probability of getting
  exactly x success among the n trails.
• Note A short notation to designate that X has
  the binomial distribution with parameter n and p
  is and
• Mean ? np
• Variance ?2 n pq

47
(No Transcript)
48

• Example 15
• A software company Mysoftcom is concerned about
  a low retention rate for employees. On the basis
  of past experience, management has seen a
  turnover of 10 of the hourly employees annually.
  Thus, for any hourly employees chosen at random,
  management estimates a probability of 0.1 that
  the person will not be with the company next
  year.
• Choosing 3 hourly employees at random, what is
  the probability that 1 of them will leave the
  company this year?

49

• Solution
• Recall the Binomial probability distribution
• Given n 3, x 1, p0.1, thus
• One can also use the Tables of Binomial
  Probabilities

50

• Example 16
• The probability that a patient recovers from
  SARS is 0.4. If 15 people are known to have
  contracted this disease, what is the probability
  that
• at least 13 survive
• at most 2 die
• Solution
• n15, survival probability, ps 0.4, dying
  probability pd0.6
• X BIN(15, 0.4)
• P(X ?13) P(X13) P(X14) P(X15)
  0.0002789
• Y BIN(15, 0.6)
• P(Y ? 2) P(Y0) P(Y1) P(Y2) 0.0002789

51
Negative Binomial Distribution

• The negative binomial random variable and
  distribution are based on an experiment
  satisfying the following conditions
• There are only two possible outcomes for each
  trial
• The trails must be independent
• The probability must remain constant for each
  trial.
• The experiment continues (trials are performed)
  until a total of r successes have been observed,
  where r is a specified positive integer.

52

• Probability mass function
• The pmf of the negative binomial random variable
  X with parameter r and p is
• x 0, 1, 2,
• where r number of success (S)
• x number of failures that precede the rth
  success
• p probability of success in any one trail
  
• Mean
• Variance

53

• For special case r 1, the pmf becomes
• x 0, 1 , 2,
• This pmf is called geometric distribution. In is
  usually written as

54
Poisson Distribution

• The Poisson distribution is a discrete
  probability distribution that applies to
  occurrences of some event over a specific
  interval. The random variable X is the number of
  occurrences of the event in an interval.
• The interval can be time, distance, area, volume,
  or some similar unit.
• Those experiment that possess the following
  properties are called Poisson experiment
• The occurrences must be random.
• The occurrences must be independent from one
  interval to another.
• The occurrence must be uniformly distributed over
  the interval being used.

55

• The probability distribution of the Poisson
  random variable X, representing the number of
  outcomes occurring over an interval is given by
  the formula
• P(Xx) f(x ) x 0, 1, 2, and
• with
• Mean ? ?,
• Variance ?2 ?
• Note A short notation to designate that X has
  the Poisson distribution with parameter ? is

56

• Example of occurrences
• The number of cars that pass through a certain
  point on a road (sufficiently distant from
  traffic lights) during a given period of time.
• The number of spelling mistakes one makes while
  typing a single page.
• The number of phone calls at a call center per
  minute.
• The number of times a web server is accessed per
  minute.
• The number of mutations in a given stretch of DNA
  after a certain amount of radiation.
• The number of pine trees per unit area of mixed
  forest.
• The number of stars in a given volume of space.

57

• Example 17
• Patients arrive at the emergency room of
  Putrajaya Hospital at the average rate of 6 per
  hour on weekend evenings. What is the
  probability of 4 arrivals in 30 minutes on a
  weekend evening?
• Solution
• Given that ? 6/hour 3/half-hour, and x 4,
  therefore probability of 4 arrivals in 30 minutes
  is

58

• Or one can also use the Tables of Poisson
  Probabilities

59

• Example 18
• Customers arrive randomly at a department store
  at an average rate of 3.4 per minute. Find
• No customer arrives in any particular minute.
• Two or more customers arrive in any particular
  minute.
• One or more customers arrive in any 30-second
  period.
• Solution
• ?3.4, X POI(3.4)
• P(X 0) 0.03337
• P(X ?2) 1? P(X?1) 0.8532
• Y POI(1.7), P(Y?1) 0.8173

60
1.3 Continuous Random Variable and Probability
Distribution

• 1.3.1 Continuous Random Variables
• 1.3.2 Continuous Probability Distribution
• 1.3.3 Expected Values of Continuous Random
  Variables
• 1.3.4 Uniform Distribution
• 1.3.5 Normal (Gauss) Distribution
• 1.3.6 Relationship between Gaussian and Binomial
  Distribution

61
Continuous Random Variables

• A continuous random variable has infinite many
  values, and those values can be associated with
  measurements on a continuous scale.
• For instance, if X is the result of rolling a die
  (and observing the uppermost face), then X is a
  discrete random variable with possible values 1,
  2, 3, 4, 5 and 6. On the other hand, if X is a
  random choice of a real number in the interval
  1,6, then it is a continuous random variable.
• If X is a random variable, we are usually
  interested in the probability that X takes on a
  value in a certain range.

62
Continuous Probability Distribution

• The function f(x) is a probability distribution
  or probability density function (pdf) for the
  continuous random variable X, defined over the
  set of real number R, if
• f(x) ? 0
• ,
• ,

63

• The cumulative distribution function, F(x) of a
  continuous random variable X with density
  function f(x) is defined by
• Note Let X be a continuous random variable with
  pdf f(x) and cdf F(x), then for any number a,
• and for any two numbers a and b with a lt b,

64

• Example 19
• Suppose that a battery failure time, measured in
  hours, has a probability density function of
• What is the probability that the battery fails
  within the first 5 hours?

65
(No Transcript)
66

• Example 20
• Let X denote the amount of time for which a book
  on two hour reserve at a university library is
  checked out by a randomly selected student and
  suppose that X has probability density function
• Calculate the following probabilities
• P(X ? 1)
• P(0.5 ? X ? 1.5)
• P(1.5 lt X)

67

• Solution
• P(x ? 1)
• P(0.5 ? X ? 1.5)
• P(x gt 1.5)

68
Expected Values of Continuous Random Variables

• Expected Value of X
• Let X be a continuous random variable with
  probability distribution f(x). The mean or
  expected value of X is given by
• ?x E(X)
• Expected Value of a Function h(X)
• The mean or expected value of a function of
  random variable h(X) is
• Eh(X) ?h(x)

69

• Variance of a Random Variable
• The variance of X, denote by V(X) or , or
  just , is
• V(X) ?2
• or
• ?2 EX2 - ?2
• The positive square root of the variance, ?x, is
  called the standard deviation of X.
• Law of Variance
• Var (a)0 where a is any constant
• Var (aX) a2Var (X)
• Var (aXb) a2Var (X)
• Var (Xb) Var (X)

70

• Example 21
• The weekly demand for Pepsi, in thousands of
  liters, from a local chain of efficiency stores,
  is a continuous random variable X having the
  probability density
• Find the variance of X.

71

• Solution
• Recall that
• where the mean,
• And where the mean of a function is,
• Therefore the variance,

72

• Example 22
• Let X be a random variable with density function
• Find the variance of the random variable g(X)
  4X 3
• Solution

73
Uniform Distribution

• The uniform distribution is a probability
  distribution in which the probability of a value
  occurring between two points, a and b, is the
  same as the probability between any other two
  points, c and d, given that the distribution
  between a and b is equal to the distance between
  c and d.
• where
• f(x) Value of the density function
  at any x value
• a lower limit of the interval from a to b
• b upper limit of the interval from a to b

74

• The probability density function and cumulative
  distribution function for a continuous uniform
  distribution on the interval are
• Mean
• Variance

75

• Example 23

76
Normal / Gaussian Distribution

• The bell shaped curve also known as normal curve
  is widely used to approximately many phenomena.
• The random variable with normal distribution is
  characterized by with its mean ? and variance ?2.
  
• The pdf is given by

77

• Standard Normal Distribution
• The standard normal distribution is a normal
  probability distribution that has a mean of 0 and
  a standard deviation of 1.
• All the observations of any normal variable X can
  be transformed to a new set of observations of
  standard normal variable z with mean 0 and
  variance 1.
• We can find the areas under the standard normal
  curve by referring to Standard Normal Tables
  which give cumulative probabilities ?(z).

78

• Standard Normal Curve Areas, ?(z) P(Z ? z)

79

• Converting Nonstandard Normal Distribution
• If X has a normal distribution with mean u and
  standard deviation s, then
• has a standard normal distribution. Thus
• and
• and

80
Normal Distribution Table
81
(No Transcript)
82

• Example 24

83

• Example 25
• The masses of articles produced in a particular
  workshop are normally distributed with mean ? and
  standard deviation ?. 5 of the articles have a
  mass greater than 85g and 10 have a mass less
  than 25g. Find
• ? and ?.
• P(X gt 60)
• a, given that P( a lt X lt 74.85) 0.75

84

• ? and ?.

85

• P(X gt 60)
• a, given that P( a lt X lt 74.85) 0.75

86
Relationship between Gaussian and Binomial
Distribution

• The Gaussian distribution can be derived from the
  binomial (or Poisson) assuming
• p is finite
• N is very large
• we have a continuous variable rather than a
  discrete variable
• Limit distribution
• ? np and

Normal approximation to the binomial
87

• Normal Approximation to the Binomial Distribution
• Normal Approximation to the Poisson Distribution

88
END CHAPTER 1