Basic Concepts of Discrete Probability

1
Basic Concepts of Discrete Probability

• (Theory of Sets Continuation)

2
Functions

• If X is a set and Y is a set, and there is a
  sequence of well-specified operations for
  assigning a well-defined object to
  every element , and by applying these
  sequence of operations to every member of a set X
  we obtain a set Y, then this sequence of
  operations forms a rule, which is commonly known
  as function.

3
Functions

• The set X is called the domain of the function,
  and the set Y is called the range.
• Functions that have numerical values ere called
  numerical functions.
• A numerical function establishing the
  correspondence between the elements of a set
  having a finite number of elements and a set of
  natural numbers will be of our particular
  interest

4
The number of elements in a set

• The number of elements in a set may or may not be
  finite.
• If the elements of the set can be placed in
  one-to-one correspondence with the set of natural
  numbers 1, 2, 3,, we say that the set has a
  denumerable or countable number of elements
  otherwise the set is nondenumerable (e.g., a set
  of points on any line or the set of points in
  0,1).

5
The number of elements in a set

• If a set is denumerable and
  is a finite number, we say that the power of
  the set X (the cardinality of the set X) is X
  (the number of elements in the set X).
• If a set is equivalent to the set of points in
  0,1 that is, it is denumerable, we say that the
  set has the power of continuum.

6
The number of elements in a set

• The following properties hold for finite sets
• If A and B are two disjoint sets, then
• For any finite sets A, B, and C

7
Basic Concepts of Discrete Probability

• Elements of the Probability Theory

8
Sample Space

• When probability is applied to something, we
  usually mean an experiment with certain outcomes.
• An outcome is any one of the possibilities that
  may be expected from the experiment.
• The totality of all these outcomes forms a
  universal set which is called the sample space.

9
Sample Space

• For example, if we checked occasionally the
  number of people in this classroom on Tuesday
  from 7pm to 9-45pm, we should consider this an
  experiment having 15 possible outcomes
  0,1,2,,13,14 that form a universal set.
• 0 nobody is in the classroom, 14 all
  students taking the Information Theory Class and
  the instructor are in the classroom

10
Sample Space

• A sample space containing at most a denumerable
  number of elements is called discrete.
• A sample space containing a nondenumerable number
  of elements is called continuous.

11
Sample Space

• A subset of a sample space containing any number
  of elements (outcomes) is called an event.
• Null event is an empty subset. It represents an
  event that is impossible.
• An event containing all sample points is an event
  that is certain to occur.

12
Probability Measure

• The probability measure is a specific type of
  function which can be associated with sets.
• Since an experiment is defined so that to each
  possible outcome of this experiment there
  corresponds a point in the sample space, the
  number of outcomes of the experiment is assumed
  to be at most denumerable.

13
Probability Measure

• Let us consider an event of interest A as the set
  of outcomes ak.
• Let a real function m(ak ) be the probability
  measure of the outcome ak.
• The probability measure of an event is defined as
  the sum of the probability measures associated
  with all the outcomes ak of that event. It is
  also referred to as the additive probability
  measure

14
Probability Measure

• The additive probability measure has the
  following important property the measure
  associated with the union of two disjoint sets is
  equal to the sum of their individual measures.

15
Probability Measure

• Two events A and B are called disjoint if they
  contain no outcome in common two disjoint events
  cannot happen simultaneously. The probability
  measure has the following assumed properties

16
Probability Measure

• If with an
  additive probability measure, then the following
  relations are valid

17
Probability Measure

• For three disjoint sets
  
• For three sets in general

18
Frequency of events

• Let us consider a specific event Xk among all
  the possible events of the experiment under
  consideration. If the basic experiment is
  repeated N times among which the event Xk has
  appeared times, the ratio
• is defined as the relative frequency of the
  occurrence of the event Xk.

19
Frequency of events. The Probability

• If N is increased indefinitely, that is if
  then, intuitively speaking,
• is the probability of the event Xk.

20
The Probability

• The classical definition of Laplace says that the
  probability is the ratio of the number of
  favorable events to the total number of possible
  events.
• All events in this definition are considered to
  be equally likely e.g., throwing of a true die
  by an honest person under prescribed
  circumstances
• but not checking of the number of people in the
  classroom.

21
The Probability

• The following properties are important

22
The Probability

• To verify whether our empiric definition of
  probability satisfies the properties of the
  probability measure, we have to consider the
  probability of two events.
• Let A and B be the events among the ones
  resulting from the experiment. Let the experiment
  be repeated n times.

23
The Probability

• Each observation can belong to only one of the
  four following categories
• 1) A has occurred, but not B ? AB
• 2) B has occurred, but not A ? BA
• 3) Both A and B have occurred ? AB
• 4) Neither A nor B has occurred ? AB

24
The Probability

• If the number of events of each category from the
  mentioned four ones is denoted by
• then
• relative frequency of A
  independent of B
• relative frequency of
  B independent of A

25
The Probability

• relative
  frequency of either A, B or both
• relative frequency of A and B
  occurring together
• relative frequency of A
  under condition that B has occurred
• relative frequency of B
  under condition that A has occurred

26
The Probability

• When the number of experiments tends to infinity,
  these relations lead to
• If A and B are mutually exclusive events, then
  PAB0 PABPAPB

27
The Probability

• Thus, we have proved that
• Let us compare the latter with the properties of
  the probability measure

28
The Probability

• We have just shown that our empirical definitions
  of the frequency and probability, respectively,
  satisfy all the properties of the probability
  measure.

29
Theorem of addition

• For two events A and B of the sample space
• The additive property of the probability measure
  suggests that
• If A and B are mutually exclusive events, then

30
Theorem of addition

• For two opposite events A and A AAU AA0,
  then

31
Theorem of addition

• For the three events A, B, and C
• In general, for a number of events

32
Theorem of addition

• If the events are mutually exclusive, then
• It is also clear now that

33
Conditional Probability

• Let A and B be two events. The conditional
  probability of event A based on the hypothesis
  that event B has occurred is defined by the
  following relation

34
Conditional Probability

• Returning to the example of two events
• 1) A has occurred, but not B ? AB ?n1
• 2) B has occurred, but not A ? BA ?n2
• 3) Both A and B have occurred ? AB ?n3
• 4) Neither A nor B has occurred ? AB ?n4

35
Conditional Probability

• The two events A and B are called mutually
  independent if PABPA and PBAPB.
• For mutually independent events PABPAPB.

36
Theorem of multiplication

• The multiplication rule for the case of two
  events A and B can be obtained through the
  definition of the conditional probability
  PABPAPBAPBPAB.
• For three events A,B, and C PABCPABPCABP
  APBAPCAB

37
Theorem of multiplication

• For n events
• For a countable infinite number of the mutually
  independent events