Chapter 5 Estimation

1
Chapter 5Estimation

• OVERVIEW
• Maximum likelihood
• Method of moments
• Confidence sets
• Quality of estimators

W02
2
Asbestos counts

• Asbestos fibers on filters were counted as part
  of an NBS project to develop measurement
  standards for asbestos concentrations. Asbestos
  dissolved in water was spread on a filter, and
  punches of 3 mm diameter were taken from the
  filter and mounted on an electron microscope. An
  operator counted the number of fibers on each of
  23 grid squares.
• What is a reasonable distribution?

3
Asbestos, cont.
l23.0
l24.9
4
Parameters

• A parameter is an unknown constant (or constants)
  used to describe uniquely one of a family of
  distributions.
• Geom(p)
• Bin(n,p)
• N(m,s2)
• fX(xq)

5
Concepts
6
Sampling distribution

• Since is a random variable, we can compute
  its cdf
• and other properties such as

7
The likelihood function

• In 1918 R, A, Fisher proposed estimating
  parameters by considering
• how likely are the data if q is the true
  parameter?
• Choosing the parameter that makes the
  observations most likely is formalized using the
  likelihood function
• The data are fixed
• The parameter is varying

8
The exponential case
n50
9
The method of maximum likelihood

• is called the mle (maximum likelihood estimate)
• Usual method
• Set L(q ) 0 and solve for q
• Check that or that L has sign change 0
  - around the root
• Alternatively plot L(q) as a function of q, and
  find the maximum numerically
• Drawback harder to derive properties of
  numerical mle
• Computational trick let l(q) log L(q)
• l(q) is called the log likelihood function

10
Exponential case
11
(No Transcript)
12
Asbestos
13
High occupancy vehicles

• The following data are for passenger car
  occupancy during one hour at Wilshire and Bundy
  in Los Angeles
• The geometric distribution is a reasonable fit.
  The log likelihood is
• whence

14
HOV, continued

• But we do not have data on the 6 group. However,
• so the log likelihood, the log probability of
  what we actually observed, becomes

15
The uniform distribution

• Let X1,,Xn be iid U(0,q). Then
• L(q) qn, so l(q) n log(q) and l(q)
  -n/q
• What is the mle?

16
The hypergeometricdistribution

• Consider X Hyp(N,n,w) and assume that we
  observe Xx. How can we estimate pw/N?

17
Random stopping

• Consider flipping a coin until the first head.
  Suppose it takes 6 tries. The likelihood is
• L(p)
• What if we instead decided beforehand to flip six
  times, and happened to get one head?
• L(p)
• Fact Changing the likelihood by a constant does
  not change the mle.

18
The normal distribution
19
The method of moments

• Karl Pearson (1900) proposed this way of
  estimating parameters. Suppose we have a
  distribution with Eq(X)h(q). By the law of large
  numbers,
• so we let our estimate solve
  the equation

20
The exponential case

• Let Xq,...,Xn be iid Exp(l). then El(X)1/l.
  Below are three sample averages from samples of
  size 50, using l 2.5, plotted against h(l).

21
(No Transcript)
22
Sampling properties

• In this case we can work out some of the
  properties of the mom estimator. Since
• we compute
• fY(y)
• E(1/Y)

23
Asbestos

• The method of moment equation is
• Yielding the same estimator as the mle.
• To find the standard error of the estimate, note
  first that
• so
• Thus we can estimate the standard error by

24
Muon decay

• The cosine x of the emission angle of electrons
  in muon decay has density
• The mean of the density is
• h(a)
• and the resulting mom estimator is

25
The uniform case

• Returning to an iid sample X1,,Xn from U(0,q),
  we have E(X) q/2, so the mom estimator is
  Furthermore,
• So the standard error of the estimator is

26
Precipitation atSnoqualmie Falls

• Consider X non-zero January precipitation
  amounts following a rainy day at Snoqualmie
  Falls. We saw earlier that X G(r,l), so
  E(X)r/l. Since there are two unknowns (a and b)
  we need two equations. To get a second one, note
  that E(X2)r(r1)/l2. Equating these expressions
  to the corresponding sample moments
• Solving for r and l we get
• The estimates given in the Chapter 4 lecture
  notes were mles, and had similar numerical
  values

27
Interval estimates

• The standard error of an estimator has two uses
• comparison to other estimators
• assessment of uncertainty
• An interval estimate (or confidence interval)
  combines an estimate and its estimated standard
  error into a random interval which covers the
  true (but unknown) value of q with a given
  probability (or confidence coefficient) 1-a.
• For a particular sample, the interval may or may
  not cover the true value of q. It does one or the
  otherwe dont know which. But in the long run
  this procedure will cover the true value in about
  100 (1-a) of all samples.

q
28
The exponential case

• Let X Exp (l). The mle is . Will
  the interval cover the
  true value of l?
• How about the interval ?
• Consider the interval How do we pick c1
  and c2 to make this a 95 confidence interval?

29
Asbestos

• Recall the problem of developing standards for
  asbestos measurements. We were observing 23 iid
  Poisson random variables, and estimating l by the
  sample mean 25.6, with standard error ,
  which we estimate by Using the central
  limit theorem we have
• so we compute
• Numerically we get the 95 CI (23.5,27.7).
• We have taken something known the sample mean
• and used a theoretical distribution the sampling
  distribution
• to estimate something unknown the population
  mean
• with a probability that we are right the
  confidence coefficient

30
Multiple intervals

• Consider a researcher constructing 90 confidence
  intervals for 15 different chemical reaction
  constants. The intervals are constructed from
  independent measurements. Some intervals may
  cover the true value, some may not.
• What is the probability that all intervals cover
  the constants?
• What is the most likely number covered?
• How can we get a set of intervals that all cover
  with probability 0.90?

31
(No Transcript)
32
Mle in large samples

• We have seen confidence intervals of the form
• As long as the likelihood function is smooth as a
  function of q, it holds that
• It is not always easy to find the ese, but a
  general formula works

33
Margin of error

• In sample surveys, such as opinion polls, results
  are often given with a margin of error. This is
  roughly the plus or minus number from a 95
  confidence interval for p, i.e. 1.96 ese ( ).
• Note that , regardless of the outcome of the
  poll.
• For a simple random sample of size n 1067 we
  have
• i.e., a 3 margin of error. This is usually set
  in advance of the poll. The actual ese may be
  smaller.

34
What sample sizedo we need?

• The US Commission on Crime is interested in
  estimating the proportion of crimes related to
  firearms in an area with one of the highest crime
  rates in the country. The Commission intends to
  draw a random sample of files on recently
  committed crimes in the area, and want to know
  the proportion of cases with firearms to within
  5 of the true proportion with probability at
  least 90. How big a sample do they need?

35
The finite populationcorrection factor

• SUppose we are sampling from a finite popilation
  of size N, such as the inhabitants of Seattle
  (N750,000 or so), and want to find out the
  proportion that support the current plan of the
  light rail system. We take a random sample of n
  individuals, and ask them their opinion on the
  light rail paln. If the Metro Council wants to
  know this proportion to within 1 (with 95
  confidence), how big a sample do they have to
  take?
• X with favorable opinion Hyp(n,N,np), so for
  large n a normal approximation yields
• The confidence interval we use is
• so we must set

36

• As before we use yielding
• If we ignore the finite population correction
  factor (here N-n/N-10.95) we get the
  binomial sample size formula

37
What are good estimators?

• In some cases the method of moments and the
  method of maximum likelihood yield the same
  estimator. But in other cases they are different.
  How do we decide which is the better estimator?

38
Unbiasedness

• An estimator is unbiased if
• An unbiased estimator is centered at its true
  value
• There may be better estimators that are not
  unbiased
• There are other ways of centering an estimator
• Sometimes one can find the best unbiased
  estimator. But it could be quite silly
• Consider estimating qe-l in a Poisson
  distribution. Then the best unbiased estimator is
  the only one

39
Exponential distribution

• For the mle we calculated
• It is not unbiased. What can we do about it?

40
Binomial distribution

• Consider a binomial experiment with n4, and
  outcomes x1,,x4. Here are three possible
  estimators of the success probability p
• Which of these are unbiased?

41
(No Transcript)
42
Uniform distribution

• Recall that the mle of q in U(0,q) is max(Xi). Is
  it unbiased?
• P(max(Xi) x) P(X1 x, X2 x,,Xn x)
• E(max(Xi))

43
The sample variance

• We saw that the mle of the variance of the normal
  distribution is
• A tedious calculation shows that it is not
  unbiased, but has expected value (1-1/n) s2. This
  is true for all distributions that have a
  variance. Hence
• is unbiased (it is often called the sample
  variance).
• Note, however, that
• the corresponding sample standard deviation, is
  not unbiased.
• If the population mean m is known, we would use
• Is it unbiased?

44
Reparametrization

• A drug reaction surveillance program was carried
  out in 9 hopsitals. Out of 11,526 monitored
  patients, 3,240 had an adverse reaction.
• Model X adverse reactions
• Log likelihood
• Standard error of the mle
• The bootstrap method of estimating standard
  error just plugs in the estimate of p into the
  formula for the standard error. But what is the
  mle of the standard error?
• Fact The mle of h(q) is
• We say that the mle is invariant under
  reparametrization.
• Unbiased estimators? Method of moments?

45
Binomial experiment

• Consider again the binomial experiment with n4,
  and the three unbiased estimators
• What are their variances?

46
Efficiency

• Define the relative efficiency of two unbiased
  estimators by the ratio of their variances
• In the binomial case,
• and

47
The uniform case

• For the U(0,q) case, the mom estimate was while
  the unbiased modification of the maximum
  likelihood estimate was
• We have
• while E(max(Xi))
• so Var(max(Xi))
• Hence

48
Mean squared error

• If we want to compare two estimators when one of
  them is unbiased and the other is not, we look at
  the mean squared error
• Note that we can write
• The mse is a combination between bias and
  variability of the estimator. Usually decreasing
  one increases the other.

49
Exponential distribution

• Recall that . Similarly,
• and
• Now let
• Then

50
Uniform case

• We need to compare the mle of q in the
  U(0,q)-case to its bias-corrected version (the
  latter is superior to the mom estimator).

51
(No Transcript)