Basics of Statistical Estimation

1
Basics of Statistical Estimation
2
Learning ProbabilitiesClassical Approach
Simplest case Flipping a thumbtack
True probability q is unknown
Given iid data, estimate q using an estimator
with good properties low bias, low variance,
consistent (e.g., maximum likelihood estimate)
3
Maximum Likelihood Principle
Choose the parameters that maximize the
probability of the observed data
4
Maximum Likelihood Estimation
(Number of heads is binomial distribution)
5
Computing the ML Estimate

• Use log-likelihood
• Differentiate with respect to parameter(s)
• Equate to zero and solve
• Solution

6
Sufficient Statistics
(h,t) are sufficient statistics
7
Bayesian Estimation
True probability q is unknown Bayesian
probability density for q
8
Use of Bayes Theorem
prior
likelihood
posterior
9
Example Application to Observation of Single
Heads"
p(qheads)
p(q)
p(headsq) q
q
q
q
0
1
0
1
0
1
prior
likelihood
posterior
10
Probability of Heads on Next Toss
11
MAP Estimation

• Approximation
• Instead of averaging over all parameter values
• Consider only the most probable value(i.e.,
  value with highest posterior probability)
• Usually a very good approximation,and much
  simpler
• MAP value ? Expected value
• MAP ? ML for infinite data(as long as prior ? 0
  everywhere)

12
Prior Distributions for q

• Direct assessment
• Parametric distributions
• Conjugate distributions(for convenience)
• Mixtures of conjugate distributions

13
Conjugate Family of Distributions
Beta distribution
Resulting posterior distribution
14
Estimates Compared

• Prior prediction
• Posterior prediction
• MAP estimate
• ML estimate

15
Intuition

• The hyperparameters ah and at can be thought of
  as imaginary counts from our prior experience,
  starting from "pure ignorance"
• Equivalent sample size ah at
• The larger the equivalent sample size, the more
  confident we are about the true probability

16
Beta Distributions
Beta(3, 2 )
Beta(1, 1 )
Beta(19, 39 )
Beta(0.5, 0.5 )
17
Assessment of aBeta Distribution
Method 1 Equivalent sample - assess ah and
at - assess ahat and ah/(ahat) Method 2
Imagined future samples
18
Generalization to m Outcomes(Multinomial
Distribution)
Dirichlet distribution
Properties
19
Other Distributions

• Likelihoods from the exponential family
• Binomial
• Multinomial
• Poisson
• Gamma
• Normal

20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)