The Bootstraps Finite Sample Distribution

1
The Bootstraps Finite Sample Distribution An
Analytical Approach
Lawrence C. Marsh
Department of Economics and Econometrics Universit
y of Notre Dame
Econometrics Seminar/Workshop Stanford University
February 23, 2005
2
This is the first of three papers
(1.) Bootstraps Finite Sample Distribution (
today !!! )
(2.) Bootstrapped Asymptotically Pivotal
Statistics
(3.) Bootstrap Hypothesis Testing and Confidence
Intervals
3
Efron (1979)
Method I Exact analytical approach.
Method II Bootstrap simulations.
Method III Taylor series expansions.
4
Review of Bootstraps Intuition
Simplist (trivial) case is the sample mean
? original data points
? average bootstrapped value
5 numbers (3)(3) 9 gap25
1
2
3
n 3 m 2
2
1
?
?
?
?
?
?
?
?
0
100
50
1
3
7
6
7 numbers (3)(3)(3)27 gap16.66
1
6
3
n 3 m 3
?
?
?
?
?
?
?
?
?
?
0
50
100
1
4
9 numbers (3)(3)(3)(3)81 gap12.5
10
16
16
4
10
1
19
n 3 m 4
?
?
?
?
?
?
?
?
?
?
?
?
50
0
100
? missing data points
5
Review of Bootstraps Intuition
Simplist (trivial) case is the sample mean
? original data points
? average bootstrapped value
6 numbers (3)(3) 9 gap12.5
1
2
2
n 3 m 2
1
2
1
?
?
?
?
?
?
?
?
?
?
?
?
0
100
25
1
3
6
1
3
3
3
10 numbers (3)(3)(3)27 gap8.33
1
3
3
n 3 m 3
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
0
25
100
5
1
4
4
4
6
14 numbers (3)(3)(3)(3)81 gap6.25
6
12
12
12
1
4
4
n 3 m 4
6
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
0
25
100
? missing data points
6
traditional approach in econometrics
Analogy principle (Manski) GMM (Hansen)
Empirical process
Analytical problem
?
?
approach used in this paper
Analytical solution
Bootstraps Finite Sample Distribution
Empirical process
?
?
7
bootstrap procedure
Start with a sample of size n
Xi i 1,,n
Bootstrap sample of size m
Xj j 1,,m
m lt n or m n or
m gt n
Define Mi as the frequency of drawing each Xi
.
8
.
.
.
9
for i ? k
10
Applied Econometrician
The bootstrap treats the original sample as if
it were the population and induces multinomial
distributed randomness.

11
Econometric theorist what does this buy you?
Find out under joint distribution of
bootstrap-induced randomness and randomness
implied by the original sample data

12
Applied Econometrician
For example,
Econometric theorist

13
Skewness
where
14
Skewness
15
Balanced Bootstrap(the unbiased bootstrap)

• Create bootstrap that ultimately draws each of
• the original numbers an equal number of times.

• Make b copies of the original sample and draw
• bootstrap samples of size m with equal
  probability without replacement, where b is some
  exact multiple of m.

16
Multinomial Bootstrap sampling with replacement.

Hypergeometric Bootstrap sampling without
replacement.
If bn 2m, then VarH 1 / ( 2 - 1/m)VarM .
If bn 3m, then VarH 2 / ( 3 - 1/m)VarM .
17
Multinomial bootstrap sampling with
replacement.

Hypergeometric bootstrap sampling without
replacement.

18
The Wild Bootstrap
Multiply each boostrapped value by plus one or
minus one each with a probability of one-half
(Rademacher Distribution).
Use binomial distribution to impose Rademacher
distribution
Wi number of positive ones out of Mi which,
in turn, is the number of Xis drawn in m
multinomial draws.
19
The Wild Bootstrap
Applied Econometrician

Econometric Theorist
under zero mean assumption
20
More generally, apply any function q as follows
.
.
.
21
where X is a p x 1 vector.
nonlinear function of ?.
Horowitz (2001) approximates the bias of
22

23
Horowitz (2001) uses bootstrap simulations to
approximate the first term on the right hand
side.
Exact finite sample solution


24
Analytical Solution for Bootstrapped Linear
Regression with Nonstochastic X
unknown error distribution
draw Bootstrap residuals, then get Y as
follows
25
e ( In X (XX)-1X)?

, , . . ., ?
e H e
, , . . ., ?
A ( In (1/n)1n1n )
EHH (1/n) 1n1n
No restrictions on covariance matrix for
errors.
26
Applied Econometrician

.
A In
or
where
A1n1n 0
1n1nA 0
A ( In (1/n)1n1n )
and
so
27
Econometric theorist

No restrictions on
where
28
Separability Condition
Definition Any bootstrap statistic, ,
that is a function of the elements of the set
f(Xj) j 1,,m and satisfies the
separability condition
where g(Mi ) and h( f(Xi )) are independent
functions and where the expected value EM
g(Mi) exists, is a directly analyzable
bootstrap statistic.
29
Even when the separability condition is violated,
such as in a ratio of sums or differences of
the bootstrapped sample values, we can still
get some insights from the exact analytical
approach.
For example, consider the block bootstrap
?
30
Fixed Block Bootstrap non-overlapping vs.
overlapping
Using T observations from the set of consecutive
times series matrices Yt, Xtt1,,T form k
matrix blocks, Bgg1,,k , each with m
consecutive time series observations. For the
non-overlapping bootstrap choose k and m such
that k m T , and, therefore, k T/m. For
the overlapping block bootstrap choose the same m
as in the non-overlapping block case, but define
k T - m 1. Next randomly select b blocks
with equal probability and with replacement from
the k blocks, Bgg1,,k , with corresponding
random frequencies Mgg1,,k to obtain the
bootstrap resampled blocks Bjj1,,b.
The ratio of the non-overlapping bootstrap
variance to the overlapping bootstrap variance of
the average of some function, of each of these j
1,,b randomly selected blocks is
31
Given k matrix blocks each with m observations
Variance of the kn T/m non-overlapping blocks
Variance of the ko T - m 1 overlapping
blocks
32
If separability condition cannot be satisfied,
use Taylor series expansion to produce
polynomial in bootstrap-induced randomness.
33
X is an n x 1 vector of original sample values.
X is an m x 1 vector of bootstrapped sample
values.
X HX where the rows of H are all zeros
except for a one in the
position corresponding to the
element of X that was randomly drawn.
EHH (1/n) 1m1n
where 1m and 1n are column vectors of ones.
?m g(X ) g(HX )
Taylor series expansion
Xo Ho X
Setup for empirical process
?m g(Xo) G1(Xo)(X -Xo) (1/2)
(X -Xo)G2(Xo)(X -Xo) R
34
Taylor series expansion
X HX where the rows of H are all
zeros except for a one in the position
corresponding to the element of X that was
randomly drawn.
?m g(X ) g(HX )
Taylor series
Xo Ho X
Ho EHH (1/n) 1m1n
?m g((1/n)1m1nX )
G1((1/n)1m1nX )(H-(1/n)1m1n) X (1/2)X
(H-(1/n)1m1n)G2((1/n)1m1nX )(H-(1/n)1m1n)
X R
Setup for analytical solution
Now ready to determine exact finite moments, et
cetera.
35
New formulation of Taylor series expansion
follows lexicographic ordering
where vec is the traditional vectorization
operator, and t is the transpose operator.
Ho EHH (1/n) 1m1n
36
Exploit lexicographic ordering of
and
to produce new version of Taylor series
expansion
Mo EMM ( m / n )
37
(No Transcript)
38
Correlation Coefficient
39

40

Second derivative of correlation coefficient
41
Paired Bootstrap Regression
Bootstrap Model
Original Model
Y Xb ?
Y Xb ?
YHY
?H?
XHX
42

43
This is the first of three papers
?
(1.) Bootstraps Finite Sample Distribution (
today !!! )
basically done.
(2.) Bootstrapped Asymptotically Pivotal
Statistics
almost done.
(3.) Bootstrap Hypothesis Testing and Confidence
Intervals