Outline

1
Session 7
2
Outline

• Chi-square Goodness-of-Fit Tests
• Fit to a Normal
• Using Crystal Ball
• Model Building
• Variable Selection Methods
• Minitab

3
Goodness-of-Fit Tests

• Determine whether a set of sample data have been
  drawn from a hypothetical population
• Same four basic steps as other hypothesis tests
  we have learned
• An important tool for simulation modeling used
  in defining random variable inputs

4
Example Warren Sapp
Financial analyst Warren Sapp wants to run a
simulation model that includes the assumption
that the daily volume of a specific type of
futures contract traded at U.S. commodities
exchanges (represented by the random variable X)
is normally distributed with a mean of 152
million contracts and a standard deviation of 32
million contracts. (This assumption is based on
the conclusion of a study conducted in 1998.)
Warren wants to determine whether this assumption
is still valid.
5
He studies the trading volume of these contracts
for 50 days, and observes the following results
(in millions of contracts traded)
6
(No Transcript)
7
Here is a histogram showing the theoretical
distribution of 50 observations drawn from a
normal distribution with µ 152 and s 32,
together with a histogram of Warren Sapps sample
data
8
The Chi-Square Statistic
9
Essentially, this statistic allows us to compare
the distribution of a sample with some expected
distribution, in standardized terms. It is a
measure of how much a sample differs from some
proposed distribution. A large value of
chi-square suggests that the two distributions
are not very similar a small value suggests that
they fit each other quite well.
10
Like Students t, the distribution of chi-square
depends on degrees of freedom. In the case of
chi-square, the number of degrees of freedom is
equal to the number of classes (a.k.a. bins
into which the data have been grouped) minus one,
minus the number of estimated parameters.
11
(No Transcript)
12
(No Transcript)
13
Note It is necessary to have a sufficiently
large sample so that each class has an expected
frequency of at least 5. We need to make sure
that the expected frequency in each bin is at
least 5, so we collapse some of the bins, as
shown here.
14
The number of degrees of freedom is equal to the
number of bins minus one, minus the number of
estimated parameters. We have not estimated any
parameters, so we have d.f. 4 1 0 3. The
critical chi-square value can be found either by
using a chi-square table or by using the Excel
function CHIINV(alpha, d.f.) CHIINV(0.05, 3)
7.815 We will reject the null hypothesis if
the test statistic is greater than 7.815.
15
Our test statistic is not greater than the
critical value we cannot reject the null
hypothesis at the 0.05 level of significance.
It would appear that Warren is justified in
using the normal distribution with µ 152 and s
32 to model futures contract trading volume in
his simulation.
16
The p-value of this test has the same
interpretation as in any other hypothesis test,
namely that it is the smallest level of alpha at
which H0 could be rejected. In this case, we
calculate the p-value using the Excel function
CHIDIST(test stat, d.f.) CHIDIST(7.439,3)
0.0591
17
Example Catalog Company
If we want to simulate the queueing system at
this company, what distributions should we use
for the arrival and service processes?
18
Arrivals
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Services
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Other uses for the Chi-Square statistic
The chi-square technique can often be employed
for purposes of estimation or hypothesis testing
when the z or t statistics are not appropriate.
In addition to the goodness-of-fit application
described above, there are at least three other
important uses for chi-square

• Tests of the independence of two qualitative
  population variables.
• Tests of the equality or inequality of more than
  two population proportions.
• Inferences about a population variance, including
  the estimation of a confidence interval for a
  population variance from sample data.

40
Model Building

• Ideally, we build a model under clean, scientific
  conditions
• Understand the phenomenon well
• Have an a priori theoretical model
• Have valid, reliable measures of the variables
• Have data in adequate quantities over an
  appropriate range
• Regression validates and calibrates the model,
  not discovers it

41
Unfortunately, we too often find ourselves Data
Mining

• Little understanding of the phenomenon
• No a priori theory or model
• Have data that may or may not cover all
  reasonable variables
• Have measures of some variables, but little sense
  of their validity or reliability
• Have data in small quantities over a restricted
  range
• We hope that regression uncovers some magical
  unexpected relationships
• This process has been referred to as Creative
  Regression Analytical Prospecting, or CRAP. This
  room is filled with horseshit there must be a
  pony in here somewhere.

42
The Model Building Problem
Suppose we have data available for n variables.
How do we pick the best sub-model
from yielding, perhaps There is no solution
to this problem that is entirely satisfactory,
but there are some reasonable heuristics.
43
Why Reduce the Number of Variables?

• Scientific Ideology In chemistry, physics, and
  biology, most good models are simple. The
  principle of parsimony carries over into social
  sciences, such as business analysis.
• Statistical Advantages Even eliminating
  significant variables that dont contribute
  much to the model can have advantages, especially
  for predicting the future. These advantages
  include less expensive data collection, smaller
  standard errors, and tighter confidence intervals.

44
Statistical Criteria for Comparing Models
45
Mallows Cp
Taking into account the possible bias that comes
from having an under-specified model, this
measure estimates the MSE including both bias and
variance If the model is complete (we have
the p terms that matter) the expected value of Cp
p. So we look for models with Cp close to p.
46
Variable Selection Algorithms
All-Subsets Forward Backward Stepwise Best Subsets
47
All-Subsets Regression
If there are p candidate independent variables,
then there are 2p possible models. Why not look
at them all? This is not really a major
computational problem, but can pose difficulties
in looking at all of the output. However, some
reasonable schemes exist for looking at a
relatively small subset of all the possible
models.
48
Forward Regression
Start with one independent variable (the one with
the strongest bivariate correlation with the
dependent variable), and add additional variables
until the next variable in line to enter fails to
achieve a certain threshold value. This can be
based on a minimum F value in the
full-model/reduced-model test, called FIN, or it
can be based on the last-in p-value for each
candidate variable. Forward selection is
basically the same thing as Stepwise, except
variables are never removed once they enter the
model. Set F to remove to zero. The procedure
ends when no variable not already in the model
has an F-stat greater than FIN.
49
Backward Regression
Start with all of the independent variables, and
eliminate them one by one (on the basis of having
the weakest t-stat) until the next variable fails
to meet a minimum threshold. This can be an F
criterion called FOUT, or a p-value
criterion. Backwards elimination starts with all
of the independent variables, then removes them
one at a time based on the stepwise procedure,
except that no variable can re-enter once it has
been removed. Set FIN at a very large number such
as 100,000 and list all predictors in the Enter
box. The procedure ends when no variable in the
model has an F-stat less than FOUT.
50
Stepwise Regression
An intelligent mixture of forward and backward
ideas. Variables can be entered or removed using
FIN and FOUT criteria or p-value criteria.
51
The F Criterion
The basic (default) method of stepwise regression
calculates an F-statistic for each variable in
the model. Suppose the model contains X1, ... ,
Xp. Then the F-statistic for Xi is with 1
and n - p - 1 degrees of freedom. If the
F-statistic for any variable is less than F to
remove, the variable with the smallest F is
removed from the model. The regression equation
is calculated for this smaller model, the results
are printed, and the procedure proceeds to a new
step.
52
If no variable can be removed, the procedure
attempts to add a variable. An F-statistic is
calculated for each variable not yet in the
model. Suppose the model, at this stage, contains
X1, ... , Xp. Then the F-statistic for a new
variable, Xp1 is The variable with the
largest F-statistic is then added, provided its
F-statistic is larger than F to enter. Adding
this variable is equivalent to choosing the
variable with the largest partial correlation or
to choosing the variable that most effectively
reduces SSE. The regression equation is then
calculated, results are displayed, and the
procedure goes to a new step. If no variable can
enter, the stepwise procedure ends. The p-value
criterion is very similar, but uses a threshold
alpha value.
53
Best Subsets
A handy procedure that reports, for each number
of independent variables p, the model with the
highest R-square. Best Subsets is an efficient
way to select a group of "best subsets" for
further analysis by selecting the smallest subset
that fulfills certain statistical criteria. The
subset model may actually estimate the regression
coefficients and predict future responses with
smaller variance than the full model using all
predictors.
54
Using Minitab
Excels regression utility is not well suited to
iterative procedures like this. More
stats-focused packages like Minitab offer a more
user-friendly method. Minitab treats Forward and
Backward as subsets of Stepwise. (This makes
sense they really are special cases where
entered variables cant leave, or removed
variables cant re-enter. Minitab 13 uses the
p-value criterion by default.
55
Example Rick Beck
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
Summary

• Chi-square Goodness-of-Fit Tests
• Fit to a Normal
• Using Crystal Ball
• Model Building
• Variable Selection Methods
• Minitab

67
For Session 8

• Cars (B)
• Include dummy variables
• Beware Perfect multicollinearity!
• Artsy case