PS371003: Lecture 7

1
PS371-003 Lecture 7

• Significance Testing

2
Significance Testing

• Statistically significant Reliability
• Significance is a statistical term that tells how
  sure you are that a difference or relationship
  exists.
• IMPORTANT It doesn't mean the finding is
  important or that it has any decision-making
  utility.
• Great for exploratory research and hypothesis
  testing
• Should be reported along with measures of
  association (effect size)

3
Confidence

• Alpha (significance level) The probability of
  deciding to reject a null hypothesis when the
  null hypothesis is actually true in the
  population.
• Null hypothesis Says there is no effect
• Typically set small, so that the probability of
  this error will be low.
• Popular levels of significance are 5, 1 and
  0.1.
• When we pick an alpha level, we set an upper
  limit on the probability of making this erroneous
  decision, called a Type I error.
• Smaller a-levels give greater confidence in the
  determination of significance, but run greater
  risks of failing to reject a false null
  hypothesis (a Type II error, or "false negative
  determination"), and so have less statistical
  power.
• Confidence Level (1 Alpha) It is expressed as
  a percentage and represents how often the
  parameter value of the population would lie
  within the confidence interval (e.g. 95, 99,
  99.9).
• It tells you how sure you can be.
• Confidence Interval The range within the upper
  and lower bounds of the parameter being estimated
  that corresponds to the desired confidence level

4
Type I Error vs. Type II Error

• Two friends are walking in the subway late one
  evening when they spot a one hundred dollar bill
  down on the tracks. As they wrestle with the
  decision to jump down on the tracks to retrieve
  the money, the bill starts to flutter and tumble
  down the tracks away from them. They sense a wind
  coming from inside the tunnel and they hear the
  sounds of screeching metal. Not knowing if a
  train was approaching, they decide to stay on the
  platform and watch the bill disappear into the
  dark tunnel. One minute passes. Two minute
  passes. They then realize that they have made a
  Type I Error. That is, they concluded that the
  information had established that a train was
  approaching the platform when in fact there was
  no relationship (false positives e.g. rejecting
  the null hypothesis when it is true).
• The next day the same two friends are, once
  again, trolling the subway late at night.
  Miraculously, the one hundred dollar bill is back
  on the tracks below them. Just as before, the
  bill starts to flutter, a breeze overtakes the
  platform area, and the sound of metal on metal is
  deafening. This time, however, the boys ignore
  the information, jump onto the tracks, and grab
  the bill only to find that the lights of the
  oncoming train are now upon them. In this last
  moment, they realize that they have made a Type
  II Error. That is, they have concluded that there
  was not a relationship when there was one (false
  negatives e.g. failing to reject the null
  hypothesis when it is false)

5
Chi-Square

• When is it appropriate to use Chi-Square?
• It is used to investigate whether distributions
  of categorical variables differ from one another.
• e.g. establishes relationship between variables
• Issues
• It does not provide information on the direction
  of the association
• Partially a product of sample size
• Cramers V takes into account sample size
• Categories must be well-populated

6
T-Test

• When is it appropriate to use a T-test?
• Interval dependent and dichotomous independent
• Used to test hypothesis of different means
• Two groups are different
• One group is higher than the other
• Caution groups are not randomly assigned and
  therefore we are not sure if other variables are
  causing the differences.. We must introduce
  control variables..

7
Final Thoughts

• A common misconception is that a statistically
  significant result is always of practical
  significance, or demonstrates a large effect in
  the population
• When you have a large sample size, very small
  differences will be detected as significant.
• Statistical significance does not adequately
  address whether the results in a given study will
  replicate
• Effect sizes (measures of association) should be
  calculated and interpreted in all analyses.