Statistical Significance

1
Statistical Significance

• Clifford Holley
• Center for Population Studies
• Stolen From
• Tony Tersine, U.S. Census Bureau

2
Overview of the Session

• Basic Concepts
• Margin of Error
• Confidence Intervals
• Standard Error Formulas
• Statistical Testing

3
Basic Concepts - 1

• Sampling error is introduced due to sampling,
  selection of a subset of the population to draw
  inferences about the entire population.
  
• Standard error is an estimate of the precision of
  the estimates. It measures the variability of an
  estimate due to sampling.

4
Basic Concepts - 2

• The sampling error is often reported as the
  estimate plus or minus the margin of error, a
  measure of how precise the estimate is.
  
• The margin of error describes the precision of
  the estimate at a given confidence level.

5
Basic Concepts - 3

• The confidence level measures the likelihood that
  the true value is within the margin of error of
  the sample estimate.
  
• The Census Bureau statistical standard for
  published data is to use the 90 percent
  confidence level.

6
Margin of Error

• The margin of error is important because relying
  on statistical inference can save you from
  drawing incorrect conclusions from data based on
  a sample.
• It can help prevent you from interpreting small
  or nonexistent differences as important.

7
Margin of Error (MOE)

• MOE 1.65 Standard Error
• 1.65 is used for the 90 percent confidence
  level
  
• Standard Error MOE / 1.65

8
Confidence Interval

• Confidence IntervalEstimate Margin of Error
• 90 percent confidence levelMargin of Error
  1.65 Std Error
  
• 95 percent confidence levelMargin of Error
  1.96 Std Error

9
Confidence Interval

• The confidence interval tells you the upper and
  lower bounds of a range of values that may
  contain the true value.
  
• It provides important information about the true
  value or the population parameter.
  
• It tells you the limitations on using the
  estimates.

10
MOE / Confidence Interval

• Median Family Income 30,000
• Standard Error 1,500
• 90 MOE 1.65 1,500 2,475
• 90 CI 30,000 2,475
• 27,525 to 32,475

11
Standard Error Sum/Difference

• Standard Error of X Y or X Y
• SE(XY) SE(X-Y)

12
Standard Error Sum

• SE(X1X2Xn)

13
Standard Error Proportions

• P X / Y X is a subset of Y
• SE(P)

14
Standard Error Ratios

• X / Y X is not a subset of Y
• SE(X / Y)

15
Statistical Testing

• Two estimates are "significantly different" at
  the 90 percent confidence level if the difference
  between them is large enough to infer that there
  was a less than 10 percent chance that the
  difference was purely random.
• Users may want to compare estimates across years
  or geographies.
  
• It is important to note that small differences,
  which may be statistically significant, may not
  have any practical significance.

16
Statistical Testing - Steps

• State that two estimates are statistically
  different if the difference between the two
  estimates is statistically different from zero.
  
• Calculate the standard error of the difference.

17
Statistical Testing - Steps

• Calculate the margin of error of the difference.
• Compare the original difference between the
  estimates to the margin of error of the
  difference.

18
Statistical Testing - Steps

• If the difference is greater than the margin of
  error, then you conclude that the two estimates
  are significantly different.
  
• If the difference is less than the margin of
  error, you conclude that the two estimates are
  not significantly different.

19
Statistical Testing - Example

• Percent with Bachelors Degree or Higher
• Geography Percent MOE CI
• Area 1 20.0 5.0 15.0-25.0
• Area 2 12.3 4.7 7.6-17.0
• Difference 20.0 12.3 7.7

20
Statistical Testing - Example

• MOE of the Difference - Standard Errors for Each
  Estimate
  
• SE MOE / 1.65
• SE(Area 1) 5.0 / 1.65 3.03
• SE(Area 2) 4.7 / 1.65 2.85

21
Statistical Testing - Example

• Standard Error of the Difference
• Margin of Error of the Difference
• MOE(X - Y) 1.65 4.16 6.9

22
Statistical Testing - Example

• Compare the Difference to MOE
• Difference 7.7
• MOE 6.9
• Difference MOE
• Conclude that the two estimates are significantly
  different with 90 percent confidence

23
Rules to Remember

• Dont make a big deal of small differences.
• If the confidence intervals overlap you cannot
  conclude the difference is not statistically
  significant.
  
• Always talk to subject matter experts before
  making any conclusions.