Reliability

1
Reliability

• Consistency
• Test Scores Error
• XTE
• As T goes up E goes down, reliability increases
• Variance Error Variance

2
Sources of Error

• Test Construction/Content
• Sampling finite number of questions
• Poorly written questions
• Test Administration
• Error related to the test taker
• Error related to the test environment
• Error related to the examiner

3
Sources of Error (cont.)

• Test scoring interpretation
• Objective v. subjective
• scoring rubrics

4
Parallel Tests

• Theoretical underpinning of reliability
• Similar content
• Same true score same error variance
• Theoretical, not produced in reality
• Not to be confused with alternate forms
• Reliability can be defined as the correlation
  between 2 parallel tests

5
Types of Reliability

• Reliability over time
• Internal consistency/reliability
• Inter-rater reliability

6
Reliability over time

• Test-retest reliability
• Obtained by correlating pairs of scores from the
  same sample on two different administrations of
  the same test
• Error related to passage of time intervening
  factors
• Alternate-Form (Immediate Delayed)
• Error related to time content

7
Internal Consistency

• Split-half
• Divide the test into two equivalent halves
• Odd-even
• Randomly assign items
• Divide by equivalency of items
• Calculate r between 2 halves
• Correct with Spearman-Brown
• Allows estimation of reliability of test that has
  been shortened or lengthened

8
Internal Consistency (cont.)

• Inter-item consistency
• Index of homogeneity of test degree to which all
  items measure same construct
• Desirable aids in interpretation of test (as
  opposed to homogeneity of groups)

9
Internal Consistency (cont.)

• Kuder-Richardson formulas
• KR-20 statistic of choice for determining
  reliability of tests with dichotomous items
  (right-wrong)
• KR-21 can be used if assumption that all items
  are of similar difficulty

10
Internal Consistency (cont.)

• Cronbachs coefficient alpha
• Function of all items on test the total test
  score
• Each item conceptualized as a test
• 36-item test, 36-parallel tests
• In addition to use with dichotomous tests can be
  used with tests containing nondichotomous items,
  e.g., opinion, tests which allow partial credit

11
Inter-rater reliability

• How well do 2 raters/judges agree?
• Correlation between scores from 2 raters
• Percentage of agreement percentage of intervals
  where both raters agreed behavior occurred
• Kappa

12
Factors influencing reliability

• Length of test
• Longer tests increase percentage of domain that
  can be sampled
• Point of diminishing returns
• Homogeneity of items
• Measure same construct easier to interpret
• Dynamic or static characteristics

13
Factors influencing reliability (cont.)

• Homogeneity of sample
• Restriction of range
• If sample is homogenous then any observed
  variance must be error
• Power v. Speed tests
• Speed use test-retest alternate forms split
  half from 2 separately timed half tests
• Internal consistency not applicable
• Speed tests easy internal consistency inflates
  reliability

14
Reliability of Individual Scores

• How much error is in an individual score?
• How much confidence do we have in a particular
  score?
• Standard Error of Measurement
• Extent to which one individuals scores vary over
  tests that are presumed to be parallel
• Assume error is distributed normally
• Where is the individuals true score?

15
Standard Error of Measurement
16
SEM (cont.)

• Odds are 68 that true score falls within plus
  or minus 1 SEM.
• Odds are __ that true score falls within plus
  or minus 2 (1.96) SEM.
• Odds are __ that true score falls within plus
  or minus 3 SEM.
• WHAT IS THE RELATIONSHIP BETWEEN RELIABILITY
  SEM?

17
Standard Error of the Difference of Two Scores

• Compare test takers performance on two different
  tests
• Compare two test takers on the same test
• Compare two test takers on two different tests

18
Standard Error of the Difference
19
Standard Error of the Difference

• Set confidence intervals for difference scores
• Difference scores contain error from both of the
  comparison measures.
• Difference scores are less reliable than scores
  from individual tests.

20
Test-retest reliabilitySocial Interaction
Self-Statement

• r12 .99
• r-1-2 .99
• r1-1 -.45
• r1-2 -.55
• r2-1 -.47
• r2-2 -.56