Reliability and Validity of Research Instruments

1
Reliability and Validity of Research Instruments

• An overview

2
Measurement error

• Error variance--the extent of variability in test
  scores that is attributable to error rather than
  a true measure of behavior.
• Observed Scoretrue score error variance
• (actual score obtained)
  (stable score) (chance/random
  error)
• (systematic error)

3
Validity

• The accuracy of the measure in reflecting the
  concept it is supposed to measure.

4
Reliability

• Stability and consistency of the measuring
  instrument.
• A measure can be reliable without being valid,
  but it cannot be valid without being reliable.

5
Validity

• The extent to which, and how well, a measure
  measures a concept.
• face
• content
• construct
• concurrent
• predictive
• criterion-related

6
Face validity

• Just on its face the instrument appears to be a
  good measure of the concept. intuitive, arrived
  at through inspection
• e.g. Conceptpain level
• Measureverbal rating scale rate your pain from
  1 to 10.
• Face validity is sometimes considered a subtype
  of content validity.
• Question--is there any time when face validity is
  not desirable?

7
Content validity

• Content of the measure is justified by other
  evidence, e.g. the literature.
• Entire range or universe of the construct is
  measured.
• Usually evaluated and scored by experts in the
  content area.
• A CVI (content validity index) of .80 or more is
  desirable.

8
Construct validity

• Sensitivity of the instrument to pick up minor
  variations in the concept being measured.
• Can an instrument to measure anxiety pick up
  different levels of anxiety or just its presence
  or absence? Measure two groups known to differ on
  the construct.
• Ways of arriving at construct validity
• Hypothesis testing method
• Convergent and divergent
• Multitrait-multimatrix method
• Contrasted groups approach
• factor analysis approach

9
Concurrent validity

• Correspondence of one measure of a phenomenon
  with another of the same construct.(administered
  at the same time)
• Two tools are used to measure the same concept
  and then a correlational analysis is performed.
  The tool which is already demonstrated to be
  valid is the gold standard with which the other
  measure must correlate.

10
Predictive validity

• The ability of one measure to predict another
  future measure of the same concept.
• If IQ predicts SAT, and SAT predicts QPA, then
  shouldnt IQ predict QPA (we could skip SATs for
  admission decisions)
• If scores on a parenthood readiness scale
  indicate levels of integrity, trust, intimacy and
  identity couldnt this test be used to predict
  successful achievement of the devleopmental
  tasks of adulthood?
• The researcher is usually looking for a more
  efficient way to measure a concept.

11
Criterion related validity

• The ability of a measure to measure a criterion
  (usually set by the researcher).
• If the criterion set for professionalism is
  nursing is belonging to nursing organizations and
  reading nursing journals, then couldnt we just
  count memberships and subscriptions to come up
  with a professionalism score.
• Can you think of a simple criterion to measure
  leadership?
• Concurrent and predictive validity are often
  listed as forms of criterion related validity.

12
Reliability

• Homogeneity, equivalence and stability of a
  measure over time and subjects. The instrument
  yields the same results over repeated measures
  and subjects.
• Expressed as a correlation coefficient (degree of
  agreement between times and subjects) 0 to 1.
• Reliability coefficient expresses the
  relationship between error variance, true
  variance and the observed score.
• The higher the reliability coefficient, the lower
  the error variance. Hence, the higher the
  coefficient the more reliable the tool! .70 or
  higher acceptable.

13
Stability

• The same results are obtained over repeated
  administration of the instrument.
• Test-restest reliability
• parallel, equivalent or alternate forms

14
Test-Retest reliability

• The administration of the same instrument to the
  same subjects two or more times (under similar
  conditions--not before and after treatment)
• Scores are correlated and expressed as a Pearson
  r. (usually .70 acceptable)

15
Parallel or alternate forms reliability

• Parallel or alternate forms of a test are
  administered to the same individuals and scores
  are correlated.
• This is desirable when the researcher believes
  that repeated administration will result in
  test-wiseness
• Sample I am able to tell my partner how I feel
• My partner tries to understand my feelings

16
Homogeneity

• Internal consistency (unidimensional)
• Item-total correlations
• split-half reliability
• Kuder-Richardson coefficient
• Cronbachs alpha

17
Item to total correlations

• Each item on an instrument is correlated to total
  score--an item with low correlation may be
  deleted. Highest and lowest correlations are
  usually reported.
• Only important if you desire homogeneity of
  items.

18
Spit Half reliability

• Items are divided into two halves and then
  compared. Odd, even items, or 1-50 and 51-100
  are two ways to split items.
• Only important when homogenity and internal
  consistency is desirable.

19
Kuder-Richardson coefficient (KR-20)

• Estimate of homogeneity when items have a
  dichotomous response, e.g. yes/no items.
• Should be computed for a test on an initial
  reliability testing, and computed for the actual
  sample.
• Based on the consistency of responses to all of
  the items of a single form of a test.

20
Cronbachs alpha

• Likert scale or linear graphic response format.
• Compares the consistency of response of all items
  on the scale.
• May need to be computed for each sample.

21
Equivalence

• Consistency of agreement of observers using the
  same measure or among alternate forms of a tool.
• Parallel or alternate forms (described under
  stability)
• Interrater reliability

22
Intertater reliability

• Used with observational data.
• Concordance between two or more observers scores
  of the same event or phenomenon.

23
Critiquing

• Was reliability and validity data presented and
  is it adequate?
• Was the appropriate method used?
• Was the reliability recalculated for the sample?
• Are the limitations of the tool discussed?