Reliability and Validity

1
Reliability and Validity

• Quality of Data

2
Are we testing what we think were testing?
3
Quantitative Data

• Reliability
• Validity
• Face
• External
• Internal

4
Reliability

• Implies that the same data would have been
  collected each time over repeated tests/
  observations.
• Would a particular technique (or survey question)
  yield the same result each time?
• Did you go to church last week? vs. How many
  times have you been to church in your life?
• Reliability does not ensure accuracy.
• Taken from Babbie, E.

5
Reliability

• Problem if interpret questions differently
• Poorly worded questions
• Inconsistent coding coding errors as with
  open-ended questions
• Lack of definition of key terms

6
Reliability

• Poorly worded Does the library have adequate
  facilities and equipment for physically disabled
  students
• Better Can patrons in wheelchairs retrieve books
  from the browsing collection?

7
Reliability (indicators)

• Pretest
• Repeat question(s)
• Test/retest
• Split half and Parallel
• Interscore or scorer

8
Validity

• A term to describe a measure that accurately
  reflects the concept it is intended to measure.
• Which is a more valid indicator of
  intelligence- an IQ score, or number of hours
  spent studying?
• Ultimate validity cannot be proven, but can be
  supported by face, internal, and external
  measures.
• Babbie, E.

9
Types of Validity

• Face validity The quality of an indicator/
  question/ test that makes it a reasonable measure
  of a variable.
• Church attendance is an indication of
  religiosity.
• Number of grievances filed is an indicator of
  worker morale

10
Internal Validity

• Approximate truth about inferences regarding
  causal relationships
• Typically applied to studies using inferential
  statistics (i.e. quantitative measures) than
  descriptive or observation studies.
• Especially useful for studies assessing affects
  of programs
• Only applicable to the study in question- not
  generalizable. Why not?
• Key question Whether observed changes can be
  attributed to your program (the cause) and NOT
  other possible impacts/ causes.

11
Internal Validity
Trochim, W.
12
Internal Validity

• History or specific events raises the issue that
  some variable other than the independent variable
  accounted for the change in the dependent
  variable. E.G. the length of time between
  conducting the pretest and posttest may have a
  detrimental effect.
• Maturation the change results from biological or
  psychological processes, which occurred over
  time, and not from the treatment itself.
  Maturation becomes more a concern the longer the
  period between the pretest and posttest
• Pretesting may affect the dependent variable.
  Pretesting may alert participants or educate them
  about the topic under investigation. Therefore
  if subjects are administered a posttest, their
  performance may reflect a marked improvement
• Measuring instruments or observational
  techniques Thesenot the treatmentmay account
  for the change in the dependent variable.
  Further, the validity of study findings may have
  been influenced by the fact that the evaluators
  as observers, raters, graders, interviewers, and
  coders gained experience, became tired, obtained
  a more complete understanding of the project, or
  eased their expectations of test subjects

13
Internal Validity (continued)

• A nonrandom assignment of subjects to groups may
  signify that the groups were dissimilar from the
  beginning. Therefore any change might be
  attributed to the differential selection of
  subjects, rather than the actual treatment.
• Statistical regression refers to the tendency for
  extreme scores to regress or move toward the
  common mean of subsequent measures. The
  assignment of subjects to a particular test group
  on the basis of extreme views may affect study
  findings.

14
Internal Validity (continued)

• Mortality refers to the possibility that some
  subjects may have dropped out of the study after
  completion of the pretest but before the
  administration of the posttest. In such
  instances, every effort should be made to
  identify any common patterns or characteristics
  to ensure that any difference between a groups
  pretest and posttest scores cannot be attributed
  to the loss of subjects.
• Interaction refers to the fact that more than one
  of the previous threats might be in play. This is
  especially likely in those cases where subjects
  were not randomly assigned to groups and the
  evaluation was based on existing, intact groups.

15
External Validity

• The approximate truth of generalizations drawn
  from a study.
• The degree to which conclusions drawn from your
  study sample would hold true to other persons in
  other places at other times
• Trochim, W.

16
External Validity
Trochim, W.
17
External Validity

• Example institutions of higher education in
  Massachusetts control, highest degree offered,
  and some characteristics of library (staff
  number, budget, and volume number)
• Return rate?
• Do respondents differ from non-respondents as a
  group?

18
Validity

• Content validity (for achievement test) How well
  does the test sample what the students learned?
  How well does a standardized test cover what was
  taught in the information literacy program?

19
Validity (continued)

• Criterion-related (predictive) (attitude test to
  predict performance in a library skills program)
  Who well does the test predict achievement for
  college freshmen?
• Criterion-related (diagnostic) How well does the
  test diagnose current problems with library use?

20
Validity (continued)

• Construct validity How well does the test
  measure comprehension of library use? Does a test
  on the use of an OPAC really measure effective
  and efficient use rather than ones ability to
  read test items?

21
Qualitative Study Equivalent

• Credibility
• Dependability
• Confirmability
• Transferrability

22
Qualitative Reliability

• Researcher is the instrument- how to test for
  reliability?
• Provide details of method, and abundance of
  evidence
• Provide evidence of qualifications as observer
• Make assumptions (and possible biases) clear
• State research questions clearly
• Use early stages of study to generate focus
• Observe for an adequate period of time, across a
  full range of activities
• Collect data from multiple sources
• Save data for reanalysis

23
Qualitative Validity

• Depends upon reliablity. Like reliability,
  asserted by documenting steps
• Triangulation- data from different sources/
  methods
• Full documentation of data- chain of evidence
• Logical connections between data and conclusions
• Conscious and deliberate inclusion of data that
  might not support thesis
• Preparedness to entertain alternatives
• Self-reflection, acknowledgement of own biases
• Review of preliminary reports by objective
  observers
• Awareness of limitations
• Gorman and Clayton

24
Qualitative Study Increasing Reliability and
Validty
Inquiry affected by Results in Account for by To lead to For findings that are
During After
Factor patternings Non-interpretability Prolonged engagement Persistent observation Peer Debriefing Triangulation Member checks Establish structural corroboration (coherence) Credibility Plausible
Situational Uniqueness Non-comparability Collect thick descriptive data Do theoretical/ positive sampling Develop thick description Transferability Context relevant
-Gorman and Clayton
25
Qualitative Study Increasing Reliability and
Validity contd
Inquiry affected by Results in Account for by To lead to For findings that are
During After
Instrumental changes Instability Use overlap methods Use stepwise replication Leave audit trail Do dependability audit (process) Dependability Stable
Investigator Predilections Bias Do triangulation Practice reflexivity (audit trail) Do confirmability audit (product) Confirmability Investigator-free
-Gorman and Clayton
26
Example

• For a sweeping study
• When conduct it?
• For how long?
• How deal with reliability and validity?

• Course evaluation

27
References

• Babbie, E. (2005). The basics of social
  research. Belmont, CA Wadsworth Publishing.
• Gorman, G.E. Clayton, P. (2005). Qualitative
  research for the information professional A
  practical handbook. London Facet Publishing.
• Trochim, W. M. K. (2006). Research methods
  knowledge base. Retrieved July 8, 2008 from
  http//www.socialresearchmethods.net/kb/intval.php
  
• http//www.socialresearchmethods.net/kb/external.p
  hp