Validity and Reliability

1
Validity and Reliability

• Adel El-Atawy
• CSC426 Values in Computer Technology

2
Agenda

• Introduction
• Definition
• Types of Reliability and Validity
• Examples
• References
• Q A

3
Introduction - Data

• Truth ? Primary Data ? Secondary Data
• Leak of information should be minimized from
  level to level. We should keep the validity and
  reliability of the data.
• Some say Validity implies Reliability, but not
  the opposite

4
Definitions

• Reliability (or Precision) is how
  dependable/reproducible/repeatable the data is.
  How confident are we that it will generate the
  same result when the experiment is repeated.
  reliability is the consistency of a set of
  measurements or measuring instrument.
• Validity (or Accuracy) is how close the measures
  to the correct value (on the average). A valid
  measure is one which is measuring what it is
  supposed to measure.

5
Definitions

• Reliability Precision Low Variance
• Validity Accuracy Low Bias

6
Types of Reliability

• Inter-Rater or Inter-Observer ReliabilityUsed to
  assess the degree to which different
  raters/observers/measuring instruments give
  consistent estimates of the same phenomenon.
• Test-Retest ReliabilityUsed to assess the
  consistency of a measure from one time to
  another.
• Parallel-Forms ReliabilityUsed to assess the
  consistency of the results of two tests
  constructed in the same way from the same content
  domain.
• Internal Consistency ReliabilityUsed to assess
  the consistency of results across items within a
  test.

7
Types of Validity

• Face Validity
• Looks like it will closely model the system.
• E.g., Heuristics
• Content Validity
• By investigating the internals, it seems to be
  correct.
• E.g., An approximation model for the system.
• Criterion Validity
• When the data should measure something that is
  defined by another quantifiable metric.
• E.g., How stable the system is
• Crashes / (CPU Load processes)
• Construct Validity
• A measuring tool/model is not available. A
  hypothesis from the structure of reality is
  built, or an effect of the phenomenon is
  measured.
• E.g., Human relations and behaviors studies.

8
Example 1

• If you have three models/instruments to estimate
  a parameter in an experiment (e.g., temperature),
  which one to chose?
• Mercury Thermometer?
• Alcohol Thermometer?
• Electric Thermometer?
• How to describe the differences between them?
• How to obtain best estimates using one (or more)
  of them?

9
Example 2

• You have a new tunable algorithm. According to
  its parameters, it can be completely reliable but
  not valid, or completely valid but not reliable,
  or anywhere in between.
• How can you use this in reporting your research
  (e.g., in writing a paper)?

10
Example 3

• Measuring the second
• The unit used has to be defined as a part of the
  experiments parameters.
• Time cannot be measured with any degree of
  accuracy. We do not have a frame of reference.
• http//news.bbc.co.uk/2/hi/science/nature/4023777.
  stm
• A change in the definition of the second is very
  probable and will never stop.
• Precision can reach 1 second in billions of years.

11
Example 4

• What is the Labs temperature at noon in Summer
  (June-August).
• Options
• A single measure at noon in July 10th might have
  a considerable bias.
• Multiple Measures in the same day
• Parallel Measures in the same Day
• Multiple Measures in the different days

12
Example 4b

• Measuring in different days, will lower the
  variance (increase precision) till it reach the
  actual temperature variance.
• Getting a better (more accurate) thermometer will
  enhance the accuracy. Also, can be achieved by
  using more similar thermometers (if error is not
  systematic error in thermometer is not biased).

13
References

• http//www.socialresearchmethods.net/kb/relval.ht
  m
• Paul Leedy, Practical Research

14
Q A