Making Systematic Observations

1
Making Systematic Observations

• Reliability, Validity, Scale of Measurements,
  Types of Measures, Reactivity in Research

2
Reliability

• Ability of a measure to produce similar results
  on repeated occasions.
• Physical Measures precision is important
• Population Estimates average values between
  samples (with reasonable margin of errors)
• Multiple Observers Interrater reliability
• Tests or Measures
• test-retest reliability parallel forms
  reliability split-half reliability

3
Accuracy of a measure

• Accuracy Does the measure produce results that
  agree with a known standard?
• Can be compromised by lack of precision, but can
  still be accurate if observations agree with the
  standard on average
• However precision is not necessarily accuracy
• Discrepancies between the standard value is
  called a bias.
• Ex False-Uniqueness Bias.
• The need for standardization for psychological
  measurements.

4
Validity

• Does the measure/test measure what it is intended
  to measure?
• Face validity judging by its appearance.
• Ex Math questions on math tests.
• Content validity Do items (content) represent
  the characteristic/behaviors that the measure was
  designed to look for?
• Ex An employment test that entails skills
  necessary for the particular job.
• Criterion-related validity Can the score of one
  measure be used to infer the value on some other
  measure? (two types concurrent validty
  predictive validity)

5
Validity (cont).

• Construct Validity Do high scores or low scores
  on a test indicate what they are supposed to as
  predicted by the theory?
• Ecological validity The extent to which results
  can be applied to the natural setting.
• Similar to external validity.

6
Four Scales of Measurements

• Scaling the data along four scales of
  measurement, depending on needs of research
  question.
• Nominal Scales Ex Yes or No
• Ordinal Scales Ex Not much, Moderately, A lot
• Interval Scales Ex thermometer
• Ratio Scale Ex Absolute Zero (in Kelvin Scale)
• Need can vary by the amount information you want
  and the need to preserve ecological validity.

7
Types of dependent measures

• Behavioral Measures Records actual behaviors of
  participants
• presence or absence of behavior
• Frequency of behavior
• latency and reaction time
• error behaviors (learning experiment)
• Physiological Measures Ex heart rate,
  respiration rate, brain activity, eye movement,
  etc.
• Self-report Measures Used to assess behaviors
  and attitudes, commonly using rating scales.

8
Reactivity in Research

• Design must consider how participants would react
  to the experimental situation
• Validity can be affected by Demand
  Characteristics
• defn Expected responses of the participant due
  to cues provided by the researcher and the
  context that may communicate the purpose of the
  study)
• Role attitude cues change in participants
  attitude to conform with role as a research
  participant (cooperative, apprehensive, negative
  attitudes)
• Experimenter bias expectancy effects
• Mitigating experimenter bias through single-blind
  and double-blind studies

9
Detecting and Correcting Problems

• Pilot study
• Clarifying instructions
• Minimize range effects
• Determining reliability and validity
• Manipulation checks Did the participants
  perceive the experiment in the manner in which it
  was intended?
• Validity of the independent variable
• Ask participants to report their perception of
  the purpose of the study, of the experimenter,
  and other factors that are important.