Chapter 5: Developing a Measurement Strategy

1
Chapter 5 Developing a Measurement Strategy

• Scales of Measurement
• Reliability and Validity
• Modalities of Measurement
• Locating and Evaluating Measures

2
Reliability Validity

• Why is reliability important?
• Theory cant do without it
• Constructs cant be valid unless they are
  reliable
• A phenomenon must be reliably demonstrated before
  it can have construct status
• E.g. ESP, Loch Ness Monster
• What are some examples of phenomena important to
• Counselors?
• I/O psychologists?

3
Reliability Validity

• Manifest variables
• Directly observable
• Hypothetical Constructs
• Not directly observed (inferred)
• What are some examples?
• Behavioral?
• Psychological constructs?
• Which types (manifest/hypothetical) are
• Operational definitions?
• Hypothetical constructs?
• How are Operational definitions and Hypothetical
  constructs related?
• Operational definitions represent hypothetical
  constructs

4
Reliability Validityand measurement error

• Reliability Validity
• Reliability consistency
• Validity accuracy
• Polygraph
• Reliable and valid?
• Can something be reliable but not valid?
• Can something be valid but not reliable?
• Measurement Error
• Xo TE (Observed score True score Error)
• Randon v. systematic error
• What kind of error is introduced when an
  applicant fakes on a personality measure? Other
  examples?

5
Assessing Reliability

• Forms
• Test-retest
• Alternate forms (equivalent forms)
• Interrater reliability
• Internal consistency
• Split half
• Cronbachs alpha (coefficient alpha)
• mean r among all items
• Which one to use?
• Give some examples
• Standards for reliability
• Whats acceptable?

6
Assessing Validity

• Validity
• Inferred
• Depends on different types of evidence
• Varies in how much (degree)
• Trichotomized into high, moderate, low
• Specific to what its used for (valid for what?)
• refers to inferences drawn, not the measure
  itself
• A unitary construct (with three types of
  evidence)
• Content,
• criterion-related (predictive, concurrent)
• Construct

7
Validity

• Convergent
• Related to other constructs it should be related
  to
• Divergent
• Not related to other constructs it should not be
  related to
• which should be convergent and which divergent?
• IQ and Depression
• Work motivation and conscientiousness
• Happiness and wealth
• Need for approval and caring about how one looks
• Ability to sell and friendliness
• Aggression and frustration level

8
ValidityDetermining Degree of Validity

• Measure validation process
• See figure 5-4
• Theory of trait
• Test hypotheses
• Confirm/disconfirm
• Do it again
• Never ending (seemingly)
• Eg. CPI, WGCTA, WPT, NEO, Beck DI

9
Validity Differential

• Valid for specific population, group?
• Moral reasoning?
• Different cultures, Gender (Gilligan, 82)?
• Math tests?
• Gender?
• PONS test?
• Gender?
• Personality inventories?
• Fakers v. honest?
• Job setting v. home setting?

10
Reliability Validity Modalities of Measurement

• Self-Report Measures
• Advantages
• Limitations
• Behavioral Measures
• Advantages
• Limitations
• Physiological Measures
• Advantages
• Limitations
• Choosing a Measurement Modality

11
Modalities

• Self-Report
• Cognitive
• Affective
• Retrospective v. Hypothetical (behavioral
  intention)
• Kinesthetic
• Advantages
• Easy to collect
• Easy to administer
• Feelings (cannot observe)
• Thought process (Policy capturing)
• May be more accurate than observation. Why?
• Limitations
• Accuracy of recall
• Willingness to report
• Verbal skills needed

12
Self Report Modalities

• How to
• Ask about actual previous behaviors
• Use dichotomous or categorical for or behavior
  or behavioral intentions
• Use Likert (5, 7 or 9) for attitudes
• Include both positively and negatively scaled
  items
• Use graphic labels or benchmarks
• Use multiple items to capture a concept
• Avoid leading double barreled questions (usually)

13
ModalitiesBehavioral Measures

• Uses
• Behavior is object of study
• Operational definitions
• Nonverbal may be clue to feelings/ physio state
• Types of measures
• Frequency (bar presses)
• Rate
• Duration
• Intensity
• Accuracy
• Persistence
• Examples?

14
Behavioral Measures

• Advantages
• Surreptitious (sneaky)
• Avoids evaluation apprehension
• Capture automatic behaviors
• Often not premeditated
• More accurate than behavioral intention
• Limitations
• what you see is what you get What limitation is
  implied here?
• Interpretation is inferred by observer
• Highly situation-specific
• Thats why R. Hogan likes to use personality
  trait as predictor
• Need trained observers

15
Modalities Physiological Measures

• Purposes
• Of interest in and of itself
• As op definition of psychological state
• E.g. anxiety, arousal, lying
• Advantages
• Most direct (no intervening human observer
• Highly precise
• Limitations
• Need equipment, trained administrators
• Obtrusive
• Constrain freedom
• Source of unreliability (testing effects)
• E.g. polygraph

16
ModalitiesChoosing Locating Measurement
Modality

• Choosing Self-report, behavioral, physiological
• Rank them for level of validity
• Which one should be used?
• See table 5-2 relative advantages and limitations
• Multiple operationism
• Categories of Measures (figure 5-7, p. 147)
• Type manifest v. hypothetical
• Hypothetical construct
• Psychometric (used for individual scores)
• research measures (used for mean scores norms)
• Developed (use when it fits procedure)
• Ad hoc (when you have to develop it)

17
Modalities Locating Evlauting Measures

• Locating Measures
• MMY
• (Buros)
• Tests in Print
• Directory of Unpublished Experimental Measures
• (Goldman Sanders, 1997)
• Measures of Personality and Social Psychological
  Attitudes
• (Robinson, Shaver, Wrightsman, 91)
• Organizational Measures
• Warr, Cook and Wall??

18
Modalities Evaluating Them

• Theoretical Background
• How much construct standing?
• Quality of Development
• Participant samples, norms
• Reliability Validity
• What are good coefficients for each?
• Valid for what?
• Freedom from Response Bias
• Social desirability
• Acquiescence bias

19
Developing a Measurement StrategySummary

• Scales of Measurement
• Reliability and Validity
• Modalities of Measurement
• Locating and Evaluating Measures