MEASURING INSTRUMENTS and SURVEY DESIGN

1
MEASURING INSTRUMENTSand SURVEY DESIGN
2
Where to get an Instrument

• Make Your Own
• Requires Considerable Time Skill
• Use a Standardized Test
• Many are Published or Available for Purchase
• Still Requires that you Evaluate the Instrument

3
Features of all Instruments

• They are a means of measuring
• Knowledge/Skill
• Feeling
• Intelligence/Aptitude
• Validity (Does it test what it says?)
• Reliability (Is it the same each time?)

4
VALIDITY

• Valid for what and for whom
• Logical vs. Criterion-Related
• Types of Validity
• Content
• Construct
• Concurrent
• Predictive

5
Content Validity

• Degree to which an instrument measures an
  intended content area
• Includes
• Item Validity Right Subject Matter
• Sample Validity Full Range of
  Subject Area
• Face Validity (Expert Judgment)

6
Construct Validity

• Degree to which an instrument measures an
  intended hypothetical construct
• Whats a Construct?
• Model of the real world in psychology
• Examples IQ, job satisfaction, aggressiveness,
  optimism
• Check Test Against Hypotheses

7
Concurrent Validity

• Degree to which the scores on this instrument
  relate to scores on another established
  instrument use at the same time
• Helps Verify a Better Test
• Paper/Pen vs. Performance Test
• FAA Tests for AP
• Check One Against the Other

8
Predictive Validity

• Degree to which an instrument predicts how well
  an individual will do in a future situation
• Examples
• AFOQT, USAF Placement Tests
• SAT, ACT, GRE
• Check Test/Survey Against Outcomes (Correlation)

9
RELIABILITY

• How Consistent or Dependable is the Instrument?
• Do you get the same result each time?
• Coefficient Zero to 1.00 (1.00 perfect)
• Validity vs. Reliability
• A Valid Survey is Always Reliable
• A Reliable Survey is Not Always Valid

10
Instrument Reliability

• Test-Retest Reliability
• Do same survey again later
• Equivalent-Forms Reliability
• Do two equivalent surveys at the same time
• Split-Half Reliability
• Compare half of questions with other half

11
Instrument Reliability(Continued)

• Rational Equivalence Reliability
• Every combo of split-halves
• Scorer/Rater Reliability
• Used for subjective questions
• Interjudge Consistency between scorers
• Intrajudge Consistency of a scorer

12
WRITING YOUR OWN SURVEY INSTRUMENTOVERVIEW

• Types of Survey Questions
• Construction of Survey Questions
• Question Sequence
• Pitfalls of Survey Design
• Pitfalls in Questionnaire Construction
• Other Survey Elements

13
Types of Survey Questions

• Demographic (Facts)
• Age, Rank, Unit, Number of Children, etc.
• Achievement /Aptitude
• Past Knowledge vs. Potential Success
• Personality Profile
• Opinion Attitude

14
Building Survey Questions

• Closed-Ended or Open-Ended
• Check one of the Above verses Fill in the
  Blank
• Scales
• Likert Scales (SA lt---gt SD)
• Semantic-Differential Scale (Fair ltgt Unfair)
• Thurstone Scale (Select from Statements)
• Guttman Scale (Agree/Disagree)

15
Question Sequence

• Funnel Sequence
• More and more specific questions
• Helps subject remember, so its good for getting
  more detailed Information
• Inverted Funnel Sequence
• Do easy questions first
• Helps motivate subject (gets them into it)
• Contingency Questions (If yes, then...)

16
Pitfalls in Questionnaire Construction

• Hard to Understand Wording
• Response Set (Firewalling)
• Leading Questions
• Threatening Questions
• Double-Barreled Questions
• Beta Test your Survey!

17
Other Survey Elements

• Cover Letter
• Establishes importance of the study
• Establishes importance of the respondent
• Instructions
• Dont assume your subjects know how to fill out
  a survey
• Follow-ups (1st, 2nd, 3rd)

18
FINDING EXISTING SURVEY INSTRUMENTS

• Yearbooks
• Mental Measurements Yearbooks
• Reviews
• Professional Journals
• Research Associations
• i.e., Institute of Social Research