MEASUREMENT ISSUES

1
MEASUREMENT ISSUES

• I. INFORMATION GATHERING
• II. MEASUREMENT
• III. RELIABILITY
• IV. VALIDITY
• V. OTHER ATTRIBUTES

2
Gathering Information

• What to look for
• Hypotheses Variables
• Whats known?
• Anything missing?
• Measures
• How Operationalized
• Reliability/Validity evidence
• Sources of data

3
Gathering Information

• What to look for
• Where to look
• Using what you find

4
MEASUREMENT

• Nature of Measurement

5
MEASUREMENT

• Levels of Measurement
• Nominal
• Ordinal
• Interval
• Ratio

6
Level of Analysis

• Is the variable of interest an individual, team,
  or company (etc.) phenomenon?
• Examples
• Individual - Pay satisfaction, Turnover
  intentions, Product knowledge
• Team - Norms, Cohesion, Decision quality,
• Company - Climate, Turnover rate, Market share

7
RELIABILITY

• Nature
• Observed Score
• True Score Error

8
Sources of Error

• Respondent
• Administrator
• Conditions
• Coder/Scorer
• Attribute

9
RELIABILITY

• Nature
• Sources of Error
• Systematic vs. Random Error

10
Random Error

(Trochim Fig. 3-7 on p. 61)
11
Systematic Error

(Trochim Fig. 3-8 on p. 62)
12
Test-Retest Reliability

• Stability over time
• Assumes stable construct

(Trochim Fig. 3-19 on p. 67)
13
Parallel-Forms Reliability

• Stability across forms

(Trochim Fig. 3-20 on p. 67)
14
Split-Half Reliability

• Consistency within a measure

(Trochim Fig. 3-23 on p. 69)
15
Split-Half Example

• Navigation Dimension of Socialization Scale
• 12. I have learned how to effectively deal with
  the bureaucracy or red tape in the MLHR
  program.
• 15. I know how to act and behave in the MLHR
  program.
• 16. I have not learned what it really takes to
  get ahead in the MLHR program. (R)
• 22. I have learned the unwritten rules in the
  MLHR program for how to stay in good standing.
• 25. I know what is expected of me in the MLHR
  program beyond my performance in classes.
• Correlation of even items with odd items

16
Estimating Reliability

• Cronbachs Alpha Reliability
• Like the average of all possible split-half
  correlations.
• Calculated as

17
Alpha Examples

• SCALE ALPHA
• Goal Commitment
• Socialization
• People
• Navigation
• Language
• Engagement
• Work Motivation
• Job Satisfaction
• Turnover Intentions

18
Inter-Rater Reliability

• Consistency across observers

(Trochim Fig. 3-18 on p. 66)
19
What is Reliable Enough?
20
VALIDITY

• To what extent do your inferences or conclusions
  reflect the truth?
• Are you measuring the right thing?
• Its better to measure the right thing with some
  error than to perfectly measure the wrong thing.

21
(No Transcript)
22
CONSTRUCT VALIDITY

• Can we generalize to the constructs?

(Trochim Fig. 3-1 on p. 53)
23
CONSTRUCT VALIDITY
24
CONSTRUCT VALIDITY

• Example - Goal Commitment Scale
• Across multiple studies, settings, samples,
  the scale has consistently been shown to be
• Reliable (alpha test-retest)
• Related to its expected causes
• Related to its expected effects
• Unrelated to unexpected effects

25

CRITERION-RELATEDVALIDITY
INDEPENDENT VARIABLE
DEPENDENT VARIABLE
Observed Relationship
Observation
What you test
26
CRITERION-RELATEDVALIDITY
27
CRITERION-RELATEDVALIDITY
28
CRITERION-RELATEDVALIDITY
29
CONTENT VALIDITY
30
CONTENT VALIDITY
31
VALIDITY
32
CASE - Phonemin Co.

• How was reliability estimated?
• Are the tests reliable enough?
• How were thy validated?
• Are the tests valid enough?
• Limitations to keep in mind?

33
Internal Validity

• How confident are we that the conclusions reached
  in this study reflect the truth? Given our
  research design and measures, how confident are
  we in our results?
• Did the independent variable variable cause the
  dependent variable or are there potential
  alternative explanations?

34
External Validity

• How confident are we that the conclusions reached
  in this study would also be true in other
  contexts?
• Can we generalize our findings to other persons,
  places, settings and times?

(Trochim Fig. 2-2 on p. 28)
35
OTHER ATTRIBUTES

• NORMS
• LEGALITY
• RESPONDENT REACTIONS