The Theory of Sampling and Measurement

1
The Theory of Sampling and Measurement
2
Sampling

• First step in implementing any research design is
  to create a sample.
• We cannot study the theoretical population of all
  conceivable events (e.g., events that have not
  occurred), nor can we usually study all instances
  of actual events. We select some instances to
  study and not others. Those we include are our
  sample.
• How our sample is selected is critical for
  external validity or generalizability.

3
Groups in Sampling
Who do you want to generalize to?
4
Groups in Sampling
The theoretical population
5
Groups in Sampling
The theoretical population
What population can you get access to?
6
Groups in Sampling
The Theoretical Population
The study population
7
Groups in Sampling
The theoretical population
The study population
How can you get access to them?
8
Groups in Sampling
The theoretical population
The study population
The sampling frame
9
Groups in Sampling
The theoretical population
The study population
The sampling frame
Who is in your study?
10
Groups in Sampling
The theoretical population
The study population
The sampling frame
The sample
11
Types of Samples

• Probability Sampling
• Simple random
• Stratified random
• Cluster or area random

• Non-Probability Sampling
• Accidental
• Modal instance
• Expert
• Snowball
• Case study (intentional selection)

12
The Sampling Distribution
Average
Average
Average
...is the distribution of a statistic across an
infinite number of samples.
The sampling distribution...
13
Population Parameter
The population has a mean of 3.75...
1
5
0
...and a standard unit of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
This means
Self esteem
About 64 of cases fall between 3.5 - 4.0.
About 95 of cases fall between 3.25 - 4.25.
about 99 of cases fall between 3.0 - 4.5
14
Sampling Distribution
The population has a mean of 3.75.
1
5
0
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
Self-esteem
15
Sampling Distribution
The population has a mean of 3.75...
1
5
0
...and a standard error of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
Self-esteem
16
Inferring Population from Sample
The sample has a mean of 3.75...
1
5
0
...and a standard deviation of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
This means
Self esteem
64 chance true population mean falls between 3.5
- 4.0.
95 chance true population mean falls between
3.25 - 4.25.
99 chance true population mean falls between 3.0
- 4.5
17
Figure 3.4 Labor Repression and Growth in the
Asian Cases, 1970-1981

18
Figure 3.5 Labor Repression and Growth in the
Full Universe of Developing Countries,1970-1981
19
Measurement

• Operationalization is the process of translating
  theoretical constructs into observable
  indicators.
• Construct validity and reliability are the
  criteria we use to evaluate how well you have
  operationalized your concepts.
• Both matter regardless of the level of
  measurement and whether you are using qualitative
  or quantitative indicators.

20
The Hierarchy of Levels
Ratio
Absolute zero
Interval
Distance is meaningful
Ordinal
Attributes can be ordered
Nominal
Attributes are only named weakest
21
Nominal Measurement

• The values name the attribute uniquely.
• The name does not imply any ordering of the cases.

22
Ordinal Measurement

• When attributes can be rank-ordered
• Distances between attributes do not have any
  meaning.

23
Interval Measurement

• When distance between attributes has meaning, for
  example, temperature (in Fahrenheit) -- distance
  from 30-40F is same as distance from 70-80F
• Note that ratios dont make any sense -- 80F is
  not twice as hot as 40F.

24
Ratio Measurement

• Has an absolute zero that is meaningful
• Can construct a meaningful ratio (fraction), for
  example, number of clients in past six months

25
Construct Validity

• Key problem is that we have abstract theoretical
  construct power, democracy, development,
  corruption, etc. that we can never observe
  directly.
• Yet, to test propositions requires that we have
  some indicator for the construct or at least
  have proxies that we can argue are capturing some
  attributes of the construct.
• Our indicator is an analogy (to an analogy).

26
Assessing Construct Validity

• Translation Validity
• Face Validity plausible on its face
• Content Validity matches lists of attributes
• Criterion-related Validity
• Predictive Validity predicts accurately
• Concurrent Validity distinguishes appropriately
  between groups
• Convergent Validity
• Discriminant Validity

27
The Convergent Principle

• Alternative measures of a construct should be
  strongly correlated.

28
How It Works
Theory
You theorize that the items all reflect
self-esteem.
29
How It Works
Theory
1.00 .83 .89 .91 .83 1.00 .85 .90 .89 .85 1.00
.86 .91 .90 .86 1.00
The correlations provide evidence that the items
all converge on the same construct.
Observation
30
Convergent Validity in Measures of Democracy

• 1985 polity2 pollib civlib
  reg
• -------------------------------------------------
  
• polity2 1.0000 -0.9148 -0.8770
  -0.8601
• pollib -0.9148 1.0000 0.9176
  0.8440
• civlib -0.8770 0.9176 1.0000
  0.8053
• reg -0.8601 0.8440 0.8053
  1.0000

31
Convergent Validity in Measures of Education

• 1985 1 2 3 4 5 6
• -------------------------------------------------
  ------------------
• Ed. spending 1.0000 -0.1217 0.2415
  0.3563 0.0214 0.0195
• Illiteracy () -0.1217 1.0000 -0.5797
  -0.7306 -0.8569 -0.6196
• Cohort to Grade 4 0.2415 -0.5797 1.0000
  0.4419 0.6553 0.3654
• Grade School 0.3563 -0.7306 0.4419
  1.0000 0.6230 0.3612
• Secondary School 0.0214 -0.8569 0.6553
  0.6230 1.0000 0.7576
• College 0.0195 -0.6196 0.3654
  0.3612 0.7576 1.0000

32
The Discriminant Principle

• Measures of different constructs should not
  correlate highly with each other.

33
How It Works
Theory
Locus-of-control construct
LOC1
LOC2
34
How It Works
You theorize that you have two distinguishable
constructs.
Theory
Locus-of-control construct
LOC1
LOC2
35
How It Works
Theory
Locus-of-control construct
LOC1
LOC2
rSE1, LOC1 .12
The correlations provide evidence that the items
on the two tests discriminate.
rSE1, LOC2 .09
rSE2, LOC1 .04
rSE2, LOC2 .11
Observation
36
We have two constructs. We want to measure
self-esteem and locus of control.
Theory
Self-esteem construct
Locus-of-control construct
SE1
SE2
SE3
LOC1
LOC2
LOC3
For each construct, we develop three scale items
our theory is that items within the construct
will converge and Items across constructs will
discriminate.
37
Theory
Self-esteem Construct
Locus-of-control construct
Green and red correlations are Convergent yellow
are Discriminant.
SE1
SE2
SE3
LOC1
LOC2
LOC3
Observation
38
Theory
Self-esteem construct
Locus-of-control construct
SE1
SE2
SE3
LOC1
LOC2
LOC3
The correlations support both convergence and
discrimination, and therefore construct validity.
Observation
39
What Is Reliability?

• The repeatability of a measure
• The consistency of a measure
• The dependability of a measure

40
True Score Theory
Observed score
True ability
Random error


e
X

T
41
The Error Component
e
X

T
Two components
er

• Random error

es

• Systematic error

42
The Revised True Score Model
er
X
es


T
43
Random Error
Frequency
The distribution of X with no random error
X
44
Random Error
The distribution of X with random error
Frequency
The distribution of X with no random error
Notice that random error doesnt affect the
average, only the variability around the average.
X
45
Systematic Error
Frequency
The distribution of X with no systematic error
X
46
Systematic Error
The distribution of X with systematic error
Frequency
The distribution of X with no systematic error
Notice that systematic error does affect the
average we call this a bias.
X
47
If a Measure Is Reliable...
We should see that a persons score on the same
test given twice is similar (assuming the trait
being measured isnt changing).
X1
X2
48
If a Measure Is Reliable...
But, if the scores are similar, why are they
similar?
X1
X2
Recall from true score theory that...
T e1
T e2
49
If a Measure Is Reliable...
The only thing common to the two measures is the
true score, T. Therefore, the true score must
determine the reliability.
X1
X2
T e1
T e2
50
Reliability Is...
a ratio
variance of the true scores
variance of the measure
var(T)
var(X)
51
Reliability Is...
a ratio
variance of the true scores
variance of the measure
We can measure the variance of the observed
score, X. The greater the variance, the less
reliable the measure.
52
This Leads Us to...

• We cannot calculate reliability exactly we can
  only estimate it.
• Each estimate attempts to capture the
  consequences of the true score in different ways.

53
We want both Reliability and Validity
54
Reliability and Validity
Reliable but not valid
55
Reliability and Validity
Valid but not reliable
56
Reliability and Validity
Neither reliable nor valid
57
Reliability and Validity
Reliable and valid
58
Assignment 1

• Assess the validity and reliability of the IRIS-3
  International Country Risk Guide.
• Can examine a single instance, compare instances,
  analyze the full variation in the dataset,
  compare with additional measures, or use any
  other form of assessment. May use outside sources
  of data, history, or analysis (but document).
• The only restriction is that the paper must be
  empirical and examine issues of validity and
  reliability.
• 3-5 pages. Be concise.
• Due Monday 10/24 at beginning of class.