Previously Learned Basic Terms

1
How to Assess the Quality of Your Measures

• Previously Learned Basic Terms
• Correlation
• Adding to Basic Terms with more new terms
• Variance explained, measurement error
• How the terms are related
• Brand New terms (not learned in Econ)
• What is reliability?
• How are non-Econ terms related to previously
  learned new terms
• Applying what you learned

2
What is a correlation?

• Reflect directions (/-) strength (0 to 1) of
  the relation between two variables
• E.g., height weight

3
Vince Carter
Tom Cruise
Julia Roberts
Calista Flockhart
4
r .76 r2 58
Vince Carter
Tom Cruise
Julia Roberts
Calista Flockhart
5
What is variance explained

• Variance explained
• Reflects the strength of relation of two
  variables
• Square of correlation
• Varies from 0 to 1
• Recall that correlation
• Reflect directions (/-) strength (0 to 1) of
  the relation between two variables

6
What is measurement error?

• A persons score on a measure can be due to their
  true standing on the measure and due to error in
  measurement
• E.g., Test scoretrue score measurement error
• The error in measurement is random
• E.g., The effects of temperature on measuring
  tape, whether object being measured is moving can
  all affect the measurement of height

7

• An example of no-measurement error
• When height is measured accurately objectively
  at two times
• e.g., with a measuring tape that is not
  susceptible to temperature changes, and the
  person being measured is not moving
• Then, correlation between the two measurements
  of height is 1 and the variance explained is 100

8
r 1.00 r2 100
9

• One example of Measurement Error
• E.g., on Correlations between objective and
  subjective reports of height
• Why would you ask subjective reports of such an
  objective thing?
• What may influence the measurement of subjective
  reports of height?

10
r .98 r2 96
11

• A second example of Measurement Error
• E.g., on Correlations between objective and
  subjective reports of weight
• What may influence the measurement of subjective
  reports of weight?

12
r .92 r2 85
13
Review whats next

• Some basic new terms
• What are
• Correlation? Variance explained? Measurement
  error?
• How does measurement error affect correlation?
  Variance explained?
• Whats next
• What if you cannot assess accurately and in an
  objective way?
• E.g., salary
• Assess it in multiple ways
• Compute the reliability of that assessment

14

• Example of assessing something in multiple ways
• Whether a person is in a dual career relationship
  (a student project)
• How to assess that? Class suggest ideas orally
• What may be random sources of measurement error?
  Class suggest ideas orally

15
What is reliability

• Reliability is assessing something consistently
  with multiple ways
• Whats next
• How is reliability different from correlation

16
Example of multiple correlations

• E.g., three TAs assessing 10 essays

17
(No Transcript)
18
Example of multiple correlations

• E.g., three TAs assessing 10 essays
• How many correlations can you get?
• ??
• Will the magnitude of correlations change
• With a decrease in the number of TAs?
• What changes with a decrease in the number of TAs
• What will change with an increase in the number
  of essays per TA?

19
Compute Correlations between raters

• Save data file from class website to your desktop
• Open SPSS
• Open the file on your desktop within SPSS
• Go to Analyze,
• Pick correlations,
• Pick bivariate

20
SPSS Screen you should arrive at when you pick
Analyze, Correlations, Bivariate
21
1. Choose all three variables in the Bivariate
menu2. Click middle button to move it to right
side, 3. Click ok to get output
22
Output file you should arrive at when you click
ok from bivariate menu choose all three
variables
23
Answer questions explore with the data

• How many correlations can you get with 3 TAs
  grading 10 essays
• Students answer orally
• Will the magnitude of correlations change
• With an increase in the number of TAs?
• Download file with more TAs to the data file
• With a decrease in the number of TAs?
• Download file with fewer TAs in the data file
• What will change with an increase in the number
  of essays per TA?
• Download file with more essays to the file

24
What can you get from multiple correlations?

• An assessment of how consistent one measure is
  with another
• i.e., the correlation of one measurement with
  another
• E.g., correlation between one TA with another
• It does not change with number of measurements
• E.g., does not change with the number of TAs
• You will get more correlations the more TAs you
  have
• It could change with the number of essays
• Dependsexplain in a later class

25
How is reliability different from multiple
correlation

• Reliability assesses the consistency of multiple
  measurements
• It is the average of the measurements
• Average is more accurate the more the number of
  measurements
• Can increase with the number of measurements
• E.g., 2 TAs gives you lower reliability than 3 TAs

26
Compute Reliability between Raters
27
1. Choose all three variables in the Reliability
menu2. Click middle button to move it to right
side, 3. Click ok to get output
28
Output file you should arrive at when you click
ok from reliability menu choose all three
variables
29
Answer questions explore with the data

• How many reliability coefficients can you get
  with 3 TAs grading 10 essays
• Students answer orally
• Will the magnitude of reliabilities change
• With an increase in the number of TAs?
• Use file with more TAs to the data file
• With a decrease in the number of TAs?
• Use file with fewer TAs in the data file
• What will change with an increase in the number
  of essays per TA?
• Use file with more essays to the file

30
Summary of differences b/w correlation
reliability

• Correlation assesses how consistent one measure
  is
• It is the correlation of one measurement with
  another
• It does not change with number of measurements
• Reliability assesses the consistency of multiple
  measurements
• It is based on the number of measurements
• It is the average of the measurements
• Increases with the number of measurements

31
Types of Reliability

• Inter-rater
• Consistency across raters
• Test-retest
• Consistency across time
• Internal
• Consistency across items
• Parallel forms
• Consistency across versions
• Split Half
• Consistency across halves of the scale

32
Inter-Rater Reliability
Compute reliability of TA1 TA2 TA3
33
Assessing life satisfaction with multiple items
The Satisfaction with Life Scale (SWLS) 1. In
most ways my life is close to ideal. 2. The
conditions of my life are excellent.3. I am
satisfied with my life.4. So far I have gotten
the important things I want in my life.5. If I
could live my life over, I would change almost
nothing. 1 2 3 4 5 6 7Strongly
StronglyDisagree Agree
34
Compute reliability of SWLS

• Use data set of IRE2002Y student responses to
  professor constructed survey
• Convert Excel spreadsheet to SPSS
• Compute reliabilities

35

• Internal Reliability of SWLS
• Satisfactory (if above .70)
• Means that participants respond similarly to
  items that are supposed to measure the same thing

36
Test-retest reliability

• Consistency of scores on the same measure taken
  at two different times
• Assumes no memory/learning effects within the two
  time intervals (or that error is random!)

37
Test-retest Reliability
38

• Example of Test-retest reliability
• Good test-retest reliability of SWLS
• Participants have similar scores at Time 1
  (beginning of semester) and at Time 2 (end of
  semester).
• Measurement error accounts for half of the
  variance in SWLS scores

39

• Why use test-retest reliability
• Retest reliability is useful for constructs
  assumed to be stable
• E.g., you wont apply test-retest reliability for
  current mood (e.g., how you feel right now)
  because it will show low-retest correlations, but
  that does not mean that the mood measure is not
  reliable

40
What you learned today

• Review of Previously Learned Terms
• What is correlation?
• Adding to Previously Learned Terms with more new
  terms
• Variance explained, measurement error
• How all these terms are related
• How measurement error affects correlations
  variance explained
• Brand New (non-Econ) terms
• What is reliability?
• How are brand new terms related to previously
  learned basic terms
• How is reliability related to correlation? How
  does measurement error affect reliability?
• Applying what you learned
• Computing correlations reliability with software

41
Types of reliability not covered in todays
lecture
42
Parallel Forms Reliability

• Consistency of scores on similar versions of the
  measure
• Assumes/checks that
• Forms are equivalent on mean, standard
  deviations
• Can have time interval between administration of
  the two forms and assumes that the time interval
  has a random effect on scores

43
Parallel forms Reliability
Pparticipant
Iitem
44
Split Half Reliability

• Correlation of scores on two halves of the
  measure
• Length of measure increases reliability

45
Types of Reliability
46
r .70 r2 49
47
Aspects of how measurement error works not
covered in todays lecture

• A persons score on a measure can be due to their
  true standing on the measure and due to error in
  measurement
• E.g., Test scoretrue score error

48
Assumptions of measurement error

• E.g., Test scoretrue score error
• The error in measurement is random
• E.g., Effects of learning, mood, changes in
  understanding etc.
• They have zero mean
• Are uncorrelated with each other
• Are uncorrelated with true score
• They are a constant part of the true score

49
Standard Error of Measurement

• SD of scores when a measure is completed several
  times by the same individual
• Mostly used in selection contexts
• Decide which of two individuals are hired
• Decide whether a test score is significantly
  higher/lower than a cutoff score

50
Correction for Attenuation

• Real correlation between two variables after
  removing unreliability of each measure
• Divide observed correlation by product of the
  square roots of individual reliabilities
• Note Selection research only controls for
  unreliability in criterion bec. we are more
  interested in the value of the predictor given a
  perfectly reliable criterion