PS28C Psychometrics

1
PS28CPsychometrics

• Lecture 5
• Item Analysis

2
ITEM ANALYSIS

• A series of simple tools to improve the quality
  of a measure
• Four inter-related techniques
• Item difficulty
• Item discrimination
• Item-total correlation
• Item response curve

3
Item Difficulty Index

• Indicates if item is suitable for pool of test
  takers
• Need items which are neither too difficult or too
  easy for population
• Items which are either too difficult or too easy
  do not discriminate between test takers

4
Item Difficulty Index

• Item difficulty is the proportion of respondents
  who choose the correct option
• pi is proportion of respondents getting item
  right
• nc is the number of people getting correct answer
• N is the total number of people who took the test

5
Item Difficulty Index

• 100 people sit an exam
• 40 of 100 people answer item 10 correctly
• Item difficulty is 0.40

6
Item Difficulty Index

• 50 people sit an exam
• 30 of 50 people answer item 8 correctly
• Item difficulty is 0.60

7
Item Difficulty Index

• Smaller numbers indicate greater difficulty as
  few people pass the item
• Item difficulty is a misnomer
• Item difficulties help to create a test which can
  sort people of high ability from those of low
  ability

8
Item Difficulty Index

• Items with difficulties of 0 or 1 are not useful
• Everyone either passes or fails the item
• Item with maximum discriminating ability is one
  with difficulty of 0.50

9
Item Difficulty Index

• When item difficulty is 0.50 the variability in
  responses is greatest
• Any other item difficulty will produce a smaller
  variance

10
Item Difficulty Index

• Half the class gets an item right the other half
  gets it wrong

11
Item Difficulty Index

• One-quarter of the class get an item right the
  other three-quarters get it wrong

12
Item Difficulty Index

• Two-thirds of the class get an item right the
  other one-third get it wrong

13
Item Difficulty Index

• Need to include items with a range of item
  difficulties
• Average item difficulty should be 0.5
• Range of difficulty depends of homogeneity of
  test
• Homogenous tests need smaller range of item
  difficulties
• Heterogeneous tests require wider range of item
  difficulties

14
Item Discrimination Index

• Item discrimination index provides a measure of
  the ability of an item to distinguish between
  people of high and low ability
• Good item is when everyone who scored well on
  test got the item correct and all who scored low
  on the test got the item wrong

15
Item Discrimination Index

• Where
• di is the item discrimination index
• Ui is the of people with total scores in the
  upper range who also got item i correct
• Li is the of people with total scores in the
  lower range who also got item i correct
• niU is the number of people with total scores in
  the upper range of total test scores
• niL is the number of people with total scores on
  the test in the lower range of total test scores

16
Item Discrimination Index

• 1000 students write the GSAT
• Good score is top 25
• Poor score is bottom 25
• 250 students should have scores in top and bottom
  25 of all scores
• Item 33 passed by 200 students who scored in top
  25 and 50 students who scored in bottom 25

17
Item Discrimination Index
18
Item Discrimination Index

• Item 44 passed by 125 students who scored in top
  25 and 25 students who scored in bottom 25

19
Item Discrimination Index

• Larger values mean greater ability to distinguish
  high and low ability
• Index should be positive
• More high scoring test takers people got item
  correct than low scorers
• Negative index indicates more low ability people
  get item right than high ability people

20
Item-Total Score Correlations

• Good item should be strongly and positively
  correlated with total test scores
• Items with weak or no correlation with total
  scores should be discarded
• Items with negative correlations with total
  scores should be dropped

21
Item Characteristic Curves

• Graph of proportion of test takers with correct
  answer by level of total score
• Line should have upwards slope
• Positive slope indicates high ability people more
  likely to get item correct
• Negative slope indicates low ability people more
  likely to get item correct

22
Item Characteristic Curves
23
Item Characteristic CurvesItem 1
24
Item Characteristic CurvesItem 2
25
Item Characteristic CurvesItem 3
26
Item Characteristic CurvesItem 4
27
Item Characteristic CurvesItem 5
28
Item Characteristic Curves
29
Next Week

• Next week explore factor analysis
• Reading for factor analysis on course website
• Mid-Term Exam on March 15, 2006
• Covers all material including factor analysis
• May be multiple choice
• Will provide formula sheet