Item Analysis

1
Item Analysis
2
Item Analysis

• Once data is gathered for a test, it is possible
  to study the items one-by-one to see which ones
  are contributing most to the test, and which ones
  are contributing the least
• This process, called item analysis, can be
  valuable for refining and improving a test
• Gives item means and variances (to look for floor
  and ceiling effects), and correlations of each
  item with the total test score, called item-total
  correlations

3
Floor and Ceiling Effects

• Floor effect For an item, scores pile up at
  the low end. Eg., on a Likert scale from 1 to 5,
  most people answer the item by responding with 1.
  Therefore, an item shows a floor effect if it has
  both a low mean and a very small std dev.
• Ceiling effect For an item, scores pile up at
  the high end. Eg., on a Likert scale from 1 to
  5, most people answer the item by responding with
  5. Therefore, an item shows a ceiling effect if
  it has both a high mean and a very small std dev.

4
Problem with Floor Ceiling Effects

• The problem with items that show floor or ceiling
  effects is that because of their extremely
  restricted range, they are more like constants
  than like variables.
• Items that are more like constants do not tell us
  much about individual differences.
• For example, items that are more like constants
  do not (and cannot) correlate well with other
  variables.

5
Identifying Floor Ceiling Effects

• A floor or ceiling effect is likely when all of
  the following occur
• The item has either a very low mean (floor
  effect) or a very high mean (ceiling effect)
• The item has a very low standard deviation
• The item has a near zero correlation with other
  items
• A frequency distribution of the item scores is
  quite skewed in the expected way

6
Item-Total Correlations (ITCs)

• Item-Total Correlations Correlations of each
  item with the total test score.
• Corrected Item-Total Correlations Correlations
  of each item with the total test score without
  this item in the total.
• Mean Inter-Item Correlation (homogeneity) The
  average of all the correlations between the items
  on a test (also known as rbar). Is an estimate
  of the reliability of a test that is 1-item long.
• Do not confuse item-total correlations (ITCs)
  with the mean inter-item correlation
  (homogeneity).

7
What does the homogeneity tell us?

• How well the items on a test hang together
• On average, how well the items on a test
  correlate with each other
• Gives us an estimate of the reliability of a test
  that is 1-item long
• Together with the total number of items on a
  test, homogeneity can be used to calculate
  coefficient alpha, the reliability of the test.

8
What Do ITCs Tell Us?

• Items with the highest ITCs contribute the most
  to the test homogeneity (and therefore to the
  test reliability)
• Items with ITCs of near 0 (or even negative ITCs)
  detract from homogeneity (and therefore depress
  reliability)
• With this item-by-item information, we can select
  and delete items to affect the overall
  reliability of a test