Science Behind the CAT

1
Science Behind the CAT

• Dennis L. Hart, PhD, PT
• Director of Consulting and Research
• Focus On Therapeutic Outcomes, Inc.
• November 10, 2006

2
What is Computerized Adaptive Testing (CAT)

• The next level in patient surveys. Evolved from
  the need to be more efficient and precise with
  our measures of function, and emphasize one
  patient at a time at the bedside.

3
When You Ask A Question

• The Answer you get is Predicated on Two Concepts
• the difficulty of the question
• the ability of the person

Calculate from data
Measure with questions
4
Efficiency vs Precision
How do you balance?

• Precision
• Ask items close to the persons ability
• Ask items that provide maximum information given
  the persons ability
• Efficiency
• Best estimate of persons ability with as few
  items as possible

5
Operational Characteristics of the CAT

• Starting item or set of items
• Ability estimation routine
• Item selection routine
• Stopping rules

Heart of the CAT
Built on IRT Mathematics
6
Basics of Item Response Theory (IRT)

• Family of mathematical models found useful in the
  design, construction, and evaluation of
  educational and psychological tests.
• Performance of a patient on a test item can be
  predicted by a set of traits, latent traits or
  abilities (ie, function).

7
Lets Ask a Question (Item)

• How much difficulty would you have using your
  affected arm to reach a shelf that is shoulder
  height?
• No difficulty
• Little difficulty
• Some difficulty
• Much difficulty
• I cant do this

Body Part Specific! Shoulder CAT
Hart DL, et al. J Clin Epidemiol.
200659(3)290-298.
8
-6 -4 -2 0 2
4 6 ----------------------------------
-------------------- Item 1
1 2 3 4 5 5 BULB 1
1 2 3 4
5 5 BCKSEATRE 1 1
2 3 4 5 5 OVRHEADSHE 1
1 2 3 4 5
5 MEDBOX 1 1 2
3 4 5 5 BED 1
1 2 3 4 5 5
OVRHEAD1 1 1 2 3
4 5 5 SKILLET 1 1
2 3 4 5 5
STRING 1 1 2 3 4
5 5 COLLAR 1 1
2 3 4 5 5 BALLUND 1
1 2 3 4 5
5 POTATOES 1 1 2
3 4 5 5 SLIDE 1
1 2 3 4 5
5 COMB 1 1 2 3 4
5 5 POCKET 1 1
2 3 4 5 5 SHELF 1
1 2 3 4 5
5 SAFESTRP 1 1 2
3 4 5 5 PULLBOX 1
1 2 3 4 5
5 WHEELSA 1 1 2 3 4
5 5 SHOULDER1 1
1 2 3 4 5 5
CROOK 1 1 2 3 4
5 5 JAR 1 1 2
3 4 5 5 PULLCHAIR 1
1 2 3 4 5
5 TIE 1 1 2 3 4
5 5 PUSHCHAIR 1
1 2 3 4 5 5
CHEST 1 1 2 3 4
5 5 JARSTEADY 1 1
2 3 4 5 5 SALT 1
1 2 3 4 5
5 DEODORANT 1 1 2 3
4 5 5 WATER 1
1 2 3 4 5
5 TABLE 1 1 2 3 4
5 5 UNDERPANTS 1 1
2 3 4 5 5
SOCKSON 1 1 2 3 4 5
5 EAROPP 1 1 2
3 4 5 5
FACESA 1 1 2 3 4 5
5 FAUCETSA 1 1 2
3 4 5 5
FLUSHING 1 1 2 3 4 5
5 GLASSES ------------------
------------------------------------ -6
-4 -2 0 2 4
6
Shoulder IRT Probability Plot
Item Difficulty
SHELF
Ability
9
-6 -4 -2 0 2
4 6 ----------------------------------
-------------------- Item 1
1 2 3 4 5 5 BULB 1
1 2 3 4
5 5 BCKSEATRE 1 1
2 3 4 5 5 OVRHEADSHE 1
1 2 3 4 5
5 MEDBOX 1 1 2
3 4 5 5 BED 1
1 2 3 4 5 5
OVRHEAD1 1 1 2 3
4 5 5 SKILLET 1 1
2 3 4 5 5
STRING 1 1 2 3 4
5 5 COLLAR 1 1
2 3 4 5 5 BALLUND 1
1 2 3 4 5
5 POTATOES 1 1 2
3 4 5 5 SLIDE 1
1 2 3 4 5
5 COMB 1 1 2 3 4
5 5 POCKET 1 1
2 3 4 5 5 SHELF 1
1 2 3 4 5
5 SAFESTRP 1 1 2
3 4 5 5 PULLBOX 1
1 2 3 4 5
5 WHEELSA 1 1 2 3 4
5 5 SHOULDER1 1
1 2 3 4 5 5
CROOK 1 1 2 3 4
5 5 JAR 1 1 2
3 4 5 5 PULLCHAIR 1
1 2 3 4 5
5 TIE 1 1 2 3 4
5 5 PUSHCHAIR 1
1 2 3 4 5 5
CHEST 1 1 2 3 4
5 5 JARSTEADY 1 1
2 3 4 5 5 SALT 1
1 2 3 4 5
5 DEODORANT 1 1 2 3
4 5 5 WATER 1
1 2 3 4 5
5 TABLE 1 1 2 3 4
5 5 UNDERPANTS 1 1
2 3 4 5 5
SOCKSON 1 1 2 3 4 5
5 EAROPP 1 1 2
3 4 5 5
FACESA 1 1 2 3 4 5
5 FAUCETSA 1 1 2
3 4 5 5
FLUSHING 1 1 2 3 4 5
5 GLASSES ------------------
------------------------------------ -6
-4 -2 0 2 4
6
Shoulder IRT Probability Plot
Item Difficulty
Low Ability Estimate
SHELF
Ability
10
-6 -4 -2 0 2
4 6 ----------------------------------
-------------------- Item 1
1 2 3 4 5 5 BULB 1
1 2 3 4
5 5 BCKSEATRE 1 1
2 3 4 5 5 OVRHEADSHE 1
1 2 3 4 5
5 MEDBOX 1 1 2
3 4 5 5 BED 1
1 2 3 4 5 5
OVRHEAD1 1 1 2 3
4 5 5 SKILLET 1 1
2 3 4 5 5
STRING 1 1 2 3 4
5 5 COLLAR 1 1
2 3 4 5 5 BALLUND 1
1 2 3 4 5
5 POTATOES 1 1 2
3 4 5 5 SLIDE 1
1 2 3 4 5
5 COMB 1 1 2 3 4
5 5 POCKET 1 1
2 3 4 5 5 SHELF 1
1 2 3 4 5
5 SAFESTRP 1 1 2
3 4 5 5 PULLBOX 1
1 2 3 4 5
5 WHEELSA 1 1 2 3 4
5 5 SHOULDER1 1
1 2 3 4 5 5
CROOK 1 1 2 3 4
5 5 JAR 1 1 2
3 4 5 5 PULLCHAIR 1
1 2 3 4 5
5 TIE 1 1 2 3 4
5 5 PUSHCHAIR 1
1 2 3 4 5 5
CHEST 1 1 2 3 4
5 5 JARSTEADY 1 1
2 3 4 5 5 SALT 1
1 2 3 4 5
5 DEODORANT 1 1 2 3
4 5 5 WATER 1
1 2 3 4 5
5 TABLE 1 1 2 3 4
5 5 UNDERPANTS 1 1
2 3 4 5 5
SOCKSON 1 1 2 3 4 5
5 EAROPP 1 1 2
3 4 5 5
FACESA 1 1 2 3 4 5
5 FAUCETSA 1 1 2
3 4 5 5
FLUSHING 1 1 2 3 4 5
5 GLASSES ------------------
------------------------------------ -6
-4 -2 0 2 4
6
Shoulder IRT Probability Plot
Item Difficulty
Average Ability Estimate
Ability
11
-6 -4 -2 0 2
4 6 ----------------------------------
-------------------- Item 1
1 2 3 4 5 5 BULB 1
1 2 3 4
5 5 BCKSEATRE 1 1
2 3 4 5 5 OVRHEADSHE 1
1 2 3 4 5
5 MEDBOX 1 1 2
3 4 5 5 BED 1
1 2 3 4 5 5
OVRHEAD1 1 1 2 3
4 5 5 SKILLET 1 1
2 3 4 5 5
STRING 1 1 2 3 4
5 5 COLLAR 1 1
2 3 4 5 5 BALLUND 1
1 2 3 4 5
5 POTATOES 1 1 2
3 4 5 5 SLIDE 1
1 2 3 4 5
5 COMB 1 1 2 3 4
5 5 POCKET 1 1
2 3 4 5 5 SHELF 1
1 2 3 4 5
5 SAFESTRP 1 1 2
3 4 5 5 PULLBOX 1
1 2 3 4 5
5 WHEELSA 1 1 2 3 4
5 5 SHOULDER1 1
1 2 3 4 5 5
CROOK 1 1 2 3 4
5 5 JAR 1 1 2
3 4 5 5 PULLCHAIR 1
1 2 3 4 5
5 TIE 1 1 2 3 4
5 5 PUSHCHAIR 1
1 2 3 4 5 5
CHEST 1 1 2 3 4
5 5 JARSTEADY 1 1
2 3 4 5 5 SALT 1
1 2 3 4 5
5 DEODORANT 1 1 2 3
4 5 5 WATER 1
1 2 3 4 5
5 TABLE 1 1 2 3 4
5 5 UNDERPANTS 1 1
2 3 4 5 5
SOCKSON 1 1 2 3 4 5
5 EAROPP 1 1 2
3 4 5 5
FACESA 1 1 2 3 4 5
5 FAUCETSA 1 1 2
3 4 5 5
FLUSHING 1 1 2 3 4 5
5 GLASSES ------------------
------------------------------------ -6
-4 -2 0 2 4
6
Shoulder IRT Probability Plot
Item Difficulty
High Ability Estimate
Ability
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
What exactly is the PROMIS developing and when
can I get it?

• Bryce B. Reeve, Ph.D.
• NIH Science Officer, PROMIS
• Program Director,
• National Cancer Institute, NIH

18
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
19
How often did you feel nervous?
Some of the time
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
20
How often did you feel nervous?
Some of the time
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
21
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
22
How often did you feel hopeless?
Some of the time
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
23
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
24
How often did you feel worthless?
Little of the time
-3
-2
-1
0
1
2
3
Severe
moderate
low
very low
high
Emotional Distress
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
25
How often did you feel worthless?
Little of the time
-3
-2
-1
0
1
2
3
Target in on emotional distress score
Severe
moderate
low
very low
high
Item Bank (Validated IRT-Calibrated Emotional
Distress Items)
26
IRT Information Summary

• For a given estimate of patient ability, ask an
  item that provides the maximum amount of
  information about the patients ability!
• If you ask a set of items that maximizes the
  amount of information for the estimates of
  patient ability, you will get the best final
  estimate of patient ability efficiently!

27
2 CAT Item Selection Routines

• Item with difficulty parameter close to ability
  of patient Closest Scale Value
• Item with maximum information given the current
  estimate of the patients ability Maximum
  Information
• For a Rating Scale Model, they are the same.

28
The New Body Part Specific CATs

• Asks questions appropriate to patients ability
  and impairment
• Asks items that maximize information about
  patient ability
• CAT stops when the patients estimate of ability
  is precise enough (ie, 95 certain the FS
  estimate is correct)

29
Operational Characteristics of the CAT - Revisited

• Starting item Item with maximum information
  (average ability)
• Ability estimation routine - MLE
• Item selection routine Max information
• Stopping rules Set SEM to acceptable level or
  number of items

30
CAT Development Testing SoftwareHart DL
Mioduski JE

• Allows us to vary CAT operational characteristics
  at the click of a mouse
• Starting item
• Estimation math
• Item selection routines
• Stopping rules

31
Shoulder CAT vs All Shoulder Items CAT Stopped
When SEMlt.33
Plot of patient Abilities 400 Patients
Average Number of Items 18
32
Shoulder CAT vs All Shoulder Items CAT Stopped
After 7 Items
Plot of patient Abilities 400 Patients
33
Shoulder CAT vs All Shoulder Items CAT Stopped
After 5 Items
Plot of patient Abilities 400 Patients
34
Steps In CAT Development

• Develop an item pool per construct
• Test items using IRT models that will drive the
  CAT
• Decide on operational characteristics of CAT
• Test effects of different operational
  characteristics of CAT on the reliability and
  validity of patient ability estimates
• Test practical application of CAT

35
Item Bank Development

• Wording of items
• Unidimensionality of items
• Local independence of items
• Model fits the data
• Limited DIF (differential item functioning)

36
IRT Model Selection

• 1-parameter vs 2-parameter models
• Primary differences the 2-p model
• Estimates item discrimination ability in addition
  to item difficulty
• Tend to increase measurement precision

37
Current Development

• Refining body part specific item pools for
  improved unidimensionality and local independence
• Exploring and testing different IRT models
• Test Differential Item Functioning
• Testing CAT operating characteristics
• Testing discriminant validity of patient ability
  estimates

38
DIF by Gender for Comb or brush your hair with
the affected arm
Probability
Ability
Black curves Females. Gray curves Males.
39
LEFS DIF by Body Part

• Item Squatting
• Item Calibrations scaled 0 to 100
• Foot/ankle 46 vs Knee 55 (t-15.6, Plt.001)
• Hip 49 vs Knee 55 (t-9.9, Plt.001)
• Conclusion Separate LEFS CATs by body part, ie,
  hip, knee, foot/ankle

40
Current Body Part Specific CATs

• Lower Extremity Functional Scale
• Hip CAT
• Knee CAT
• Foot/Ankle CAT
• Shoulder CAT (Shoulder Flexi Scale)
• Lumbar CAT (BPFS and PF Items)

In development Elbow, Wrist/hand, Cervical CATs
41
More Precise General CAT

• Patients with medical, neurological and
  orthopedic conditions
• 2-parameter IRT model
• Will be in next PI version January 2007
• Will equate body part specific CATs to the
  General CAT as time permits

42
Summary

• Analyses confirm IRT models can
• Link items from different outcomes instruments
• Co-calibrate items into unidimensional banks
• Generate expandable item banks
• Develop CATs for multiple construct assessment at
  the bedside that produce precise measures with
  improved efficiency
• Improve patient classification for better
  discriminant validity of measures
• Make the process semi artificially intelligent
• CATs are the future!

43
NIH Funded PROMIS Project
http//nihroadmap.nih.gov/clinicalresearch/promis.
asp
44
PROMIS Domain Framework
Physical Health
Pain
Symptoms
Fatigue
Other
Satisfaction
Emotional Distress
Health
Mental Health
Cognitive Function
Satisfaction
Subjective Well-Being (positive effect)
Satisfaction
Meaning and Coherence (spirituality)
Positive Psychological Functioning
Mastery and Control (self-efficacy)
Positive Impacts of Illness
Performance
Role Participation
Social Health
Satisfaction
Social Support
Satisfaction
45
Thank you