Science Behind the New ConditionSpecific CATs

1
Science Behind the NewCondition-Specific CATs

• Dennis L. Hart, PhD, PT
• Director of Consulting and Research
• Focus On Therapeutic Outcomes, Inc., Knoxville,
  TN
• November 12, 2005

2
We understand best those things we see grow from
their very beginnings. Aristotle in
Metaphysics About 350 BC Development of
potentiality to actuality
3
Progression of Thought

• Started with paper and pencil surveys
• Both general health status and condition specific
  surveys
• Very lengthy and cumbersome
• Time consuming
• Asked many questions that were not pertinent to
  the patient
• Functional status change measure was an effect
  size no one understood it!

1992
Goal Always better!
4
Progression of Thought

• Dropped condition-specific surveys and went with
  general health-related quality of life surveys
• Still lengthy and cumbersome
• Time consuming
• Asked many questions that were not pertinent to
  the patient
• Functional status change measure was effect size
  no one understood it!

1994
5
Progression of Thought

• Began progression to Computerized Adaptive
  Testing technology using general health-related
  quality of life surveys
• Shorter and much less cumbersome
• Less time consuming
• Asked fewer questions that were more focused
  clinician determined to be pertinent to the
  patient
• Was multidimensional
• Functional status measure became a linear 0 to
  100 scale

1999
6
Progression of Thought - NOW

• Refining Computerized Adaptive Testing technology
  using condition-specific surveys and advanced
  mathematics
• Shorter, very focused
• Directly related to patients impairment and
  level of ability mathematically precise AND
  clinician determined to be pertinent to the
  patient
• Functional status measure still a linear 0 to 100
  scale

2005
7
New vs. Older CATsOrthopedic Care Type

• Cervical Older
• Shoulder New
• Elbow Older
• Wrist/Hand - Older

• Lumbar New
• Hip New
• Knee New
• Foot/Ankle - New

8
Lower Extremity CATs

• Started with Binkley et als Lower Extremity
  Functional Scale survey for patients with hip,
  knee, or foot/ankle impairments
• Why?
• Great psychometrics conceptually sound,
  reliable, valid, responsive, low measurement
  error
• Well received by clinicians
• Great researchers with whom to work

9
How did we develop the new CATs?

• PI software allowed use of Optional Surveys And
  you used them!!
• Started with n1,772 patients with intake LEFS
  data
• Analyzed the data using Item Response Theory
  mathematics
• Developed the CATs

10
Why IRT Mathematics?

• IRT math has been used for 50 years in
  educational and armed services research
• Over 2 million GRE CAT examines taken
• Many standardized tests becoming CATs
• IRT allows good instrument development
• IRT drives CAT
• http//www.psych.umn.edu/psylabs/catcentral/

11
Concepts that Drive IRT

• Unidimensionality measures one construct
• Functional status patients perception of their
  ability to perform functional tasks described in
  the FS items.
• IRT mathematical model selection
• Measure of FS ability is related to the
  difficulty of the FS item.
• Differential Item Functioning patients with
  same FS ability but from different groups answer
  the item differently.

12
Operating Characteristics of a CAT

• Start with a pool of calibrated items
• Select starting item most informative item
• Estimate FS ability with SE MLE
• Stopping rules
• SElt4 out of 100 FS units
• Change in FS is lt1 out of 100 FS units over the
  last three items
• Selecting the next item most informative item
  given the current estimate of FS ability

13
How Did the Analyses Go?

• Unidimensionality
• Supported one construct
• One item was deleted (rolling in bed) because of
  local dependence
• IRT Model Fit
• Model fit the data well, but one item did not fit
  well (sitting for 1 hr), which was deleted
• 18-item scale fit well, strong reliability,
  discriminating ability

14
Hierarchical Structure of 18-Item LEFS Scale
Item Responses
---------------------------------------------
--------- Item 1 1
2 3 4 5 5 TURNS 1
1 2 3 4 5 5
RUNUNEV
1 1
2 3 4 5 5 RUNEVEN 1
1 2 3 4 5
5 HOP

1
1 2 3 4 5 5
SQUAT
1 1 2
3 4 5 5 WALKMILE 1
1 2 3 4 5
5 HOBBY 1 1 2 3 4
5 5 STAND 1
1 2 3 4 5 5
HEAVY 1 1 2 3 4 5
5 STAIRS
1
1 2 3 4 5 5
WORK 1 1 2 3 4 5
5 WALK2BLK


1 1 2 3 4 5
5 LIFT 1 1 2 3
4 5 5 CAR 1
1 2 3 4 5 5
BATH 1 1 2 3 4 5
5 LIGHT
1 1
2 3 4 5 5
SHOE
1 1 2 3 4
5 5 WALKRM -----------
-------------------------------------------
-10 10 30 50 70
90 110
Items
Easy to Hard Level of Item Difficulty
Low to High FS Ability
15
How Did the Analyses Go?

• DIF
• Items demonstrated DIF by body part effected
• For example, patients with knee impairments
  perceived squatting as more difficult compared to
  patients with foot or ankle impairments
• So, separated sample by body part (hip, knee,
  foot/ankle) and generated 3 CATs

16
Lower Extremity CATs

• Used on average 6 items before a stopping rule
  was satisfied.
• Number of items used ranged from 4 to 17.

17
Number of Items Error by FS Ability Knee CAT
Items/Error
FS Ability
18
Comparison of FS Measures CAT vs. All Items
Knee N949 R.97
19
Discriminant Validity

• FS CAT ability measures discriminated patients in
  similar ways as the FS ability measures that did
  not use a CAT.
• Acuity
• Age

20
Shoulder CAT

• Used the Shoulder Flexi Scale from Cook and
  Roddey
• Again, great psychometrics
• Started with 60 items
• Tested unidimensionality, item fit, DIF, etc.
• Developed the shoulder CAT using 37 items

21
Hierarchical Structure of 37-Item Shoulder Scale
---------------------------------------------
--------- Item 1 1
2 3 4 5 5 BULB 1
1 2 3 4 5 5
BCKSEATRE 1 1 2 3
4 5 5 OVRHEADSHE 1
1 2 3 4 5 5
MEDBOX 1 1 2 3 4
5 5 BED 1 1
2 3 4 5 5 OVRHEAD1 1
1 2 3 4 5
5 SKILLET 1 1 2
3 4 5 5 STRING 1
1 2 3 4 5 5
COLLAR 1 1 2 3 4
5 5 BALLUND 1 1
2 3 4 5 5
POTATOES 1 1 2 3 4
5 5 SLIDE 1 1
2 3 4 5 5 COMB 1
1 2 3 4 5
5 POCKET 1 1 2
3 4 5 5 SHELF 1
1 2 3 4 5 5
SAFESTRP 1 1 2 3 4
5 5 PULLBOX 1 1
2 3 4 5 5
WHEELSA 1 1 2 3 4
5 5 SHOULDER1 1 1
2 3 4 5 5
CROOK 1 1 2 3 4
5 5 JAR 1 1 2
3 4 5 5 PULLCHAIR 1
1 2 3 4 5
5 TIE 1 1 2 3 4
5 5 PUSHCHAIR 1
1 2 3 4 5 5
CHEST 1 1 2 3 4
5 5 JARSTEADY 1 1
2 3 4 5 5 SALT 1
1 2 3 4 5
5 DEODORANT 1 1 2 3
4 5 5 WATER 1
1 2 3 4 5
5 TABLE 1 1 2 3 4
5 5 UNDERPANTS 1 1
2 3 4 5 5
SOCKSON 1 1 2 3 4 5
5 EAROPP 1 1 2
3 4 5 5
FACESA 1 1 2 3 4 5
5 FAUCETSA 1 1 2
3 4 5 5
FLUSHING 1 1 2 3 4 5
5 GLASSES ------------------
------------------------------------ -6
-4 -2 0 2 4
6
Items
Easy to Hard Level of Item Difficulty
No Holes!
22
Lumbar CAT

• Used the Back Pain Functional Scale from
  Stratford et al and physical functioning items
  from Ware et al and Hart
• Again, great psychometrics
• Started with 28 items
• Tested unidimensionality, item fit, DIF, etc.
• Developed the lumbar CAT using 25 items

23
Hierarchical Structure of 25-Item Lumbar Scale
-----------------------------------------
--------- Item 1
1 2 3 3 VIGOR


1 1
2 3 3 SPORT
1
1 2 3 4 5
6 6 HEAVY 1 1
2 3 3 WALKMILE 1
1 2 3 3
MODERATE
1 1
2 3 3 LIFTGROC 1
1 2 3
3 STAIRS 1 1 2
3 3 BLOCKS 1 1
2 3 4 5 6 6
HOBBY 1 1 2 3 4 5
6 6 BENDING
1 1
2 3 4 5 6
6 STAND 1 1 2
3 3 LIFTOVER 1 1
2 3 3
VACATION 1 1 2
3 3 STAIRONE 1 1
2 3 4 5 6 6
WORK
1 1 2 3 4
5 6 6 STAIRS 1
1 2 3 3
CULTURAL 1 1 2 3 4 5
6 6 DRIVE 1 1 2
3 4 5 6 6 LIFT 1
1 2 3
3 ONEBLOCK 1 1 2
3 3 BED
1
1 2 3
3 CHAIR 1 1 2 3 4
5 6 6 SHOES


1 1 2 3
3 WALKRM 1 1
2 3 3
BATHING -----------------------------------
--------------- -5 -4 -3 -2 -1
0 1 2 3 4 5
Limited Holes
3 vs 6 Responses
Easy to Hard Level of Item Difficulty
Items
24
Bottom Line

• CATs are more efficient because they direct a
  limited number of the most informative items to
  the patient.
• CATs are more effective because they match the
  level of difficulty of the item with the level of
  ability of the patient.
• CAT measures are as discriminating as measures
  that use all items.
• Condition-specific CATs direct more pertinent
  items to the patient and facilitates clinical
  interpretation.
• FUTURE CAT measures are more responsive than
  summative measures because CAT measures are
  linear.

25
Future

• Never satisfied!!
• Will develop more condition-specific CATs
• Elbow and wrist/hand CAT under development
• Need a cervical CAT
• Will expand item banks by seeding items one at a
  time to fill holes, ie, strengthen construct
  validity of the scales