History of Adaptive Survey Administration

1
History of Adaptive Survey Administration
Evolutionof FOTOs Item Bank

• Dennis L. Hart, PhD, PT
• Director of Consulting and Research
• Focus On Therapeutic Outcomes, Inc.
• November 5, 2000

2
What is Adaptive Survey Administration (ASA)

• The next level in patient surveys. Evolved from
  the need to be more efficient and precise with
  our measures of function, and emphasize one
  patient at a time, at the bedside.

3
ASA Philosophy

• We never know if the patient (or you for that
  matter!) answered the question correctly
• Everything is a probability
• From our analyses, the patient is correct almost
  all (gt90) of time

4
Dynamic, Computer Controlled, ASA

• The Answer to Any Question is Predicated on Two
  Concepts
• the difficulty of the question
• the ability of the person

5
Dynamic, Computer Controlled, ASA

• The Answer to Any Question is Predicated on Two
  Concepts
• the difficulty of the question
• the ability of the person

Calculate from data
Measure with questions
6
So, what do we have now
7
Patient Reported Measures

• Generic Health Related Quality of Life (HRQL) -
  SF-36, SF-12
• Disease specific measures
• Oswestry, NDI, Lysholm
• Patient Specific Functional Scales

8
Generic HRQL

• Reliable
• Good construct validity
• Moderate responsiveness
• Requires calculations, not done at the bedside

9
Condition Specific HRQL

• Reliable
• Good content validity
• Good responsiveness
• Requires calculations, can be done at the bedside

10
Patient Specific Functional Scales

• Reliable
• Strong patient-specific content validity, one to
  two functional activities directly related to why
  patient is seeking care now
• Very responsive
• No calculation rate at bedside

11
Current Scoring HRQL

• Patient must answer all questions
• Summative, Likert scoring
• Does not manage missing data easily
• Makes a false assumption of linearity of ordinal
  scores to calculate a linear measure of function

12
ASA Scoring

• Manages missing data well (patients do not need
  to answer all questions)
• Rasch probabilistic models transform ordinal
  scores into linear measures of function
• Calculates hierarchical order of questions from
  easy to hard

13
PF-10 for All Impairments at Discharge
Response Performing Vigorous Activities Performing
Moderate Activities Walking More Than a
Mile Bending, Kneeling or Stooping Climbing
Several Flights of Stairs Lifting or Carrying
Groceries Walking Several Blocks Climbing One
Flight of Stairs Walking One Block Bathing or
Dressing
N35,366
14
PF-10 for All Impairments at Discharge
Response Performing Vigorous Activities Performing
Moderate Activities Walking More Than a
Mile Bending, Kneeling or Stooping Climbing
Several Flights of Stairs Lifting or Carrying
Groceries Walking Several Blocks Climbing One
Flight of Stairs Walking One Block Bathing or
Dressing
Hard
N35,366
Easy
15
PF-10 for All Impairments at Discharge
Response Performing Vigorous Activities Performing
Moderate Activities Walking More Than a
Mile Bending, Kneeling or Stooping Climbing
Several Flights of Stairs Lifting or Carrying
Groceries Walking Several Blocks Climbing One
Flight of Stairs Walking One Block Bathing or
Dressing
Hard
Clinically Logical!
N35,366
Easy
16
PF-10 for All Impairments at Discharge
Response Perform Vigorous Activities Perform
Moderate Activities Walk More Than a
Mile Bending, Kneeling or Stooping Climb Several
Flights of Stairs Lift or Carry Groceries Walking
Several Blocks Climb One Flight of Stairs Walking
One Block Bathing or Dressing
Hard
Clinically Logical!
N35,366
Easy
17
PF-10 for All Impairments at Discharge
Response Perform Vigorous Activities Perform
Moderate Activities Walk More Than a
Mile Bending, Kneeling or Stooping Climb Several
Flights of Stairs Lift or Carry Groceries Walking
Several Blocks Climb One Flight of Stairs Walking
One Block Bathing or Dressing
Hard
Clinically Logical!
N35,366
Easy
18
All Impairments - Discharge
Response Measure Infit VIGOR 3.29 .99 MODERATE
1.07 1.05 WALKMILE .75 .83 BENDING .56 1.07
STAIRSEV .38 .88 LIFTGROC 0 1.16 BLOCKS -.49
.80 STAIRONE -1.37 .81 BLOCK -1.91 .83 BATHIN
G -2.29 1.62
Intake Normal
N35,366
19
ASA Application Hierarchical Item Order
Response VIGOR MODERATE WALKMILE BENDING STAIRSE
V LIFTGROC BLOCKS STAIRONE BLOCK BATHING
Unique Starting Point
N35,366
20
ASA Application
Response VIGOR MODERATE WALKMILE BENDING STAIRSE
V LIFTGROC BLOCKS STAIRONE BLOCK BATHING
If too easy, move up
N35,366
21
ASA Application
Response VIGOR MODERATE WALKMILE BENDING STAIRSE
V LIFTGROC BLOCKS STAIRONE BLOCK BATHING
If too hard, move down
Reduces number of questions that we will ask the
patient
N35,366
22
Development of the new FOTO Index of Functional
Health Status

• Dennis L. Hart, PhD, PT
• Ben D. Wright, PhD
• Focus On Therapeutic Outcomes, Inc. and
• MESA Psychometric Laboratory
• University of Chicago

23
Purpose

• To describe the development of an index of
  functional health status (FHS) in patients
  seeking rehabilitation in acute, orthopedic
  outpatient centers and longer term care
  facilities
• To examine the hierarchical structure,
  unidimensionality and reproducibility of item
  calibrations along a scale of FHS

24
Subjects

• 92,343 adults
• 4817 years (14 to 99)
• 40 males
• 715 acute and 62 longer term facilities
• 41 states (USA)
• 1993 to 1999
• Diverse neuromusculoskeletal diagnoses

25
Age
Percent
Year Groups
N92,343
26
Data Set 1

• 5,712 patients
• Acute, orthopedic outpatients
• 1993-1994
• Males 50, 3912 Yrs Females 4113 Yrs
• SF-36 plus three additional depression questions
  (n5,712)
• Oswestry Low Back Pain Questionnaire (n2,970)
• Neck Disability Questionnaire (n1,415)
• Lysholm Knee Questionnaire (n1,589)

27
Data Set 2

• 88,155 patients
• Acute, orthopedic outpatients
• 1999
• Males 40, 4716 Yrs Females 4917 Yrs
• Questions from SF-36 PF-10 BP-2
• SF-12
• Three additional UE questions

28
Data Set 3

• 2,858 patients
• Longer term, more involved patients with
  neuromusculoskeletal impairments
• 1999
• Males 35, 6118 Yrs Females 6917 Yrs
• Questions from SF-36 PF-10 BP-2
• SF-12
• Three additional UE questions
• Ten additional lower level functional questions

29
Data Analyses

• 80 items, n92,343 patients
• Rasch Rating Scale (WINSTEPS)
• Intake to discharge assessment of item
  calibration for hierarchical structure
• Principal component analysis
• Co-calibration of items from all data sets
• Identified misfitting items (infit then outfit)
• Reliability

30
Data Matrix
Items
SF-39 Oswestry NDI Lysholm UE
Self-care
XXXXXX XXXXXX
XXXXXXXXXXX
XXXXXX
XXXXXX XXXXXXXXXXXXXXXXXX XXXXXX

XXX XXXXXX
XXX XXXX X XXXX
XXX X
XXXX
XXX XX X
Patients
180,766 Data Entries, 80 Items
31
Results

• All questions (SF-36, three depression, 10
  Oswestry, 10 NDI, 8 Lysholm, 3 UE, 10 Lower
  functioning) were analyzed for the FHS core
• Eight questions deleted Ü FHS-72
• Principal component analysis demonstrated one
  dimension of FHS (Rasch model explained 99 of
  FHS-72)

32
FHS-72

• N176,925 non-extreme patients
• Reliability .89
• Separation 2.82
• RMSE 4.27 (90 MDC 10 FHS-72 units)
• Infit 1.02(.4), Outfit 1.00(.5)
• Item calibrations intake to discharge r1.0

33
FHS-72 Keyform
Physical FHS-36 Example
34
Ms. Q
35
Ms. Q
Unexpected Disabilities
36
Ms. Q
Unexpected Abilities
37
Mr. V
1
1
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
Precision of Measurementof the Sample

• Every measure has an error
• Need a change of X to be greater than error (90
  Minimal Detectable Change - MDC)
• For FHS-72, MDC is 10 FHS-72 units as calculated
  for the entire sample
• Now, what about the precision of measurement for
  a specific patient?

43
Precision of Measurementfor a Patient

• Each FHS-72 measure has an error term
• Use that error term to calculate the MDC for that
  measure

44
Precision of Measurementfor a Patient
90 MDC per FHS-72 Measure
FHS-72 Units
FHS-72 Measure
45
Precision of Measurementfor a Patient
90 MDC per FHS-72 Measure
70 Patients
FHS-72 Units
MDC5
FHS-72 Measure
46
Precision of Measurementfor a Patient
90 MDC per FHS-72 Measure
87 Patients!
FHS-72 Units
MDC7
FHS-72 Measure
47
Conclusions

• Analyses confirmed
• Empirical item hierarchy of FHS
• Unidimensionality of FHS
• Reproducibility of item calibrations across
  repeated testing (intake to discharge for SF-36,
  SF-39, PF-10, Oswestry, NDI Lysholm), PF-10 2
  to 4 wk recall, and proxy to patient self-report

48
Conclusions

• Analyses confirmed
• Content construct validity of FHS
• Maintaining and exploring misfitting items may
  have potential for identifying patients that are
  clinically different, which suggests improved
  clinical diagnostic assessment

49
Conclusions

• Analyses confirmed
• Can calculate level of precision of the FHS-72
  measure
• Improved precision of measurement when assessing
  a specific patient at a specific time

50
Clinical Significance

• Analyses confirm the potential for dynamic,
  computer controlled adaptive administration for
  FHS assessment at the bedside
• Limited number of questions necessary to measure
  functioning with expanding item bank in computer
  for improved precision of measurement

51
Adding New QuestionsExpanding the Item Bank

• 3 New higher functioning items
• Split bending from kneeling (squatting covered by
  Lysholm and LEFS)
• Add all 20 LEFS items, but collapsed
• Add 17 of 20 UEFS items, collapsed
• Examine PSFS questions for items
• Etc., etc., etc.

52
Who Will Use These Measures

• John Q. Public
• Payers
• Case Managers
• Clinicians

53
So, what are you going to do with your paper
system?
Thanks...