SSA-NIH-BU Collaboration to Improve the Disability Determination Process:

1
SSA-NIH-BU Collaboration to Improve the
Disability Determination Process Feasibility of
Claimant- and Provider-Reported Computer
Adaptive Tests (CAT)
Stephen M. Haley, Ph.D. Alan M. Jette,
Ph.D Elizabeth E. Barfield, MPH
2
The BU HDR Team

• Steve Haley , PI
• Alan Jette, Co-PI
• Kara Bogusz, Project Director
• Mary Slavin, Training
• Pengsheng Ni, IRT Analyst and CAT programmer

• Beth Barfield, Pre-Doctoral Research Fellow
• Ron Hambleton, IRT consultant
• Bill Frey, Director of Westat, Inc.

3
Overview of the Presentation

• Functional Domains
• Formative Work- Item Pools
• Calibration Study
• Computer Adaptive Testing (CAT)
• Interpretation of Individual Scores
• Pilot study

4
Point of Clarification
This exploratory work, being conducted by NIH and
BU, examines ways in which claimants and their
health care providers can quickly and easily
provide information about claimants
function. SSA has not yet adopted or endorsed
this approach.
5
1. Functional Domains

• Physical Demands
• Interpersonal Interactions

6
(No Transcript)
7
Physical Demands Domain Framework
Changing Maintaining Body Position
Physical Demands
Whole Body Mobility
Carrying, Moving, Handling Objects
8
Kneeling
Standing
Changing Basic Body Position
Whole Body Transfer
Lying Down
Shifting Center of Gravity
Bending
Changing Maintaining Body Position
Sitting
Squatting
Lying
Sitting
Maintaining Body Position
Squatting
Kneeling
Standing
9
Changing Maintaining Body Position
Physical Demands
Whole Body Mobility
Carrying, Moving, Handling Objects
10
Walking Short Distances
Walking
Walking Long Distances
Walking on Different Surfaces
Stepping Around/Over Obstacles
Whole Body Mobility
Climbing
Crawling
Moving Around
Running
Device Aided Mobility
11
Changing Maintaining a Body Position
Physical Demands
Whole Body Mobility
Carrying, Moving, Handling Objects
12
Unilateral Lift
Bilateral Lift
Lifting Carrying Objects
Unilateral Carry
Bilateral Carry
Picking Up
Carrying, Moving, Handling Objects
Gripping
Hand Arm Use
Holding
Turning
Pinching
Manipulating Objects
Pulling
Pushing
Reaching
13
Changing Maintaining Body Position
Physical Demands
Whole Body Mobility
Carrying, Moving, Handling Objects
14
(No Transcript)
15
Interpersonal Interactions Domain Framework
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
16
(No Transcript)
17
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
18
(No Transcript)
19
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
20
(No Transcript)
21
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
22
(No Transcript)
23
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
24
(No Transcript)
25
Temperament Personality
Behavior Modulation
Interpersonal Interactions
Adaptability
Basic Interactions
Workplace Behaviors
26
(No Transcript)
27
1. Functional Domains

• Questions??

28
2. Formative Work
29
The Process of Building CATs
30
2. Formative Work

• Extensive literature review
• Focus groups (interviews)
• 6 providers
• Six content experts
• Cognitive Interviews 7 claimants

31
Cognitive Interviews

• After establishing critical item content, time
  frame and response scales the item pools were
  prepared for cognitive interviews with both
  claimants and health care providers.
• Cognitive Interviewing is a technique used to try
  to get at sources of response error in
  questionnaires. Cognitive interviewing asks
  interviewees about question comprehension,
  decision processes and response processes .

32
Cognitive Interviews

• This process involves asking the participant a
  question and then asking a series of probes.
  Participants were probed with questions like
• In your own words, what do you think this
  question is asking?
• How confident are you in your answer to this
  question?
• Can you think of a better way to ask this
  question?
• How did you arrive at that answer?
• Was it easy or hard to answer that question?

33
Results of Cognitive Interviews

• Interpersonal Interactions
• Of the 136 items administered in the
  interpersonal interactions item bank, 7 items
  were completely re-written based on claimant
  feedback and 1 item was deleted due to
  comprehension difficulty.

• Physical Demands
• Of the 129 items administered in the physical
  demands group, 10 items were completely
  re-written based on claimant feedback and 8 were
  removed due to comprehension difficulty.

34
Sample Physical Demands Items

• Are you able to lift a 20 lb. object (a toolbox)
  from table height to a high shelf?
• Responses without difficulty, with a little
  difficulty, with some difficulty, with a lot of
  difficulty, unable to do, I dont know
• How far are you able to walk without stopping?  
  Responses More than an hour, An hour, 30
  minutes, 15 minutes, Unable to do, I don't know

• How quickly are you able to walk? Responses
  Faster than those around me, At a normal pace
  compared to those around me, At a slower pace
  than those around me, Unable to do, I don't know
• Are you able to work overhead for 20 minutes
  (e.g. organizing a high shelf in a closet)?
   Responses without difficulty, with a little
  difficulty, with some difficulty, with a lot of
  difficulty, unable to do, I dont know

35
Sample Interpersonal Interactions Items

• I get in conflicts with others
• I have difficulty calming down
• I cant stop myself from doing the same thing
  over and over again

• I feel good about myself.
• I am so tired when I wake up, it's hard to get
  going.
• I can get back on track when I am distracted.

Strongly agree, agree, disagree, Strongly
disagree
36
OIDAP RecommendationsPhysical Demands

• Content Areas Addressed

• Repetitive items
• Varying force requirements
• Duration of typical day
• Balance
• Reaching levels
• Unilateral/Bilateral

37
OIDAP RecommendationsInterpersonal Interactions
Emotional Behaviors

• Content Areas Addressed

• Content Areas not Addressed

• Interpersonal Functioning
• Friction, cooperation, working with others,
  criticism acceptance and response to, expressing
  oneself
• Initiative Persistence
• Task completion, persistence of task performance,
  leaving ones home
• Information Process Decision Making
• Ability to make a decision
• Self Management Self Monitoring
• Disturbing behaviors, social appropriateness,
  controlling symptoms, managing mood, stress
  tolerance, expression and controlling emotion

• Neuro-cognitive

38
Item Pools for each Domain

• Interpersonal Interactions
• 361 initial items
• 31 other instruments
• 57 PROMIS/NeuroQOL
• 273 new
• ______________________
• 155 final items to calibrate
• 90 provider items

• Physical Demands
• 174 initial items
• 22 other instruments
• 31 PROMIS/NeuroQOL
• 75 new items
• ______________________
• 124 final items to calibrate
• 97 provider items

39
2. Formative Work

• Questions???

40
3. Calibration Study
41
Goals of the Calibration Study

• Develop a sample of claimants providers with
  geographic diversity of SSA regions, offices, and
  work disability claimants.
• Target samples for each content domain
• 1000 SSA claimants.
• 500 primary providers
• 250500 supplemental providers
• Create calibrated item banks for each domain
• Create a CAT instrument for each domain
• Analyze claimant provider dyads

42
Plans for Calibration Study

• Work closely with SSA to develop sampling
  procedures
• Work with Westat, an outstanding survey research
  firm with SSA experience
• Administer item pools by web and telephone
  interview
• Implement analyses to build item banks

42
43
Data Collection Strategy

• Westat to contact each claimant to enroll and
  secure consent
• Calibration survey (Item pools) administered by
  web or telephone interview
• Provider samples identified by claimants
• Westat to contact providers by telephone

44
3. Calibration Study

• Questions???

45
4. CAT
46
Building a CAT Instrument ..

• Score estimation show it in terms of
  probability estimates
• Item selection criteria- maximum item information
  criteria
• Stop rule- static or dynamic stopping rules
• CAT administers a small sample of Items selected
  from a large IRT calibrated item bank
• The items that are administered are chosen based
  on how a person responds to previous items
• CAT reduces number of assessment items needed for
  an accurate assessment

47
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
48
BETTER
WORSE
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
49
Score Std. Error
44.2 4.1
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
50
BETTER
WORSE
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
51
Score Std. Error
44.2 4.1
45.0 2.9
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
52
WORSE
BETTER
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
53
Score Std. Error
44.2 4.1
45.0 2.9
46.6 2.5
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
54
With some difficulty
WORSE
BETTER
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
55
Score Std. Error
44.2 4.1
45.0 2.9
46.6 2.5
46.9 2.1
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
56
The Process of Building CATs
57
4. CAT
Questions?
58
5. Interpreting Scores
59
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
Example Item Map for a Mobility Scale
60
BU example of how CAT can express claimants (and
providers) own assessment of their functioning
Example Item Map for a Mobility Scale
Claimant score 60
Level 1
Level 2
Level 3
Level 4
Level 5
34
52
66
84
Cut score
61
6. Pilot Study
62
Pilot Study

• Goal is to determine feasibility of CAT and
  concurrent validity of the two new content
  domains
• N120 claimants
• Collect response burden data for CAT
• Compare CAT scores with legacy measures

63
6. Pilot Study

• Questions?

64
Summary of the Presentation

• Functional Domains
• Formative Work- Item Pools
• Calibration Study
• Analysis Plan and CAT
• Interpretation of Individual Scores
• Pilot study

65
Next Steps

• Future work
• Develop calibrate item banks in other
• content domains (e.g., Communication,
• General Tasks and Demands, Learning and
• Applying Knowledge, Self-care)
• Large demonstration project comparing use of CAT
  with typical procedures

66
Point of Clarification
This exploratory work, being conducted by NIH and
BU, examines ways in which claimants and their
health care providers can quickly and easily
provide information about claimants
function. SSA has not yet adopted or endorsed
this approach.
67
(No Transcript)