Assessing Change in Clinical Practice

1
Assessing Change in Clinical Practice

• Richard Bost, Ph.D., ABPP
• Frances Wen, Ph.D.
• Michael Basso, Ph.D

Oklahoma Psychological Association 11/6/04
2
Key issue

• How does one demonstrate statistically
  significant change in an individual patient?
• To document the clinical utility of an
  intervention
• To document spontaneous recovery or deterioration

3
Compare Groups
4
Compare Individual from Group
5
Compare New Individualto Group
6
Presenters

• Richard H. Bost, Ph.D., ABPP, BCIAC
• Assoc Prof Behavioral Science at OSU-COM
• Board certified in Rehabilitation Psychology
  (ABPP) and in Biofeedback (BCIAC)
• Clinical experience in neurological impairment
  and physical health problems
• Relative newcomer to research in computerized
  interventions in ABI

7
Presenters

• Frances Wen, Ph.D.
• Asst Prof Family Medicine at OSU-COM
• Focus on consulting with physicians on research
  regarding clinical health questions, grant
  writing
• Clinical experience in physical and mental health
  problems
• Researches physician competencies, assessment in
  medical education, availability/utilization of
  behavioral and health care services in rural areas

8
Presenters

• Michael Basso, Ph.D.
• Assoc Prof of Psychology, Director of Clinical
  Training at TU
• Teaches neuropsychology, cognitive psychology,
  and tests and measures
• Researches cognitive function in psychiatric
  impairment, immune disorders, and MS

9
Organization of Presentation

• Independent but coordinated presentations
• Basic concepts (Bost)
• Statistical foundations (Basso)
• Application to emotional/behavioral medicine
  measures (Wen)
• Application to cognitive measures (Bost)
• Some content overlap repetition of key points
• Some review of familiar concepts new application
• Todays slides and handouts are available at
  http//www.healthsciences.okstate.edu/bost.html

10
Organization (cont)

• Software availability
• RCI Generator 2.0 (Devily, 2004)
  http//www.swin.edu.au/victims/resources/software/
  reliablechange/reliable_change_generator.html
• SRB Bivreg (Crawford Howell, 1998)
  http//www.abdn.ac.uk/psy086/dept/Journal_urls_re
  direction.htm
• Multiple Regression Multreg (Crawford Howell,
  1998) http//www.abdn.ac.uk/psy086/dept/Journal_u
  rls_redirection.htm
• SRB Data Prep (Bost, 2004) http//www.healthscienc
  es.okstate.edu/bost.html

11
Key task

• Distinguish error in measurement from clinically
  meaningful change
• Specify random variation
• Identify minimize effects of bias

12
Common Sources of Variance in Serial Assessment

• Bias ? systematic change
• Extraneous events (e.g. 9/11 ? anxiety)
• Practice effects
• Memory for content
• Procedural learning
• Familiarity with examiner testing context
• Performance anxiety

13
Common Sources of Variance in Serial Assessment

• Bias (cont)
• Demographics
• Pre-test abilities
• Age
• Education
• Gender
• Ethnicity

14
Common Sources of Variance in Serial Assessment

• Error sources of random change statistical
  error
• Measurement error (SEM)
• Regression to the Mean (Seest)

15
Guidelines for Selecting Tests

• Determine examination goals
• Determine type of data generated by each test
  considered nominal, ordinal, interval, ratio.
• Only interval and ratio can be used in RCI and
  SRB analyses
• Appraise validity
• Appraise reliability
• Consider reading ability, time, costs

16
Change Assessment Strategies

1. Alternate forms
2. Reliable Change Index (RCI)
3. Adjusted RCI
4. Standardized Regression-Based change (SRB)
5. Multiple Regression

17
Standard Error of Difference (SEdiff)
18
Standard Error of Difference (SEdiff)
19
Standard Error of Difference (SEdiff)
20
Standard Error of Difference (SEdiff)
21
Standard Error of Prediction (SEpred)
22
Standard Error of Prediction (SEpred)
23
Standard Error of Prediction (SEpred)
24
Standard Error of Prediction (SEpred)
25
Application to Cognitive Measures

• Must have raw data from norm sample to use
  regression analysis
• This info usually not in test manuals
• Many clinicians will have this info in their
  clinical files

26
Application to Cognitive Measures (cont)

• Guidelines for using raw data from clinical
  files
• Specify norm sample characteristics
• Pull T1 and T2 test scores for 30 patients
• Apply to SRB Data Prep spreadsheet
• (demo spreadsheet)

27
On-line Cognitive Measure

• Headminder Cognitive Stability Index (CSI)
• Designed for serial measurement to document
  change over time
• 10 subtests
• Administration time25-30 minutes
• Four factors
• Processing speed
• Response speed
• Memory/learning
• Attention

28
On-line Cognitive Measure (cont)

• Headminder Cognitive Stability Index (CSI)
• Displays results in standard scores
• Displays RCIp with repeated administrations
• Available at http//www.headminder.com
• (demo CSI)