The Global Appraisal of Individual Needs (GAIN) Evaluator

1
The Global Appraisal of Individual Needs (GAIN)
Evaluators Handbook Practical Guidelines for
Using GAIN Data To Support Local and Cross-Site
Program Evaluation and Development

• Michael Dennis, Melissa Ives, Rodney Funk
• Chestnut Health Systems, Bloomington, IL
• Joint Meeting on Adolescent Treatment
  Effectiveness,
• April 25-27, 2007, Washington, DC

2
Objectives

1. Identify common questions used in evaluations and
   available GAIN tools and reports.
2. Understand how to respond to these questions
   using GAIN data, tools and reports.
3. Identify and answer questions that will help you
   use GAIN to support program evaluation and
   development.
4. Get your input on what would be most useful to
   have in the GAIN Evaluators Handbook

3
Questions

1. We will encourage you to ask questions as we go
   if something is not clear
2. We are handing out note cards to get more
   detailed questions to answer at the end. If you
   put your e-mail or address on them (or sign up)
   we will send you copies of our answers in
   writing.
3. Are there anything you are specifically here for
   that you want us to be sure and cover?

4
Common Questions in Local Program Evaluation
and Clinical Research

1. Who is being served?
2. What services are they receiving?
3. To what extent are services being targeted at the
   those in need?
4. To what extent are services being delivered as
   expected? (Performance/Fidelity)
5. Which is most effective of several services
   delivered?
6. What does it cost, cost effectiveness?

Source Dennis, Fetterman Sechrest (1994)
5
GAIN Scales Variable File

1. Purpose
2. Type of Measure
3. Interpretative Cut Points
4. Description
5. Syntax
6. References
7. Items
8. Summarizing in a table

6
Purpose

1. Diagnosis based on APA
2. Treatment planning based on CARF, COA, JCAHO,
   NIDA principals and SAMHSA TIPs
3. Placement based on ASAM and statistical models
4. Covariates based on lifetime or past year
   measures
5. Change Scores based on past 90 days, month, week
   or current status or time since last event
6. Methods Measures
7. Economic Measures

7
Types of Measures

1. Scale a set of symptoms or items that are
   inter-correlated (e.g.., dependence, depression)
   where we are interested in the pattern (i.e.
   Common variance, ONLY one where alpha makes
   sense)
2. Index a set of items that may not be directly
   related but add up to predict (e.g., sources of
   stress, barriers to treatment, expenses)
3. Ratio Estimators one measure divided by another
   (e.g.., percent of unprotected sex acts)
4. Status measures a categorical status based on a
   single question or created across multiple
   (e.g.., vocational status, housing status)
5. Survival Time to first event (e.g. time to first
   use)

8
Interpretative Cut-Points

• Definition of low, moderate and high clinical
  significance bands to aid interpretation and
  decision making (scale name g for group)
• Useful for defining need at both the client and
  program level
• Basis
• DSM or other clinical standards where available
  (e.g.., clinical is 3/7 dependence)
• 50th 90th percentile for common issues (e.g.
  days of alcohol use)
• 1 and median of 1 for zero saturated (more than
  half) and right skewed variables
• Reversed coded if up is low clinical
  significance

9
Descriptions

• GAIN-I SV excel file has text based
  descriptions, literal syntax (including older
  version if applicable), items, and references
• GAIN main scales and indexes word file
  includestext to put in a journal article or
  report, including
• - short definition
• - any subscales
• - source of measure
• - key reports/citations
• - alphas for adolescents and adults if
  applicable
• 3. The articles in the GAIN bibliography (many of
  which are included on the CD) have more details
  as well.

10
Possible Comparison Groups

• published data
• site over time
• subsites, staff, or clinics
• compare site to larger program (all sites)
• compare site to similar level of care, geography,
  demographic subgroup, or clinical subgroup
• match clinical subgroups from GAIN related
  presentations or papers
• formal matching or propensity scoring to make
  groups more statistically comparable
• formal randomized experiments
• path or mediation models to test whether it is
  actually the dosage or key ingredient driving the
  change

11
Major Predictors of Effective Programs that we
have to be cognizant of..

• An explicit intervention protocol (typically
  manualized) that a priori evidence that it works
  when followed
• Use of monitoring, feedback, supervision and
  quality assurance to ensure protocol adherence
  and project implementation
• Use proactive case supervision at the individual
  level to ensure quality of care
• Triage to focus on the higher severity subgroups
  of individuals

12
Impact of Intake Severity on Outcome
10
SPSM groupings
OVERALL
8
6
Substance Problem Scale (0-16 Past Month
Symptoms)
4
Dot/Lines show Means
2
Intake Severity Correlated -.66 with amount of
change
0
0
6
Wave
Source ATM Main Findings data set
13
Different than Regression to the Mean
10
SPSM groupings
OVERALL
8
No problems (0-25ile)
1-3 problems (25-50ile)
6
Substance Problem Scale (0-16 Past Month
Symptoms)
4-8 problems (50-75ile)
4
9 problems (75-100ile)
Dot/Lines show Means
2
In its most basic form, the mean variance are
the same at both time points no correlation
between intake amount of change
0
0
6
Wave
Source ATM Main Findings data set
14
Different than Regression to the Mean
10
SPSM groupings
OVERALL
8
No problems (0-25ile)
1-3 problems (25-50ile)
6
Substance Problem Scale (0-16 Past Month
Symptoms)
4-8 problems (50-75ile)
4
9 problems (75-100ile)
Dot/Lines show Means
2
If it was regression around the mean combined
with an mean effect it would but still no change
in variance or correlation between intake
amount of change
0
0
6
Wave
Source ATM Main Findings data set
15
Example of Multi-dimensional HIV Subgroups
0.40
0.20
0.00
-0.02
-0.03
-0.10
Cohen's Effect Size d
-0.20
-0.40
Unprotected Sex Acts (f.14)
Days of Victimization (f.22)
-0.60
Days of Needle Use (f1.19)
-0.80
A.
B.
C.
D.
Total
Low Risk
Mod. Risk Low
Mod. Risk High
Very High Risk
W/T
W/T
Source Lloyd et al 2007
16
Key things to Test and Monitor

• Assumptions about population characteristics and
  needs (using site profiles)
• Comparability of comparison groups (using site
  profiles)
• Simple performance measures and early outcomes
  for monitoring implementation
• Measure of competence, fidelity and
  implementation
• Variability in outcomes by subgroup

17
Melissa Ives

• Melissa will now demonstrate how to use some of
  the data and tools we provide to do these things.

18
GAIN Evaluators Handbook Resources for
answering 'Who is being served?'

• Melissa L. Ives, MSW
• Research Associate
• Chestnut Health Systems
• Lighthouse Institute
• GAIN Coordinating Center

Joint Meeting on Adolescent Treatment
Effectiveness, April 25-27, 2007, Washington, DC
19
Introduction and goals

• The first of the 5 key questions is
• Who is being served?
• Two goals of this portion of the presentation
• Identify tools that are already available from
  the GCC.
• Explore the use of one key tool for examining
  characteristics of those being served.
• Always our goal To answer your questions.
• Be sure to write down any questions that are not
  answered during the presentation.
• Answers to these questions will be used to
  enhance the Evaluators Handbook.

20
Overview

• It is always easier to use the right tool than to
  create a new one
• especially if the tool is readily available.
• I used the AutoContent Wizard provided by
  PowerPoint to create these slides.
• The GCC currently provides several tools to
  support evaluators or analysts in answering the
  key questions.

21
Tools
GAIN-I / M90 data
Electronic Encyclopedia (GI SV)
Site Profiles
Evaluator Or Analyst
LI Analytic Training Series Memos
Syntax template files
TTL Report
FUL Report
Adult Adolescent Norms
22
Site Profiles

• Excel file containing information about the
  characteristics of clients being served.
• Aggregated by site within a program or study.
• Contents
• Title page defining what groups are included
  (with grant numbers as acknowledgement) and what
  time period is covered.
• Chart Options Interactive tab to select desired
  site(s) included in graphs.
• Table of Contents list of graph
• Single site charts
• Two-group comparison charts
• Data tables
• Worksheets

23
Site Profiles

• Provided quarterly for CSAT Programs on the APSS
  website.
• Can be created as ltvarnamegt Profiles (based on a
  variable other than site).
• A version for Level of Care is provided on
  todays CD.

24
Example from ESD 113 Olympia, WA

• EAT site with additional GAIN data from 2 other
  locations.
• Interested in examining one of these locations in
  comparison with the rest of their own EAT site
  and with the whole EAT program.
• Used the SPSS syntax and template in Excel
• Open ESD Site Profiles Open ESD
  Presentation

25
Paste Special -Picture
26
Paste Special - Device Independent Bitmap
27
Summary

• At this point you should
• Be aware of the existence of several tools to
  assist you in understanding who is being served.
• Be able to find information about tools you want
  to use.
• Be excited about how you can use these tools for
  your own analysis.
• NOT be worried if you still have questions!
• WRITE any questions on your index card.
• For a direct reply after this meeting
• Write legibly and include your name and e-mail
  address.

28
Where to Get More Information

• Our website
• http//www.chestnut.org/li/
• FTP Common Site Evaluators Folder
• ftp//data.chestnut.org
• Username Common
• Password public
• Send e-mail to
• GAINEval_at_chestnut.org

29
(No Transcript)
30
(No Transcript)
31
Where to Get More Information

• Our website
• http//www.chestnut.org/li/
• FTP Common Site Evaluators Folder
• ftp//data.chestnut.org
• Username Common
• Password public
• Send e-mail to
• GAINEval_at_chestnut.org

32
Examples of Analysis Using GAIN Data

• Rod Funk
• Chestnut Health Systems, Bloomington, IL

33
Acknowledgement

• This presentation was developed under contract
  270-2003-00006 from the Center for Substance
  Abuse Treatment (CSAT) of the Substance Abuse and
  Mental Health Services Administration (SAMHSA)
  and presents data from the Persistent Effects of
  Treatment Study (PETS, Contract No. 270-97-7011)
  and the Cannabis Youth Treatment (CYT)
  Cooperative Agreement (Grant Nos. TI11317,
  TI11320, TI11321, TI11323, and TI11324) as well
  as the Assertive Continuing Care Study supported
  by funds and data from the National Institute on
  Alcoholism Alcohol Abuse (RO1 AA 10368). The
  opinions are those of the authors and do not
  reflect official positions of the government.

34
Evaluating the Effects of Treatment
Month
Z-Score
Source Dennis et al, 2003, 2004
35
Change in Substance Frequency Scale in CYT
Experiment 1 Incremental Arm
Months from Intake
Source Dennis et al, CPDD, 2003
36
Change in Substance Frequency Scale inCYT
Experiment 2 Alternative Arm
Months from Intake
Source Dennis et al, CPDD 2003
37
Average Episode Cost (US) of Treatment
--------------------------------------------Econo
mic Cost------------------------------------------
--------- Director Estimate-----
4,000
3,322
3,500
3,000
2,500
Average Cost Per Client-Episode of Care
1,984
2,000
1,559
1,413
1,500
1,197
1,126
1,000
500
-
ACRA (12.8 weeks)
MET/CBT5 (6.8 weeks)
MET/CBT5 (6.5 weeks)
MET/CBT12 (13.4 weeks)
FSN (14.2 weeks w/family)
MDFT(13.2 weeks w/family)
Source French et al., 2002, 2003
38
Cost Per Person in Recovery at 12 and 30 Months
After Intake by CYT Condition

Experiment 1 (n299)
Experiment 2 (n297)
Cost Per Person in Recovery (CPPR)
30,000
ACRA Effect Largely Sustained
25,000
20,000
15,000
10,000
5,000
0
MET/ CBT5
MET/ CBT12
FSNM
MET/ CBT5
ACRA
MDFT
6,437
10,405
24,725
27,109
8,257
14,222
CPPR at 12 months
3,958
7,377
15,116
6,611
4,460
11,775
CPPR at 30 months
Plt.0001, Cohens f 1.42 and 1.77 at 12
months Plt.0001, Cohens f 0.76 and 0.94 at 30
months
Source Dennis et al., 2004 2005
39
Environmental Factors are also the Major
Predictors of Relapse
AOD use in the home, family problems,
homelessness, fighting, victimization, self help
group participation, structure activities
Baseline
Family
.32
.77
.18
Conflict
Recovery
Environment
-.54
Risk
-.13
.17
.58
.74
Family
.22
.32
Substance-
-.09
Cohesion
Substance
.43
Related
Use
Problems
.32
.82
.19
.11
Social
Social
.19
-.08
.22
Support
Risk
Baseline
Baseline
Model Fit CFI.97 to .99 by follow-up
wave RMSEA.04 to .06 by wave
.21
Peer AOD use, fighting, illegal activity,
treatment, recovery, vocational activity
Baseline
Source Godley, Kahn et al (2005)
40
Assertive Continuing Care (ACC) Hypotheses
Assertive Continuing Care
41
ACC Improved General Continuing Care Adherence
(GCCA)
100
20
30
10
40
50
60
70
80
90
0
Weekly
Tx
Weekly 12 step meetings
Relapse prevention
Communication skills training
Problem solving component
Regular urine tests
Meet with parents 1-2x month
Weekly telephone contact
Contact w/probation/school
Referrals to other services
Follow up on referrals
Discuss probation/school compliance
Adherence Meets 7/12 criteria
Source Godley et al 2002, 2007
42
ACC was associated with Reduced Relapse
1.0
.9
ACC almost doubled the time before relapse and
reduce long term relapse
.8
.7
.6
Proportion Remaining Abstinent from Marijuana
.5
.4
.3
.2
UCC
.1
0.0
270
240
210
180
150
120
90
60
30
0
Days to First Marijuana Use plt.05
Source Godley et al 2002
43
GCCA Improved Early (0-3 mon.) Abstinence
100
Regardless of condition
90
80
70
60
50
38
36
40
30
24
20
10
0
Any AOD (OR2.16)
Alcohol (OR1.94)
Marijuana (OR1.98)
Low (0-6/12) GCCA
Source Godley et al 2002, 2007
44
Early (0-3 mon.) Abstinence Improved Sustained
(4-9 mon.) Abstinence
100
90
80
70
60
50
40
30
22
22
19
20
10
0
Any AOD (OR11.16)
Alcohol (OR5.47)
Marijuana (OR11.15)
Early(0-3 mon.) Relapse
Source Godley et al 2002, 2007
45
Victimization and Level of Care Interact to
Predict Outcomes
CHS Outpatient
CHS Residential
40
35
30
25
Marijuana Use (Days of 90)
20
15
10
5
0
Intake
6 Months
Intake
6 Months
OP -High
OP - Low/Mod
Resid-High
Resid - Low/Mod.
Source Funk, et al., 2003
46
How do CHS OPs high GVS outcomes compare with
other OP programs on average?
1.00
CYT Total (n217 d0.51)
0.80
0.60
ATM Total (n284 d0.41)
0.40
CHSOP (n57 d0.18)
0.20
Z-Score on Substance Frequency Scale (SFS)
0.00
-0.20
-0.40
-0.60
-0.80
-1.00
Intake
Mon 1-3
Mon 4-6
Mon 7-9
Mon 10-12
Source CYT and ATM Outpatient Data Set, Dennis
2005
47
Which 5 OP programs did the best with high GVS
adolescents?
1.00
0.80
0.60
0.40
0.20
Z-Score on Substance Frequency Scale (SFS)
0.00
-0.20
-0.40
-0.60
-0.80
Currently CHS is doing an experiment comparing
its regular OP with MET/CBT5
-1.00
Intake
Mon 1-3
Mon 4-6
Mon 7-9
Mon 10-12
Source CYT and ATM Outpatient Data Set, Dennis
2005
48
Methodological Issues to Be Aware of..

• Site differences Beware of demographic
  differences between sites, such as on gender and
  race. You can use cluster analysis to create
  homogeneous subgroups or propensity scores to
  create more equivalent groups.
• Floor Ceiling Effects Check distributions of
  outcome variables. If wanting to look at needle
  use, there is very little to begin with in the
  CSAT data which would make it difficult to look
  at change over time.
• Non-normal distributions A lot of variables used
  for outcome analysis can be very zero saturated
  and therefore highly right skewed.

49
Methodological Issues Continued..

• Co-Occurring Disorders Beware that adolescents
  are more than likely presenting for more problems
  than just substance use, such as internal and
  external disorders.
• Controlled Environment Be sure to check for days
  in controlled environment. You may need to adjust
  your outcomes, such as days of abstinence. You
  could subtract days in a controlled environment
  from your dependent variable, use it as another
  outcome variable or use it as a covariate in your
  analysis

50
References

• Dennis, M. (2005). State of the art of treating
  adolescent substance use disorders Course,
  treatment system, and evidence based practices.
  Paper presented at the 2005 State Adolescent
  Coordinators (SAC) Grantee Orientation Meeting,
  Baltimore, MD. http//www.chestnut.org/LI/Posters
• Dennis, M. L., Godley, S. H., Diamond, G., Tims,
  F. M., Babor, T., Donaldson, J., Liddle, H., et
  al. (2004). The Cannabis Youth Treatment (CYT)
  study Main findings from two randomized trials.
  Journal of Substance Abuse Treatment, 27,
  197213.
• Dennis, M. L., et al. (2003).Cannabis Youth
  Treatment Experiment 12 and 30 Month Findings.
  Presentation at College of problems of Drug
  Dependence, Bal Harbour, FL. http//www.chestnut.
  org/LI/Posters
• French, M.T., Roebuck, M.C., Dennis, M.L.,
  Diamond, G., Godley, S.H., Tims, F., Webb, C.,
  Herrell, J.M. (2002). The economic cost of
  outpatient marijuana treatment for adolescents
  Findings from a multisite experiment. Addiction,
  97, S84-S97.
• French, M. T., Roebuck, M. C., Dennis, M. L.,
  Diamond, G., Godley, S. H., Liddle, H. A., and
  Tims, F. M. (2003). Outpatient marijuana
  treatment for adolescents Economic evaluation of
  a multisite field experiment. Evaluation
  Review,27(4)421-459.
• Funk, R. R., McDermeit, M., Godley, S. H.,
  Adams, L. (2003). Maltreatment issues by level of
  adolescent substance abuse treatment The extent
  of the problem at intake and relationship to
  early outcomes. Journal of Child Maltreatment, 8,
  36-45.
• Godley, M. D., Godley, S. H., Dennis, M. L.,
  Funk, R., Passetti, L. (2002). Preliminary
  outcomes from the assertive continuing care
  experiment for adolescents discharged from
  residential treatment. Journal of Substance Abuse
  Treatment, 23, 21-32.
• Godley, M. D., Godley, S. H., Dennis, M. L.,
  Funk, R. R., Passetti, L. L. (2007). The effect
  of Assertive Continuing Care on continuing care
  linkage, adherence, and abstinence following
  residential treatment for adolescents with
  substance use disorders. Addiction, 102, 81-93.
• Godley, M. D., Kahn, J. H., Dennis, M. L.,
  Godley, S. H., Funk, R. R. (2005). The
  stability and impact of environmental factors on
  substance use and problems after adolescent
  outpatient treatment. Psychology of Addictive
  Behaviors, 19, 62-70.

51
Reduced Relapse Marijuana
1.0
.9
.8
.7
.6
.5
Proportion Remaining Abstinent
.4
ACC
.3
.2
UCC
.1
0.0
270
240
210
180
150
120
90
60
30
0
Days to First Marijuana Use plt.05
Godley, CPDD Poster, 2003
52
Logistic Regression Example
ACC main findings, Godley, et al)
53
Garner, Godley, Godley, Funk, Dennis (in
press). Psychology of Addictive Behaviors