Chris Borgmeier

1
Evaluating Researchon Challenging Behavior

• Chris Borgmeier
• Portland State University

2
From Spencer, Detrich Slocum, 2012
3
www.pbis.org

• Horner, R., Sugai, G. (2008). Is school-wide
  positive behavior support an evidence-based
  practice? OSEP Technical Assistance Center on
  Positive Behavioral Interventions and Support.
• http//www.pbis.org/files/101007evidencebase4pbs.
  pdf.

4
Evidence Basics From George Sugai
5
Why evidence-based?

• Maximize outcomes
• Minimize harm
• Increased accountability
• Increase efficiency
• Improve decision making
• Improve resource use

6
Basic Approach

• Start w/ what has greatest likelihood of
  addressing (evidence-based) confirmed
  problem/question
• Explained/supported conceptually/empirically
• Adapt to local context/culture/need
• Monitor regularly adjust based on data
• Adapt for efficient durable implementation

7
4 Evaluation Criteria

• Effectiveness
• Has/will practice produced desired outcome?
• Efficiency
• What are costs (time, resources, ) to implement
  practice?
• Relevance
• Is practice outcomes appropriate for situation?
• Conceptually soundness
• Is practice based on theory?

8
Basic Practices Evaluation
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
Guidelines for Selecting Practice

1. Define desired outcome
2. Delineate implementation setting
3. Identify evidence-based practice
4. Evaluate relevance of practice against outcome
   setting/context
5. Adopt/adapt practice to setting/context
6. Arrange supports for accurate implementation
7. Continuously monitor effectiveness

13
Design Questions

• Has functional or cause-effect relationship been
  demonstrated replicated?
• Have alternative explanations been accounted
  controlled for?
• Have threats or weaknesses of methodology been
  controlled for?
• Was study implemented w/ fidelity/accuracy?

14
Research Designs

• Experimental - RCT SSR
• Evaluation - Descriptive w/ baseline
• Case Study - Descriptive w/o baseline
• Testimonial - No/Limited data

15
Results Questions

• Who were subjects?
• How much like my participants?
• Where was study conducted?
• How much like where I work?
• What measures were used?
• Do I have similar data?
• What outcomes were achieved?
• Are expected outcomes similar

16
Effectiveness Logic

• Significance (believe)
• Likelihood of same effect by chance
• Effect Size (strength)
• Size of effect relative to business as usual
• Consequential Validity (meaning)
• Contextually meaningful

17
PBIS Evidence Base
18
School-wide PBIS Lets compare!

• www.pbis.org
• Click on Resource Catalog
• Then Literature List
• Evidence Base for SW-PBIS
• Randomized Control Trials

19
Randomized Control Trials of SW-PBIS

• Tier 1/ Universal SW-PBIS
• Bradshaw, C. P., Mitchell, M. M., Leaf, P. J.
  (2010).Examining the effects of school-wide
  positive behavioral interventions and supports on
  student outcomes Results from a randomized
  controlled effectiveness trial in elementary
  schools. Journal of Positive Behavior
  Interventions, 12(3), 133-148.
• Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N.,
  Leaf, P. (2008). The impact of school-wide
  positive behavioral interventions and supports
  (PBIS) on the organizational health of elementary
  schools.School Psychology Quarterly.
• Bradshaw, C., Reinke, W., Brown, L., Bevans, K.,
  Leaf, P. (2008).Implementation of school-wide
  positive behavioral interventions and supports
  (PBIS) in elementary schools Observations from a
  randomized trial. Education and Treatment of
  Children, 31, 1-26.
• Horner, R. H., Sugai, G., Smolkowski, K., Eber,
  L., Nakasato, J., Todd, A. W., Esperanza, J.,
  (2009). A randomized, wait-list controlled
  effectiveness trial assessing school-wide
  positive behavior support in elementary schools.
  Journal of Positive Behavior Interventions,
  11(3), 133-144.
• Sprague, J., Biglan, A., et al (in progress).A
  Randomized Control Trial of SWPBS with Middle
  Schools.

20
90-School RCT StudyHorner et al., in press

• Schools that receive technical assistance from
  typical support personnel implement SWPBS with
  fidelity
• Fidelity SWPBS is associated with
• Low levels of ODR
• .29/100/day v. national mean .34
• Improved perception of safety of the school
• reduced risk factor
• Increased proportion of 3rd graders who meet
  state reading standard.

21
RCT Project TargetBradshaw Leaf, in press

• PBIS (21 v. 16) schools reached sustained high
  fidelity
• PBIS increased all aspects of organizational
  health
• Positive effects/trends for student outcomes
• Fewer ODRs (majors minors)
• Fewer ODRs for truancy
• Fewer suspensions
• Increasing trend in of students scoring in
  advanced proficient range of state achievement
  test

22
Impact Factor
23
Ask a Faculty member
24
Collaborative Problem Solving

• Visit the website
• http//www.livesinthebalance.org/
• What do I notice?
• A canoe?
• Advertising products for purchase
• Lots of testimonials
• Little bit of research (10 citations listed under
  research)
• lets take a closer, evaluative look at the
  research

25
Evaluating a Research Study

• Abstract
• Introduction Literature Review
• Research Questions
• Method Design
• Subjects Settings / Measures/ Procedures
• Results
• Discussion Conclusions
• References

26
Results Questions

• Who were subjects?
• How much like my participants?
• Where was study conducted?
• How much like where I work?
• What measures were used?
• Do I have similar data?
• What outcomes were achieved?
• Are expected outcomes similar

27
Collaborative Problem Solvingdata from CPS
website on 6/18/12

• Johnson, M., Ostlund, S., Fransson, G., Landgren,
  M., Nasic, S., Kadesjo, B., Gillberg, C., and
  Fernell, E. (2012).  Attention-Deficit/Hyperactivi
  ty Disorder (ADHD) with Oppositional Defiant
  Disorder (ODD) in Swedish Children  An Open
  Study of Collaborative Problem Solving.  Acta
  Paediactrica, in press. 
• Ollendick, T. H. (2011). Invited Address
  Effective Psychosocial Treatments for Emotional
  and Behavioral Disorders in Youth. University of
  Stockholm, Sweden..
• Fraire, M., McWhinney, E., Ollendick, T.
  (2011). The effect of comorbidity on treatment
  outcome in an ODD sample. In T. Ollendick
  (Chair), Comorbidities in children and
  adolescents Implications for evidence-based
  treatment. Symposia presented at the
  41st European Association for Behavioral and
  Cognitive Therapies, Reykjavik, Iceland.
• Halldorsdottir, T., Austin, K. Ollendick, T.
  (2011). Comorbid ADHD in children with ODD or
  specific phobia Implications for evidence-based
  treatments. In T. Ollendick (Chair),
  Comorbidities in children and adolescents
  Implications for evidence-based
  treatment. Symposia presented at the
  41st European Association for Behavioral and
  Cognitive Therapies, Reykjavik, Iceland.
• Epstein, T., Saltzman-Benaiah, J. (2010).
  Parenting children with disruptive behaviors
  Evaluation of a Collaborative Problem Solving
  pilot program. Journal of Clinical Psychology
  Practice, 27-40.
• Martin, A., Krieg, H., Esposito, F., Stubbe, D.,
  Cardona, L. (2008). Reduction of restraint and
  seclusion through Collaborative Problem Solving
  A five-year, prospective inpatient study.
  Psychiatric Services, 59(12), 1406-1412.
• Greene, R.W., Ablon, S.A., Martin, A. (2006).
  Innovations Child Psychiatry Use of
  Collaborative Problem Solving to reduce seclusion
  and restraint in child and adolescent inpatient
  units. Psychiatric Services, 57(5), 610-616.
• Greene, R.W., Ablon, J.S., Monuteaux, M., Goring,
  J., Henin, A., Raezer, L., Edwards, G., Markey,
  J., Rabbitt, S. (2004). Effectiveness of
  Collaborative Problem Solving in affectively
  dysregulated youth with oppositional defiant
  disorder Initial findings. Journal of Consulting
  and Clinical Psychology, 72, 1157-1164.
• Greene, R.W., Biederman, J., Zerwas, S.,
  Monuteaux, M., Goring, J., Faraone, S.V. (2002).
  Psychiatric comorbidity, family dysfunction, and
  social impairment in referred youth with
  oppositional defiant disorder. American Journal
  of Psychiatry, 159, 1214-1224.
• Greene, R. W., Beszterczey, S. K., Katzenstein
  T., Park, K., Goring, J. (2002). Are students
  with ADHD more stressful to teach? Patterns of
  teacher stress in an elementary school sample.
  Journal of Emotional and Behavioral Disorders,
  10, 27-37.

NO studies have been conducted in school
settings all research is either with parents or
in-patient clinical settings
These studies to not evaluate effectiveness of CPS
28
Evaluating the Research studies
2004 Greene et al 2006 Greene , Ablon, Martin 2008 Martin et al 2010 Epstein Saltzmann
Subjects Age 47 kids w ODD 4-12 yrs. old 3-14 yrs. old School-age Kids w ODD Under 12 yrs.
Settings Outpatient MH clinic _at_ hospital Inpatient Psyc unit (13 beds) Inpatient Psyc unit (15 beds) Outpatient clinic
Procedure Compare CPS w parent training (PT) group Trained unit staff (pre/post) Trained unit staff (pre/post) Group CPS parent training (pre/post)
Outcome measure ODDRS (unpublished rating scale created by Greene Improvement ratings (maternal therapist) Restraints seclusion Restraints Seclusions Eyberg Child Beh. Inv. Parent Stress Index
Outcome Improved slightly more than PT Reduced Reduced Improvement pre to post
29
What does the research tell us?

• So what do we know?
• Based on 4 evaluation studies
• All include children ages 12 or less (2008 study
  does not specify an age range simply school
  age)
• 2 are in inpatient psychiatric hospitals
• 1 is an outpatient mental health clinic
• 1 is a parent training program
• in school settings 0
• The research tells us nothing about the efficacy
  of CPS in school settings

30
What does the research tell us?

• Outcome measures
• ODD Rating Scale (unpublished assessment created
  by the author) improvement ratings from parent
  therapist
• Similar scores to parent training
• Reductions in restraint seclusion (Pre/Post)
• Is this due to student behavior change or adult
  behavior change?
• Eyberg CBI Parent Stress Index (Pre/Post)
• No studies directly measure changes in student
  behavior

31
Concerns

• Only 4 research studies evaluating CPS in 4 years
• 2 on parent training (1 individual training 1
  group training)
• 2 in inpatient psyc facilities
• Make sure research you are looking at takes place
  in settings that match your application
• E.g. school settings v. treatment centers
• 2 of 4 studies have been conducted by the author
  of the program
• Concern if authors are benefiting financially
  from sale of the program

32
Research on CPS in Schools!

• Schaubman, A., Stetson, E., Plog, A. (2011).
  Reducing Teacher Stress by Implementing
  Collaborative Problem Solving in a School
  Setting. School Social Work Journal, 35(2),
  72-93.
• Abstract
• Student behavior affects teacher stress levels
  and the student-teacher relationship. In this
  pilot study, teachers were trained in
  Collaborative Problem Solving (CPS), a
  cognitive-behavioral model that explains
  challenging behavior as the result of underlying
  deficits in the areas of flexibility/adaptability,
  frustration tolerance, and problem solving. It
  was hypothesized that teacher stress would be
  reduced when teachers' understanding of the
  underlying causes of student behavior shifted to
  a framework of skills development, and they began
  using a proactive, positive approach to
  misbehavior (CPS), with the support of mental
  health consultation. Results showed a significant
  decrease in teacher stress, as measured by
  self-report. Further, discipline referrals were
  significantly reduced. Limitations of the study
  and implications for school mental health
  consultation are also discussed. (Contains 4
  figures.)