From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

1
From Concept to Implementation The Challenges
Facing Evidence-Based Social Work

• Faculty Research Insights A Series Featuring
  CUSSW Faculty Research
• Edward Mullen Aron Shlonsky
• September 22, 2004

2
Evidence-Based Social Work Is A Concept Awaiting
Implementation

• Much discussed
• Required in EPAS
• Not clearly Defined
• Not yet implemented in pure form

--- the conscientious, explicit and judicious
use of current best evidence in making decisions
regarding the welfare of service-users and
carers (Sheldon, 2003, p. 1).
Placing the clients benefits first,
evidence-based practitioners adopt a process of
lifelong learning that involves continually
posing specific questions of direct practical
importance to clients, searching objectively and
efficiently for the current best evidence
relative to each question, and taking appropriate
action guided by evidence (Gibbs et al., 2003,
p. 6).
Evidence-based medicine has been defined as ---
the conscientious, explicit and judicious use of
current best evidence in making decisions about
the care of individual patients (Sackett, et
al., 1996, p. 71) and the "integration of best
research evidence with clinical expertise and
patient values" (Sackett, Straus, Richardson,
Rosenberg, Haynes, 2000, p. 1).
3
Origin of Evidence-Based Social Work

• Adaptation of evidence-based approaches found in
  medicine and healthcare
• Evolution from earlier attempts to join
  policy-practice-research
• Effectiveness research
• Eclecticism
• Empiricism the scientist-practitioner model
• Social intervention research
• Outcomes measurement
• Systematic review methodologies meta-analysis
• Eileen Gambrill early translator (1999)
• Leonard Gibbs first workbook-text (2003)
• Aaron Rosen Enola Proctor on guidelines (2003)

4
The EBP Model
Sackett et al., 1997
5
Newest EBP Model
Haynes, Devereaux, and Guyatt, 2002
6
Steps of EBP

• Become Motivated to Apply EBP
• Step 1Convert information need (prevention,
  assessment, treatment, risk) into an answerable
  question.
• Step 2Track down current best evidence.
• Step 3Critically appraise the evidence.
• Step 4Integrate critical appraisal with practice
  experience, clients strengths, values, and
  circumstances.
• Step 5Evaluate effectiveness and efficiency in
  exercising steps 1-4 and seek ways to improve
  them next time.
• Step 6Teach others to follow the same process

Gibbs, 2003
7
The Cycle of EBP
Appropriate for this client?
Actuarial risk assessment
Other valid assessment measures?
Clients preferred course or at least willing to
try?
Effective services
Barriers (e.g., cultural conflict
Adapated from Haynes, Devereaux, and Guyatt,
2002 as presented in Shlonsky and Wagner, in
press.
8
Hallmarks of EBP

• Individualized assessment (strengths, needs,
  values, preferences of client)
• Well-formulated question (database query)
• Well-executed search of research literature
• Deciding its applicability to clients
• Considering the evidence together with the values
  and preferences of the client.

Sackett et al., 1997 Gibbs, 2003
9
Origins of Evidence-Based Healthcare Public
Policy

• Used to inform health care policy at hospital,
  regional, national levels (Davies Boruch,
  2001 Romanow, 2002 Gray, 2001 Davies, Nutley,
  Smith, 2000)
• Became component of New Labours modernizing
  government strategy - extended to other sectors
  including social care (Davies, 2004 Walker,
  2001)
• Facilitated by
• Cochrane Campbell Collaborations
• UK ESRCs evidence-based practice policy
  centres
• UK Social Care Institute for Excellence
• Sweden Institute for Evidence-Based Social Work
• Publications e.g., Evidence-based Policy A
  Journal of Research, Debate and Practice

10
Policy Applications

• An approach to policy that
• helps people make well informed decisions about
  policies, programs projects
• EBPolicy contrasts with opinion-based policy
• which relies heavily on either the selective use
  of evidence (single studies irrespective of
  quality)
• untested views of individuals or groups
• Philip Davies (2004), Director of Policy
  Evaluation in the Cabinet Office of the UK Prime
  Ministers Strategy Unit

11
What is evidence-based healthcare?

• Discipline centered on evidence-based
  decision-making about
• -Groups of patients-Populations
• Evidence from wide range of disciplines
• Use of scientific research findings and logic
• Healthcare problems
• Health improvement
• Decision-making in healthcare

Gray 2001
12
Evidence-Based Policy Definition
Decision-making process in which policy-makers,
purchasers, managers, or practitioners together
with key stakeholders make decisions about
problem or need assessment, goals objectives,
interventions outcomes
13
EBPolicy Process (Gray, 2001)
14
Sources of Evidence in EBP
Adapted from Davies, 2004
Evidence
15
EBPolicy Steps

• Finding and appraising evidence pertaining to the
  decision at hand
• Assessment or intervention options
• Intended outcomes
• Research method
• Developing organizational capacity
• Getting the evidence into practice
• Preparing a policy
• Making cultural changes
• Designing systems for implementation
• Implementation
• Monitoring
• Audit

Gray, 2001
16
Examples of Evidence-based Policy Initiatives

• EVIDENCE AND POLICY A JOURNAL OF RESEARCH,
  DEBATE AND PRACTICE
• ESRC Evidence Network - UK Centre for Evidence
  Based Policy Practice
• U.K. Cabinet Office
• University of St. Andrews
• Romanow Commission Canada
• U.S. Implementing Evidence-based Practices
  Project

17
Arguments For Evidence-Based Social Work (EBSW)

• Enhances decision quality
• Fosters learning of assessment skills
• Incorporates client values and expectations
• Fosters evidence search and appraisal skills
• Makes best use of best evidence
• Framework for self-directed, life-long learning
• Identifies gaps in knowledge
• Common interdisciplinary language

Sackett et al, 2000
18
Arguments For Evidence-Based Social Work (EBSW)

• Eschews authoritarian practices and policies
• Promotes SW ethics through
• Informed consent
• Using effective services
• Wisely allocating scarce resources
• Involving clients in practice process
• Individualizing personalizing decisions

Gambrill, 2003
19
Arguments For Evidence-Based Social Work (EBSW)

• Helping clients develop critical appraisal skills
• Involving clients in design and critique of
  practice and policy related research
• Involving clients as informed participants who
  share in decision making
• Recognizing clients unique knowledge in terms of
  application concerns
• Promoting transparency and honesty
• Encouraging a systemic approach for integrating
  practical, ethical and evidentiary issues
• Maximizing the flow of knowledge and information
  about knowledge gaps

Gambrill, 2003
20
Challenges Facing EBSW

• Evidence-of-effectiveness
• Authority
• Conflicting hierarchy
• Definition of evidence
• Shortage of evidence
• Variation by field of practice
• Inflation of evidence
• Small transient effects
• Assessment validity
• Nomothetic versus ideographic knowledge
• Efficacy versus effectiveness
• Diffusion, adoption and implementation
• Policy system level issues
• Training sustaining

21
Evidence-of-effectiveness Challenge

• A key assumption of EBP is that it results in
  better outcomes than other forms of practice
• Yet, this assumption is not empirically testable
• This proof is no more achievable for the new
  paradigm (EBM) than it is for the old, for no
  long-term randomized trials of traditional and
  evidence-based medicine are likely to be carried
  out. (Evidence-Based Medicine Working Group,
  1992, 2424)

22
Authority Challenge

• As originally conceived the founders of
  evidence-based medicine envisioned that medical
  practitioners would seek out evidence appraise
  the quality of that evidence, alone or in teams
  of practitioners
• The new paradigm puts a much lower value on
  authority. The underlying belief is that
  physicians can gain the skills to make
  independent assessment of evidence and thus
  evaluate the credibility of opinions being
  offered by experts (Evidence-Based Medicine
  Working Group, 1992)

23
Authority Challenge

• Lack of evidence that EBM can be transmitted into
  practice
• Medical students have not shown an interest in
  becoming evidence-based practitioners only
  evidence users
• As evidence users rather than independent
  appraisers, practitioners are simply replacing
  one authority with another researchers are the
  new authorities!

24
Authority Challenge

• To date it has not been shown that average
  practitioners can gain the skills needed to
  conduct independent assessments of the evidence
• Practitioners can become evidence-users not
  evidence-based practitioners say Upshur Tracy
• Evidence-users accept evidence that has been
  vetted by others thus replacing one authority
  for another! But what is the basis for the
  authority of the evidence appraisers?

25
Conflicting Hierarchy Challenge

• Hierarchy of Evidence Ranked by Quality
• N of 1 randomized trials
• Systematic reviews/Meta-analysis of randomized
  trials
• Randomized Controlled Trials
• Systematic review of observational studies
  addressing patient-important outcomes
• Physiological/Laboratory experiments
• Unsystematic clinical observation

26
Conflicting Hierarchy Challenge

• Practice recommendations (e.g., practice
  guidelines) are to be based on the soundness of
  the evidence
• Many evidence hierarchies have been published
  these hierarchies are not commensurable (e.g.,
  randomized, controlled trials and meta-analysis
  versus well designed epidemiological studies)
• Fundamentally these hierarchies do not rest on
  evidence but rather on the consensus of experts
  the beliefs of the few!

27
Definition of Evidence Challenge

• Because contemporary health care and human
  services are multidisciplinary there are many
  differing perspectives that require consideration
• Accordingly, the concept of a univocal
  understanding of evidence is not sustainable
• A restrictive view of evidence is not sustainable

28
Shortage of Evidence Challenge

• The question that faces proponents of EBP is
  whether there are enough high-quality studies so
  that evidence-based decisions can be made.
• Surprisingly for a field that places a high
  premium on research, few studies have examined
  this.
• The conclusion at this point, based on just a few
  studies, is that there are still many decisions
  that are made that are not based on good
  evidence, but the picture is not nearly as bleak
  as opponents to EBP would have us believe.
• Professionals must remember, though, that when
  they make decisions for which little or no
  evidence exists, that they should exercise
  caution and perhaps be even more vigilant in
  monitoring outcomes.

29
Shortage of Evidence

• Varies by field of practice
• Confounded by inflation of evidence by proponents
  and users
• Clouded by dominance of small and transient
  effects
• Applies to assessment evidence as well as
  intervention evidence

30
Shortage of Evidence Varies by Field of Practice

• Adequacy varies considerably by field of practice
• Most robust is mental health
• In other fields of practice much more of a
  patchwork
• Unevenness of the research base across fields of
  practice looms as a major challenge

31
Inflation of Evidence Challenge

• Bias in reported research due to
• Tendency for investigators or assistants to give
  a leg up to interventions they favour -
  expectation for improvement
• Dependence on client self-report to measure
  intervention outcomes which may reflect social
  desirability, expectancy, cognitive dissonance
  effects

32
Small Transient Effect Size Challenge

• The rule is small effect sizes which may not be
  clinically or social important
• Gains frequently do not persist, especially for
  complex and chronic problems such as addressed by
  social workers

33
Assessment Challenge

• Reliable, valid, relevant assessment tools are
  needed to support evidence-based social work
  practice
• Rapid assessment measures have contributed
  greatly (Corcoran Fisher, 2000)
• Social work practitioners generally do not use
  any standardized assessment procedures to guide
  their practice (Mullen Bacon, 2004)

34
Shortage of Evidence Challenge

• Evidence-based social workers must remember that
  when they make decisions for which little or no
  evidence exists, that they should exercise
  caution and perhaps be even more vigilant in
  monitoring outcomes
• Knowledge gaps point the way to needed research
• Methodological corrections should be instituted
  to deal with the inflation of evidence issue
• Where interventions are found to have small with
  transient effects of limited importance this
  should be acknowledged and addressed
• The underutilization of assessment instruments
  should be corrected by enhance training in the
  use of appropriate instruments dissemination of
  information about available instruments.

35
Challenge of Applying Results to Individuals

• Results of RCTs are analyzed by comparing the
  mean score of the experimental group against that
  of the placebo or control group (or some
  comparable summary statistic). This masks the
  fact that there is always individual variability
  around the means, overlap in the distributions
  of scores for the two groups. The result of this
  is that a proportion of people in the
  experimental group actually do worse than some in
  the control group and, conversely, some in the
  comparison group improve more than some people in
  the active treatment group.
• Practitioners cannot blindly apply a proven
  procedure and assume that a particular individual
  receiving that procedure will benefit

36
Challenge of Applying Results to Individuals

• We are at least able to quantify the probability
  with which an individual person will respond to a
  given procedure (Number Needed to Treat)
• Alternative to using evidence-based interventions
  with their known rate of failure is to use
  unproven procedures, based only on the hope that
  they may work
• Practitioners can and should view each case or
  situation as an N 1 study and collect data

37
Efficacy Versus Effectiveness Challenge

• The efficacy versus effectiveness challenge
  requires a careful consideration of the trade off
  between internal and external validity
• Claims to evidence-based practice typically must
  be grounded in random, controlled trials which
  usually translate into efficacy studies
• Such studies do not address how effective such
  interventions would be in real world contexts
• At the present time there is a paucity of
  evidence of relevance to social work based on
  effectiveness studies conducted in real world
  contexts
• it may be more efficient to begin with studies
  based on conditions as realistic as possible so
  as to avoid the misinformation that too
  frequently comes from misinterpretations of
  efficacy studies

38
Adoption Implementation Challenge

• Two major challenges
• (1) adoption effective implementation of
  evidence-based practice in social agencies
• (2) educating for evidence-based practice

Sisyphus
39
Training, Time, Resources Challenges

• EBP requires
• Training in search techniques
• Training in critical appraisal
• Computer resources
• Electronic resources

40
Killer Bs

• Low BASE rate
• Incompatible client or community BELIEFS
• BAD BARGAIN in terms of scarce resources
• BARRIERS too high

41
Adoption Implementation Challenge

• Motivate core group
• Acceptance of EBP
• Facilitate adoption
• Implementation of effective services
• Evaluate outcomes

Rogers Shoemaker, 1971
42
Gira, Kessler and Poertner (2004)

• Survey of reviews of adoption of research
  evidence medical practice
• Educational outreach visits and audit and
  feedback showed weak to moderate effects
• Certain types of continuing education and the use
  of computers as decision aids showed moderate
  effects
• There are no magic bullets (Oxman et al., 1995)

• The literature from health care suggests that
  disseminating information alone is insufficient.
  Many interventions have been designed to improve
  practitioners adherence to EBP guidelines and
  are differentially effective. To date, no
  intervention has demonstrated powerful effects
  (p. 77-78).

43
Adoption Implementation Challenge

• Can social agencies adopt EBP approach?
• Can social workers become EB practitioners?
• What are the barriers? Facilitators?
• What resources are needed?
• What training is required?
• What systems need to be established?
• How will EBP change client experience?
• What outcomes will be achieved?

44
Challenges in Teaching Evidence-based Social Work

• Fear of Loss of Human Context
• Disconnect between classroom and field
• Overwhelmed by new skills and knowledge needed

(Bilsker Goldner, 2004)
45
A Challenge to Critics

• Knowledge and information revolution
• Valuable resource
• Cannot be ignored
• Must be harnessed

46
A Measured Optimism

• Real world constraints-limitation of
  approach-social and political realities
• Great promise, but need for caution-harnessing
  knowledge and information-management of scarce
  resources-unknowns abound
• Hold EBP to its own evidentiary standards-the
  jury is still out

47
Knowledge needs to be managed much more
effectively than it has been in the past
The challenge is to rethink our view of what
knowledge is how best to facilitate its rapid
generation, sharing, and application in a manner
that closes the policy-research-practice gap