A Pragmatic approach to Evaluation in Utilisation Research

1
A Pragmatic approach to Evaluation in Utilisation
Research

• Professor Brendan McCormack
• (with Rob Garbett)

2
Current Challenges in Evaluating Utilisation

• Little conceptual clarity and murky language
• Dominant focus on evaluating technical
  interventions (instrumental utilisation)
• Linear and logical approaches to evaluation that
  dont always take account of complex
  organisations
• Reliance on input and output models of evaluation

3
Current Challenges in Evaluating Utilisation

• Complexity of decision-making processes, meaning
  that no single evaluation mechanism can capture
  this complexity
• The balance of process and outcome and the
  privileging of one over the other
• Evidence-based person-centred practice what
  are we doing to evaluate the person-centredness?
• What is it we are evaluating?

4
Where is the evaluation of person-centeredness in
the current utilisation agenda?

• Receiving the right intervention, at the right
  time etc based on an assessment of a need/problem
• Person as a disconnected entity from the context
  in which they receive healthcare
• Practitioners as disconnected entities from the
  contexts in which they provide healthcare

5
A Problem - Unclean endoscopes

• Major public concern and media profile
• RGH 3000 patients recalled
• 2 departments in RGH involved DPU and main
  theatre suite (Same unit)
• DPU no cases of contaminated scopes
• Main theatres many cases of contaminated scopes
• DPU use guidelines developed by the Association
  of Endoscopic Surgeons
• Main theatres use guidelines developed by the
  National Association of Theatre Nurses
• Similar but differently worded guidelines leading
  to differing interpretations

6
What questions does this example raise for us?

• Why the use of two different sets of guidelines
  in the same unit?
• What interpretations led to safe and unsafe
  practices?
• What changes would have led to the agreement of
  best practice?
• What contextual factors acted as barriers to the
  sharing of knowledge?

7
Contextual Factors as Barriers

• Person-centred perioperative nursing project
  (Kerr, McCormack et al, in progress)
• 150 hours of non-participant observation
• Appreciative inquiry groups
• Stakeholder analysis with steering group

8
Barriers Identified

• Culture of busyness call for the next one!
• Inconsistent/inappropriate approaches and styles
  of leadership
• Ineffective use of the nursing resource available
• Poor delineation of roles
• Reactive training rather than learning through
  practice
• Factions, Cliques and Tribes
• Lack of person-centredness, i.e.
• Patient as product
• Nurse as producer
• Doctors as product controllers
• Managers as quality controllers

9
Context

• PARiHS Framework Culture, Leadership and
  Evaluation
• Evaluation weak
• Absence of feedback
• Narrow use of performance information
• Reliance on single methods

10
Being person centred

• Creating the conditions that help us to
• Know another as a unique individual
• Understand and acknowledge individuals beliefs,
  values, wants and needs
• Provide care characterised by flexibility,
  mutuality, respect and care
• Create environments that are flexible, respectful
  and caring of peoplebbbb

11
The person centered nursing project

• 8 clinical areas in the intervention group, 3 in
  the control group
• Includes critical care, acute areas,
  peri-operative care, rehabilitation and clinics

12
Methodology

• Quasi-experimental design
• Pre-post test Instruments to measure particular
  dependent variables via patient and staff
  questionnaires at 4 monthly intervals
• Intervention Phase
• Practice Development (PD) Framework
• Mapping the journey
• Qualitative data sources

13
Quantitative measures

• Two questionnaires developed and administered
• Nursing Context Index (NCI)
• 21 Constructs/ Caring Dimensions Index
• Perceptions of Nursing Index (PNI)
• Satisfaction and Experience of being nursed

14
Qualitative sources

• Taping of interactions
• Practically challenging
• Field notes
• Meetings, working with individuals
• Thoughts and feelings
• Contextual issues staffing, competing priorities
  etc.

15
Using the data

• Problematising
• Workload scores and well, what do we do with
  our time?
• Questioning and reflecting
• Communicating constructs and how do we get
  through to people?
• Confirming

16
Food for thought

• The project is giving us food for thought,
  especially the information from the nurse
  questionnaires, it gives us an idea of how the
  staff are really thinking. Also what (our
  facilitator) has told us what she is hearing on
  the tapes.
• This project has spurred us to make changes
  which we never had thought we needed to make, for
  example how EENT and recovery are actually
  managed staff wise. It has identified that we
  need to perhaps have more senior staff nurse
  grades. And the noise level in the unit has been
  reduced after feedback.
•  I particularly enjoy being involved. It has
  certainly motivated me to make and accept changes
  and differences to the department to ensure that
  it is more patient centred and person centred.
• I know that over time when staff do see the
  benefits to patients, the department and to
  themselves that they will be less timid about the
  project. This will take time and more upbeat
  promotion from the team

17
Mapping the journey

• Evidence of reduced stress in all intervention
  areas over 1 year
• Greater sense of feeling supported and of
  workload becoming more manageable

18
Some examples

• Not enough time to complete all nursing tasks
• October 03 40 said often, frequently or always
• April 04 21 said often, frequently or always
• Not enough staff to adequately cover the unit
• October 03 80 said often, frequently or always
  
• April 04 53 said often, frequently or always
• Lack of opportunities to talk openly about
  problems on the ward
• October 03 40 said often, frequently or always
  
• April 04 5 said often, frequently or always

19
Tape recorded data Interaction between nurses
and patients (2)
Getting work done
Sharing decision making
Opportunities for interaction occurred while
getting tasks done for example drug
rounds Interaction could be mainly task focused
There was evidence that these could be used to
work with patients, for example making choices
about whether to take analgesia
20
Reduction in stress

• Evidence that staff feel
• More appreciated
• Clearer about their own development
• More supported in the workplace

21
Some examples

• Work in the clinical areas has provided
  opportunities for people to talk, and the
  emphasis of person centeredness is resulting in
  more openness
• People are saying things to each other that
  wouldnt do at the beginning
• Working on changes provides people with
  recognition and affirmation
• This project has given me a new interest

22
Feeling more in control

• The journey in intensive care
• Taking control over continuity of care
• Becoming more reflective, learning from practice

23
Tape recorded data Interaction between nurses
and patients (1)
Showing sympathetic presence
Chatting
For example, picking up on a persons comments
to find more out about them and their home
circumstances
For example, talk that appears warm and friendly
but without obvious therapeutic intent
24
Evaluation as praxis

• Praxis
• Doing action.
• Concerned with an ethical end that cannot
  always be predetermined in advance and is context
  dependent
• Quality of the end product is inseparable from
  the process of getting there.
• Not rule-following behaviour but based on
  practical wisdom, i.e. combined perception,
  reasoning, virtue and technical competence.

25
A Programmatic Approach
Mechanism
Context
Stakeholder, concerns, claims and issues
Multi-method
Stakeholder, concerns, claims and issues
Multi-method
Interactions
Process Evaluation/Outcomes
Multi-method
Outcomes
after McCormack, 2000
26
Conclusions

• Using multiple sources of data allows us to
  demonstrate the impact of a practice development
  programme over time
• Placing a quasi-experimental design within an
  overarching philosophy of praxis enables
  ownership, participation and changes in culture
  whilst ensuring rigour.
• A programmatic approach to evaluation may offer
  us a way forward in the development of our
  understandings of knowledge use