Evaluation and Feedback Using the matrix framework

1
Evaluation and Feedback Using the matrix
framework
2
Matrix, evaluation and feedback

• Aims
• Describe our use of matrix as a means for
  developing evaluation and feedback
• A little context
• The emerging importance of evaluation
• Matrix
• Moving it forwards engaging the staff
• The future

3
Student Services

• Accommodation
• Careers
• Chaplaincy
• Counselling
• Disability Services
• Nursery
• Sport and Recreation
• Student Support

4
Context

• Integrated department
• Need to encourage collaboration
• Need to share best practice and experience
• Growing University
• Need to underpin with clear framework
• Enhance credibility for department
• Need for a quality enhancement process

5
Matrix standards

• People are made aware of the Service and how to
  engage with it
• Peoples use of the service is defined and
  understood
• People are provided with access to information
  and support in using it
• People are supported in exploring options and
  making choices

6
Matrix Standards

• Service Delivery is planned and maintained
• 6. Staff competence and the support they are
  given are sufficient to deliver the service
• 7. Feedback on the quality of the service is
  obtained
• 8. Continuous quality improvement is ensured
  through monitoring, evaluation and action

7
Matrix

• We decided that Careers should seek matrix
  accreditation in 2003
• Needed to have some external validation
• Would act as a pilot
• We decided that the whole Department would seek
  accreditation, and Careers re-accreditation, in
  April 2006
• Re-accreditation of whole Department in 2009

8
Staff Reaction ?
9
Staff Reaction

• Mostly Positive
• Get recognition
• Give clarity
• Emphasis on staff involvement in planning and
  development
• Importance of good staff development processes
• Enthusiastic
• Confidence in the quality of what we do

10

• But,
• Evaluation and Feedback ????

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Staff Reaction

• Yet more work !
• Anxiety about scrutiny by the uninformed
• Some trepidation about a new challenge
• The unknown
• Fear of intrusive scrutiny
• But also some excitement and willingness to
  engage

17
Staff Reaction

• The challenge
• keep the enthusiasm
• use the concern about feedback and evaluation as
  a positive thing

18
Staff Reaction

• So, whats the problem with evaluation and
  feedback
• Some definitions

19
Evaluation and feedback

• Evaluate - verb
• 1 form an idea of the amount or value of assess.
  
• 2 Mathematics, find a numerical expression or
  equivalent for (a formula, function, etc.).
• Feedback - noun
• 1 information given in response to a product,
  performance etc., used as a basis for
  improvement.
• 2 the modification or control of a process or
  system by its results or effects.
• 3 the return of a fraction of the output of an
  amplifier, microphone, or other device to the
  input, causing distortion or a whistling sound.
• OED - selective

20
Evaluation and Feedback

• But in reality, dont we all evaluate things all
  the time ?How was lunch ?What do you think of
  the venue?How was your journey here?What do you
  think of this so far ?
• What we may do less, is give feedback

21

• However, knowing if something works means knowing
  what it was supposed to be doing in the first
  place
• Thats not always as obvious as it seems
• Matrix offers a way to clarify the intention
• And then the possibility of evaluating it

22
The journey

• Using matrix standards
• The first four elements are easy
• People are made aware of the Service and how to
  engage with it
• Peoples use of the service is defined and
  understood
• People are provided with access to information
  and support in using it
• People are supported in exploring options and
  making choices
• Well, maybe not quite so easy

23
The journey

• Diversity of services
• What links together ?
• Draw together with clear mission and aims
• Agreed by broad consultation
• Essential to get active engagement at all levels
• Set in the context of the Universitys mission
  and aims

24
The journey

• Also developed a Code of Conduct for staff
• Covers interactions with clients
• Practice
• Staff development
• Boundary issues
• Ethical behaviour and integrity
• Developed by and applied to ALL staff
• Also clear confidentiality guidelines and
  policies on data protection and other matters

25
(No Transcript)
26
The Journey

• As teams,
• Developed clear statements of service
• How do we work ?
• Who can use the services that we provide ?
• What can our service users expect from us ?
• How do we let them know about this ?
• How can they access these services ?
• What are the limitations of what we do ?
• Do these statements fit with the overall aims and
  mission ?

27
(No Transcript)
28
The journey

• So, now, emerging clarity around what we do
• The first four elements of matrix
• The next two elements
• Service Delivery is planned and maintained
• 6. Staff competence and the support they are
  given are sufficient to deliver the service

29
Planning

• Strong link between departmental planning and
  university processes
• University processes define corporate objectives
• Departmental planning reflects particular
  contribution to objectives
• Annual cycle
• Production of development plans (3 years forward
  but concentrating on coming year)
• Annual review on performance in last year

30
(No Transcript)
31
Planning

• Engage staff
• Consult through teams
• Get their ideas about what can be done
• Emphasis that resource allocation is through the
  planning process
• Encourage creativity
• What can we improve ?

32
Staff Development

• Already institutionally strong
• Whole institution IIP since 1999
• Process of annual Personal Development Reviews
• explicit link to development plan
• Strengthened by clarity around departmental
  mission and by development of code of conduct and
  statements of service

33
Matrix elements

• And so finally,
• 7. Feedback on the quality of the service is
  obtained
• 8. Continuous quality improvement is ensured
  through monitoring, evaluation and action

34
(No Transcript)
35
Matrix elements

• And so finally,
• 7. Feedback on the quality of the service is
  obtained
• Continuous quality improvement is ensured through
  monitoring, evaluation and action
• First step
• Addressing the fears
• Identifying existing practice
• Uncovering the everyday

36

• Evaluation happens all the time
• Existing major mechanisms
• Annual Review
• National Student Survey
• International Student Barometer
• External Assessments
• Committee Scrutiny
• Internal Audit
• Risk assessment
• KPIs
• Some of this unfocussed and not terribly helpful

37

• But we all do it all the time too
• Matrix helped us to understand what it was that
  we were trying to get to and to focus on what
  mattered
• Also to use existing evaluation for our purposes
  and to rebut the ill-informed
• And staff uncovered huge amounts of feedback
  all the time in their everyday work.

38

• Informally and formally
• did that give you the information you needed
• thanks
• E-mails
• Cards
• User surveys
• Comment cards
• Specific events
• Focus groups
• Liaison with other staff

39

• Staff were constantly getting feedback
• Matrix
• helped us be less British about it
• It was okay to get positive
• Negative was good too because it showed how we
  might do better
• Made it about the service we offered not about us
  our professional behaviour not us

40

• Second step
• Sharing the practice
• Understanding what worked
• Recognising that there are a necessary variety of
  ways to evaluate and gather feedback no one
  size fits all no perfect handbook
• If it gives you information about how your
  service works and whether it meets its aims
  its good

41

• Third step
• Making sure that this all links back to further
  development
• At a strategic level
• Through the development plan
• Into university policy development
• At a broad operational level
• Through regular consideration at management team
• Through quarterly reviews of progress
• At an immediate day-to-day level
• By reflection
• By sharing practice
• By planning activity

42

• Fourth step
• Identifying enhancements to evaluation
• Broad survey of student users
• Detailed monitoring of service users
• Using Student Experience Committee

43

• Where are we now ?
• Clear sense of purpose
• Fit with University
• Confidence about our practice
• Openness to change
• Culture of welcoming scrutiny

44

• The Future
• Ever more important
• Scarcer resources
• More diverse students
• More challenging situation
• More scrutiny
• Need for credibility, and the capacity to prove
  that we know what we do and that we deliver

45

• One note of caution
• We have to evolve new ways of seeking evaluation
  and feedback and can never depend simply on
  asking students
• That is always a component
• But there is a risk of survey fatigue
• A variety of ways to evaluate what we do but a
  clear framework for translating that into
  development

46

• Finally,
• Thank you and dont forget to fill in your
  evaluation forms
• we will shred them later