Evaluating widening participation a series of unfortunate events

1
Evaluating widening participation a series of
unfortunate events

• A case for a multi-functional approach to the
  evaluation of widening participation
• Chris Howard
• Institute for Access Studies
• c.d.howard_at_staffs.ac.uk

2
Overview

• Evaluation theory uncovered
• Evaluation of widening participation
• A Multi-functional approach to evaluation
• Evaluating widening participation using a
  multi-functional approach to evaluation.

3
Evaluation Theory

• There is no clear account of evaluation (Shaw,
  2000 Thomas, 2000).
• The discourse on evaluation contains
  contradictions and overlaps on the description,
  terminology and uses of evaluation (Howard, 2004
  Shulha and Cousins, 1997).
• In order to develop a coherent account of
  evaluation it is necessary to focus on both the
  theoretical and practical dimensions of
  evaluation.

4
Evaluation Theory Theoretical dimension

• Evaluation comprises of four functions (Howard
  2004).
• Impact function addresses the question what
  difference has occurred as a result of the
  project?
• Improvement function addresses the question how
  can a project be improved to become more
  effective?
• The adherence function addresses the question
  Did a project do what it was supposed to do?
  Thus adherence focuses on the comparison between
  the implementation of a project as it was
  observed with the intended delivery.
• The generalised knowledge production function
  addresses questions centred on What knowledge
  have we gained from the project? This function
  has predominantly addressed the specific question
  What knowledge have we gained about the social
  issue/problem a project has addressed?.

5
Evaluation Theory Theoretical dimension

• Theoretical paradigms conceptualise evaluation.
• There are four theoretical paradigms, which are
  positivism, critical realism, pragmatism and
  constructivism (Milne et al. 2004 Pawson and
  Tilley, 1997).
• Paradigms are distinguished by their underlying
  epistemological and ontological assumptions
  (Ovretveit, 2002).
• Paradigms conceive evaluation based on
  incorporating functions that correspond with
  their own assumptions about knowledge and the
  social world.

6
Positivism

• Assumes natural world is synonymous with the
  social world in that they both consist of
  objective and quantifiable truths (Morss,1996
  Walsh, 1998). Therefore the methods of inquiry
  used in the natural world should be applied to
  the social sciences (Bryman, 1988).
• Ontology- social reality is objective
• Epistemology- there are objective truths
• Evaluative Functions- impact, adherence,
  generalised knowledge production.

7
Critical Realism

• Assumes the experience of reality is a social
  construct but claims that there is a real world
  with underlying structures and mechanisms (Hughes
  and Sharrock, 1997).
• Ontology- a real world that is stratified into
  different domains.
• Epistemology- there are underlying causal
  mechanisms that connect structures and events.
• Evaluative Functions- impact, improvement,
  adherence and generalised knowledge production.

8
Constructivism

• Assumes reality is a product of social activity
  (Kukla, 2000) where individuals create meaning
  through interaction with each other and their
  environment (Gredler, 1997 Prat and Floden,
  1994).
• Ontology reality is subjective.
• Epistemology existence of multiple truths with
  equal validity.
• Evaluative Functions improvement.

9
Pragmatism

• Assumes that evaluation should meet the needs of
  policy makers and those in a position of power
  (Patton, 1986, 1997 Weiss, 1987).
• Ontology reality regulated and constructed by
  those in power.
• Epistemology- knowledge is what is considered
  pragmatically acceptable by those in power.
• Evaluative Functions- not fixed.

10
Evaluation Theory Practical Dimension

• Evaluation approaches
• Translate the theory espoused by theoretical
  paradigms and apply this in a practical setting
  (Hall and Hall, 2004 Shaw, 2000).
• Specific to theoretical paradigms.
• Share the underlying epistemological and
  ontological assumptions of theoretical paradigms.
• Advocate a set of research methods that
  compliment the assumptions of theoretical
  paradigms.
• Do not necessarily adopt all the functions
  espoused by theoretical paradigms

11
Widening Participation

• Widening participation has become a national
  concern since the publication of the Dearing
  report (1997) and Kennedy report (1997).
• Previous strategies intended to diversify the
  Higher Education (HE) student population have
  been of limited value (Lumby and Wilson, 2003).
• The statistical evidence for widening
  participation
• Participation rates for low socio-economic
  groups have increased by 8 percent during
  1990-2000 in comparison to an increase of 11
  percent for high socio-economic groups during the
  same period (DfES, 2003).

12
Widening Participation

• The national response has been to focus on a
  structured national strategy (HEFCE 06/04).
• This has taken the form of Aimhigher, which is an
  aspiration raising project implemented regionally
  using a range of activities (DfEs, 2003).
• The Widening Participation and Fair Access
  Research Strategy consultation paper (WPFARS)
  (HEFCE 06 2004), acknowledges that it is crucial
  to evaluate this widening participation strategy.
  

13
Widening Participation

• The suggested evaluation outlined in the WPFARS
  and the National Aimhigher evaluation seminar
  (May 2004), is based on meeting measurable
  targets e.g. 50 people attend a taster day.
• The evaluation of Higher Education Institutions
  (HEI) widening participation activities is based
  on meeting targets to determine the distribution
  of HE funds (HEFC 59/2003).
• This corresponds with the target of 50 per cent
  of 18-30 year olds in HE by 2010 (DfES, 2003).
• Therefore the evaluation approach is derived from
  pragmatism and adopts an impact function based on
  meeting measurable targets.

14
Limitations of current evaluation

• Can not identify how or why widening
  participation works.
• Can not identify aspects of current widening
  participation that may need improving.
• Can not identify the unintended impact of
  widening participation.
• Outputs accounted for may not actually relate to
  widening participation.

15
Multi-Functional Approach to Evaluation

• Using a critical realist framework, we can
  develop an evaluation approach that adopts all
  four evaluative functions.
• The four functions are not exclusive or in
  competition with each other.
• The four functions can be incorporated within an
  evaluation approach to inform and triangulate
  with each other.

16
Multi-Functional Approach to the Evaluation of
Widening Participation

• Adopting a multi-functional approach to
  evaluation would
• Impact function would establish what widening
  participation drivers work and in what contexts.
• Identifying the impact of a project also enables
  areas to be identified for improvement (Scriven,
  1993) (improvement function).
• Generates knowledge about the mechanisms that
  produce widening participation into HE (Pawson
  and Tilley, 1997) (Generalised Knowledge
  Function).

17
Multi-Functional Approach to the Evaluation of
Widening Participation

• Improvement function challenges the
  appropriateness of current widening participation
  and identifies aspects for improvement.
• Identifying and implementing improvements to
  widening participation will relate to a positive
  effect on project impact (Sanders, 1999) (Impact
  function).
• Identifying areas for improvement contributes to
  producing new knowledge about widening
  participation (Lipsey, 1993).

18
Multi-Functional Approach to the Evaluation of
Widening Participation

• Adherence function Measures whether the widening
  participation strategy is being delivered.
• Monitoring the delivery of widening participation
  is necessary when measuring its impact (Weiss,
  1998) (impact function).
• If widening participation is implemented
  incorrectly this can by improved by ensuring that
  is delivered correctly (Mark et al. 2000)
  (Improvement function).
• The means of monitoring the delivery of the
  strategy informs how to monitor future and/or
  strategies in other areas (Generalised knowledge
  production function).

19
Multi-Functional Approach to the Evaluation of
Widening Participation

• Generalised knowledge production is informed by
  the impact, improvement and adherence functions.
• Impact provides data about what mechanisms work
  for widening participation, which can be used to
  design future strategies.
• Improving widening participation provides
  knowledge when designing future strategies.
• Monitoring widening participation provides
  knowledge about the means of monitoring the
  delivery of future widening participation
  strategies.
• Finally, the function can be used to indicate how
  a multi-functional approach operated in the
  context of widening participation and how it can
  be applied across other settings.