Dia 1

1
ENQA experiences with external reviews of
agencies - Challenges -
Bruno Curvale President of the ENQA board Bologna
expert Head of international affairs at AERES
29th of May 2009 Vilnius
2
Outline of the presentation

• Introduction
• I - European Quality AssuranceMajor steps
• II - The ESG and the ENQA membership criteria
• Similarities and differences
• III - ENQA experiences
• ENQA coordinated reviews
• The work of the board
• IV - The issue of consistency
• Interpretation
• Notions
• Cultural dimension
• V - Improvements and remediation
• Debriefing
• Training

3
There are many challenges related to

• The context
• The objectives of ENQA
• The link between the ESG and the ENQA membership
  criteria
• The complexity of the two stages evaluation
  process
• The nature of the judgements to be made by the
  review panels and the ENQA board
• The methodology itself

4
I - European Quality AssuranceMajor steps
5
Bologna process and quality assurance developments
Primary responsibility of HE institutions for
quality
Evaluation of EQAR
Cooperation of QA agencies and HE institutions
European cooperation in quality assurance
European Standards and Guidelines
European Register EQAR
04/2008
E4 Group
2009 Leuven Louvain-la-Neuve
1999 Bologna
2001 Prague
2003 Berlin
2005 Bergen
2007 London
2006 Reco. 2006-143-CE
1998 Reco. 98-561-CE
Source Colin Tück, E4, B. Curvale
6
II - The ESG and the ENQA membership
criteriaENQA objectivesSimilarities and
differences between the ESG and the ENQA criteria
7
ENQA objectives

• Promotion of agencies
• ENQA full membership as a recognised sign of
  professionalism
• A single external review of agencies for ENQA
  membership and EQAR listing

8
Differences and similarities between the ESG and
the ENQA Criteria

• ENQA criteria
• Principle based or rules based? Agencies shall
  follow the ESG and have to comply with the
  membership criteria
• Correspond to part 3 of ESG
• Integrate the part 2 of ESG (and by consequence
  the part 1 of the ESG)
• Occasionally associate Standards (ENQA criteria
  1)
• Occasionally mix Standards and guidelines but
  keep the requirement in the conditional (ENQA
  criteria 4)
• Occasionally go further than the standard or the
  guidelines (ENQA criterion 3, ENQA criterion 7)
• Occasionally split Standards in different
  criteria (ENQA criteria 6i and 6ii)
• Have additional requirements (ENQA criteria 8i,
  8ii, 8iii)

But are fundamentally identical to the letter and
the spirit of the ESG, unless it is the opposite
9
III - ENQA experienceENQA coordinated
reviewsThe work of the board
10
The experience

• 7 ENQA coordinated reviews organised so far
  (ANECA, AQA, AQU, ACSUG, AGAE, ARACIS, EUA-IEP)
• 2 to be done (SKVC for sure and ECCE)
• 24 agencies (among 39 full members) are already
  externally reviewed (AAC, AQA, FHR, VLHORA, NEEA,
  EVA, ACQUIN, AQAS, ASIIN, FIBAA, GAC, ZEvA, HAC,
  HETAC, NQAI, NVAO, NOKUT, PKA, AGAE, ANECA, AQU,
  NAHE, OAQ, QAA)
• 8 candidates members are preparing to the review

11
The types and the coordination of the external
review process

• Type A (ENQA membership criteria / ESG)
• Type B (also other aspects)
• - statutory functions (national missions, )
• - ECA Code of Good Practice
• - special context (e.g. bi-national character,
  NVAO)
• - follow-up on previous review
• Co-ordination
• national
• ENQA ( other non-national)

Sources Tibor Szanto, Barcelona 17/03/09
12
The work of the panel

• Is the agency
• - fully compliant
• - substantially compliant
• - partially compliant
• - non-compliant
• with the criterion/ESG standard
• Overall judgement not required (but can be made
  if the panel wishes to do so)

Sources Tibor Szanto, Barcelona 17/03/09
13
The work of the board

• To answer the question Can it follows the
  analysis and thejudgements made by the panel and
  decide to grant the full ENQA membership.
• What the board expects from the report
• ESG Parts II and III fully covered
• Each criteria discussed separately
• Structure (under each criteria)
• - evidence (description, facts and data)
• - analysis (consideration)
• - conclusion (judgement on compliance)
• - recommendation (if any)

Sources Tibor Szanto, Barcelona 17/03/09
14
IV - The issue of consistencyConsistency of
judgement
15
Consistency of judgement is a challenge

• For the panels and for the ENQA board because of
• Situations have to interpreted
• Notions used are difficult
• Language issues
• Cultural dimension
• Implicit knowledge

16
V - Improvements and remediationDebriefingTraini
ng
17
The experience is gathered

• Well organised review processes are determinant
• Internal processes and external processes are
  equally important
• ENQA is learning from what is done by the panels
• Debriefing/brainstorming meeting for the
  training of experts for agency reviews,
  Barcelona, Spain, 17 March 2009
• ENQA has started to train external reviewers
• Training session/workshop for experts for agency
  reviews, 21-22 May 2009, Paris, France

18
The experience is shared

• ENQA develops its documentation and makes it
  available to all
• The Guidelines for national reviews of ENQA
  member agencies
• The Briefing pack for review panel members of
  ENQA coordinated reviews
• The Principles for ENQA coordinated reviews
• are now merged in one single document
• The Guidelines for external reviews for the
  purpose of ENQA membership
• ENQA will continue to develop the methodology of
  external reviews
• The board has now a book of precedents,
  regularly updated by the ENQA secretariat

19
Thank you for your attention
bruno.curvale_at_orange.fr www.enqa.eu
20
The European parliament and the council of the
European Union
Hereby recommend that member States
4. Enable higher education institutions active
within their territory to choose among quality
assurance or accreditation agencies in the
European register an agency which meets their
needs and profile, provided that this is
compatible with their national legislation or
permitted by their national authorities 5. Allow
higher education institutions to work towards a
complementary assessment by another agency in the
European Register, for example to enhance their
international reputation.
Extract of the Recommendation of 15 February 2006
on further cooperation in QA in HE
21
EQAR Criteria
All agencies which comply substantially with the
European Standards and Guidelines for Quality
Assurance (ESG) can be admitted to the Register.
Substantial compliance with the European
Standards and Guidelines is to be evidenced
through an external review by independent
experts. Such a review is coordinated either by a
national authority or another organisation that
is independent from the quality assurance agency
under review. Full ENQA membership, being also
based on substantial compliance with the ESG,
will normally constitute satisfactory evidence
for inclusion in the Register.
Extract from the EQAR website http//www.eqar.eu
22
ESG 3.1, 3.3 / ENQA Criterion 1 Activities

• The Agency undertakes external quality assurance
  activities (at institutional or programme level)
  on a regular basis. These may involve evaluation,
  review, audit, assessment, accreditation or other
  similar activities and should be part of the core
  functions of the Agency. In undertaking its
  activities, the Agency should take into account
  the presence and effectiveness of the external
  quality assurance processes described in Part 2
  of the European Standards and Guidelines (ESG

Extract from the Application form for membership
of ENQA
23
ESG 3.5 / ENQA Criterion 4 Mission statement
The Agency should have clear and explicit goals
and objectives for its work, contained in a
publicly available statement. This statement
should describe the goals and objectives of the
Agencys quality assurance processes, the
division of labour with relevant stakeholders in
higher education, especially the higher education
institutions, and the cultural and historical
context of its work. The statement should make
clear that the external quality assurance process
is a major activity of the Agency and that there
exists a systematic approach to achieving its
goals and objectives. There should also be
documentation to demonstrate how the statement is
translated into a clear policy and management
plan.
Extract from the Application form for membership
of ENQA
24
ESG 3.4 / ENQA Criterion 3 Resources
The Agency should have adequate and proportional
resources, both human and financial, to enable it
to organise and run its external quality
assurance process(es) in an effective and
efficient manner, with appropriate provision for
the development of its processes, procedures and
staff.
Extract from the Application form for membership
of ENQA
25
ESG 3.8 / ENQA Criterion 7 Accountability
procedures
The Agency should have in place procedures for
its own accountability. These procedures are
required to include the following i. a published
policy for the assurance of its own quality, made
available on its website ii. documentation which
demonstrates that ii. a mandatory
cyclical external review of its activities at
least once every five years which includes a
report on its conformity with the membership
criteria of ENQA. Documentation could here either
indicate that a review has taken place or is
planned.
Extract from the Application form for membership
of ENQA
26
Review AND evaluation of the review
Inclusion in EQAR
ENQA full membership
Evaluation of - the review process - the review
outcomes
ENQA board
Register committee
Review of the agency (ESG 3, 2 and 1) specific
ENQA criteria
External review
Procedures, behaviour
Agency
National regulations
Purposes
Q Standards
Specific objectives
1
2