Qualityand training for 'auditors' and 'reviewers' in Higher Education Geoffrey Robinson Honorary Pr

1
Quality and training for 'auditors' and
'reviewers' in Higher EducationGeoffrey
RobinsonHonorary ProfessorSchool of Geography
and GeosciencesUniversity of St Andrews
2
Background

• QUALITY ASSURANCE AGENCY FOR HIGHER EDUCATION
  (QAA)
• Teaching Quality Assessment from 1993
• Subject Reviews variously from 1995
• Institutional Audits and Enhancement Led
  Institutional Reviews
• Degree-awarding Powers and University Title

3
Subject Reviews Required Qualities of Reviewers

• Personal credibility, in the subject area, with
  academic peers in UK higher education, or
  equivalent industrial/professional credibility.
• Current experience of delivering teaching,
  supporting learning, and examining at level H or
  M within the subject area.
• Experience of working with programme
  specifications written for programmes in the
  subject area a good understanding of programme
  entry requirements, and an ability to interpret
  progression statistics and familiarity with
  comparable programmes and standards of awards in
  other institutions.
• Ability to assimilate a large amount of disparate
  information, and to analyse and draw reliable
  conclusions about complex arrangements.
• Ability to identify, plan and follow lines of
  investigations to meet a task specified by an
  audit team, using a variety of sources, including
  documentary and oral evidence, in order to draw
  secure conclusions.

4
Subject Reviews Training of Reviewers

• Review process involved a lengthy period with
  activities for individual subjects not unlike
  sequence presented later for audit
• Review visits were of four days duration and
  training was to prepare reviewers for all
  activities involved
• Review co-ordinators similarly trained, several
  sessions over a year with refresher training over
  period of contract
• Shadowed mentor for a review and observed by
  mentor for first of own reviews.

5
ENQA Guidelines for Quality Assurancein the
European HE Area

• Noteworthy elements include
• the experts undertaking the external quality
  assurance activity must have appropriate skills
  and be competent to perform their task
• the exercise of care in the selection of experts
• the provision of appropriate briefing or training
  for experts
• the use of international experts
• participation of students
• ensuring that the review procedures provide
  adequate evidence to support the findings and
  conclusions reached
• the use of the self-evaluation/site visit/draft
  report/published report/follow-up model of
  review
• recognition of the importance of institutional
  improvement and enhancement policies as a
  fundamental element in the assurance of quality.

6
Outline of the QAA institutional audit process

• Preliminary information (includes published TQI)
  at least 28 weeks prior
• Preliminary meeting at least 24 weeks
• Institutional briefing paper -10 weeks
• Briefing visit - 5 weeks
• Audit visit up to 5 working days
• Audit findings - 2 weeks
• Schedules for draft report etc.

7
Institutional Audit Aims and Methods (1)

• Institutional audits examine
• the effectiveness of an institution's internal
  quality assurance structures and mechanisms
• arrangements for maintaining appropriate academic
  standards and enhancing the quality of
  postgraduate research programmes
• building systematically upon the outcomes of
  their internal quality assurance procedures,
  reports of external reviews, etc
• the accuracy and completeness of the
  institutions published information about the
  academic standards of its awards and the quality
  of its educational provision.

8
Institutional Audit Aims and Methods (2)

• ACADEMIC INFRASTRUCTURE
• Code of practice for the assurance of academic
  quality and standards in higher education
• The framework for higher education qualifications
  in England, Wales and Northern Ireland
• Subject benchmark statements
• Programme specifications
• Progress files
• Guidelines on the accreditation of prior learning

9
Institutional Audit Aims and Methods (3)

• Trailing - gathering evidence in an organised
  way, from documents and meetings, to test the
  effectiveness of particular institutional
  approaches, policies or procedures and their
  impact.
• Sampling trails - follow the implementation and
  outcomes of institutional policies and procedures
  from institutional to programme level, and back
  a vertical sample through the institution's
  management levels to help the audit team
  understand how the institution's approach to
  managing quality and standards works in practice.
  
• Focus on an institution's internal periodic
  review procedures which conduct trails of this
  type for themselves (Code of practice).
• Selection of trails made by audit team at the
  briefing visit following discussion with the
  institution.

10
Role of Auditor (Reviewer)

• Comment on briefing paper(s) from institution
• Attend both visits
• Chair meetings with staff and students
• Read and note any supplementary information
  requested and provided
• Conduct investigations - includes trailing - keep
  personal notes on developing themes
• Begin preparing text (interim) between visits
• Prepare text for report
• Comment on draft report
• Comment on institutional response.

11
Institutional Audit Required Qualities for
Auditors

• At appointment, auditors expected to have current
  expertise and experience of teaching and
  management in HE.
• Wide experience of academic management and
  quality assurance at institutional level in UK
  higher education.
• Personal and professional credibility with heads
  of institutions and senior managers in HE.
• Ability to assimilate a large amount of disparate
  information analyse and draw reliable
  conclusions about complex arrangements and
  undertake research and investigation into
  documentary and oral evidence in order to form
  judgements.
• Clear oral and written communication skills.
• If representing a specified academic discipline,
  to have current or recent (within the last two
  years) experience of delivering teaching,
  supporting learning and examining at specified
  levels.

12
Role of Audit (Review) Secretary

• Comment on briefing paper(s) from institution
• Collate reviewers comments
• Attend both visits, keeping notes of all
  institutional and team meetings
• Complete agreed list of key outcomes from each
  meeting
• Provides QAA with copies of notes of meetings and
  key outcomes lists
• Comment on draft report
• Comment on institutional response.

13
Qualities required in audit secretaries

• Current or recent experience (within five years)
  of administration of academic management and/or
  quality assurance at institutional level in UK
  higher education.
• Wide experience of working with senior committees
  in UK higher education.
• Ability to assimilate a large amount of disparate
  information, and to analyse and make reliable
  judgements about complex arrangements.
• Ability to keep a reliable record of discussions
  to summarise the key outcomes and to draft notes
  to a specified format to set deadlines.

14
Co-ordinating role

• In QAA this is performed by senior member of
  staff (AD), who
• Co-ordinates the review
• Conducts preliminary meeting
• Stays with team at briefing visit (chairs teams
  private meetings attends all meetings with
  institution)
• Joins team on final evening and final day of main
  visit
• Ensures that teams commentaries and judgement
  are soundly based
• Edits the report

15
Institutional Audit Training of Auditors (1)

• Purpose of training - to ensure that all
• understand the aims and objectives of the audit
  processes
• are acquainted with the procedures involved
• understand their own roles and tasks, the
  importance of team coherence, the Agency's
  expectations, and the rules of conduct governing
  the process
• have opportunities to explore and practise the
  techniques of data assimilation and analysis,
  development of programmes for visits,
  construction and testing of hypotheses, forming
  of judgements and statements of confidence, and
  preparation of reports.

16
Institutional Audit Training of Auditors (2)

• Typical activities designed to provide the
  opportunities referred to in previous slide
• 2 full days preceded by preparatory work
• Provision of further information on the aims and
  methodology of the review method (audit) and the
  outcomes, including inputs from experienced
  auditors
• Briefing on roles
• Briefing on protocols for meetings preparation,
  during, after.

17
Institutional Audit Training of Auditors (3)

• Small-group (generally two groups of typical
  audit team size) activities and related plenary
  discussions. Small-group meetings aim to develop
  teamwork in same way as actual audit, e.g.
• Identifying themes to be pursued in briefing
  visit
• Planning for the main visit
• Roles of auditors and other participants
• Managing discussion in the briefing visit
• The student experience
• Report writing (including drafting allocations)