AUQA Cycle 2 Briefing 13 December

1
Universities and Other SAIs
Cycle 2 Briefing and Consultation
13 December 2006
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
2
Briefing Outline

• Approach to Cycle 2
• Academic Outcome Standards
• Themes
• Performance Portfolio
• Steps in the Audit Process
• Audit Report Structure

2
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
3
1. Approach to Cycle 2
3
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
4
Cycle 2 Consultations

• In 2005, AUQA consulted widely on the form and
  type of cycle 2 audits. Responses were received
  from about 20 institutions and agencies, and from
  the AVCC and DEST
• Input was also obtained through discussion with
  AUQAs auditors
• Further submissions on cycle 2 were made to the
  review of AUQA, conducted earlier this year. The
  review panel included some comments on cycle 2 in
  its report (e.g. inclusion of benchmarking)

4
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
5

• Responses from the sector indicated that support
  was fairly evenly divided across three of several
  options
• Retain the whole of institution audit focus
• An increased focus on outcomes and greater use of
  internal and external reference points
• Themed audits (greater focus and depth)
• Support for reduced documentation

5
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
6
Minister Bishops Comment on Cycle 2 Outline

• it will be important that the proposal from
  AUQA reflect an approach to cycle 2 audits which
  encompasses a clear focus on both academic
  standards and external benchmarking of
  institutional outcomes.

6
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
7
Labor Party White Paper Need to address
standards

• The quality of Australian education remains
  under pressure, with risks to the reputation of
  Australian degrees there are no systems in place
  for assuring the standards of degree quality.
• We have no minimum, acceptable degree standards
  that must be met by all providers. Nor do we have
  any reliable information about the grades of
  excellence achieved by different graduates of
  different institutions in different fields of
  education.

7
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
8
Labor Party White Paper Manner of addressing
standards

• Australian Higher Education Quality Agency,
  responsible for assessing the standards of
  learning outcomes across all higher education
  providers
• Dual approach referenced to
• Different purposes of different institutions
• Minimum standards for qualifications, as in the
  AQF
• Higher education standards reviews

8
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
9
Cycle 2 Reference Group

• Establishment of Reference Group agreed by AUQA
  Board in June 2006
• The Group is intended to develop a set of
  guidelines for AUQA in addressing the issue of
  academic standards and outcomes and their
  measurement
• Phase 1 will focus on immediate requirements for
  2008 audits
• Other phases will evolve

9
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
10
Terms of Reference for Phase 1

• Develop a framework to define and identify
  academic risk that may be used to inform
  selection of themes
• Advise on how to interpret and use the academic
  risk assessment
• Develop and advise on the areas to be covered to
  investigate academic standards and outcomes
• Advise on the types of evidence auditees may be
  able to provide for the areas identified
• Advise on what outcome measures are available
• Elaborate/advise on what we might find in the
  portfolios on these areas and how the audit panel
  may interpret and investigate the evidence

10
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
11
Academic Risk

• ISO AS/NZS 43601999 Risk Management defines risk
  as The chance of something happening that will
  have an impact upon objectives. It is measured in
  terms of consequences and likelihood. We would
  therefore define academic risk as The chance
  of something happening that will have an impact
  upon academic objectives

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
12

• Examples of academic risk
• Curriculum is not up to date
• Assessment tasks do not measure relevant learning
• Inconsistent levels of quality and academic
  standards across campuses and locations
• Provision of inadequate levels of support and
  service for teachers and students offshore
• New course proposals driven by opportunity rather
  than strategic direction
• Lack of professional development for sessional
  staff
• Inadequate space for research students to permit
  them to interact academically
• Greater attention to administration and
  facilities than to teaching capacity
• Competition for students universitys
  distinctive features (brand)

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
13

• An initial assessment of areas of Academic Risk
  for an institution could for example be based on
• Cycle 1 AUQA Audit Report findings
• Institutions Progress Report to AUQA
• Current Strategic Plan
• IAF/LTPF/RQF
• Auditors-General reports
• Hot Topics

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
14
Any issues or questions?
14
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
15
2. Academic Outcome Standards
15
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
16
Standards

• A specification or other precise criteria
  designed to be used consistently as a rule,
  guideline or definition of a level of performance
  or achievement

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
17
Standards (contd)

• Help to increase reliability and effectiveness of
  application/service and its evaluation or
  measurement
• Summary of best practice
• Created through expertise experience of all
  interested parties any standard is a collective
  work

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
18
Considerations

• How well does institution perform against its own
  standards set as part of its overall purpose?
• How well does it perform against external
  expectations (regardless of what the institution
  aspires to achieve)?
• i.e. thresholds considered through, for example,
  professional accreditation processes, BUT how do
  these compare with other relevant
  national/international institutions?

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
19
Areas for Audit Focus

• How are standards in these areas determined?
  What input is there from internal/external
  stakeholders?
• How are they applied consistently across the
  institution? What processes are in place to
  assure implementation?
• How are outcomes monitored?
• How are standards and outcomes compared
  nationally? Internationally?
• What is the result of comparisons?
• How is this information used to improve
  standards? What review processes are in place to
  ensure that standards are updated?

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
20
Academic Outcome Standards Examples

• Student Assessment
• Student Progress
• Graduate Attributes
• Research

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
21
Example Graduate Attributes

• Types of evidence
• Clearly articulated and promulgated statement of
  graduate attributes, linked to Mission/goals
• Scrutiny of curriculum content for attributes in
  course/program approval and review processes
• Employer feedback on valued attributes and input
  into their development and review
• Comparative attributes from other appropriate
  national/international institutions
• Sample of assessment/examination tasks
• Graduate attributes embedded in course/subject
  guides
• External examiner reports

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
22
Example Graduate Attributes

• Possible measures of outcomes
• Generic skills scale GDS
• Graduate employment outcomes
• Completion rates
• Employer feedback
• Alumni feedback
• Graduates attaining professional registration
• Professional accreditation outcomes

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
23
External Reference Points

• National Protocols for Higher Education Approval
  Processes
• AVCC Code of Practice for Provision of Education
  to International Students
• AVCC Universities and their Students
  Principles for Education by Australian
  Universities
• AVCC Policy Guidelines on Cross-sector
  Qualifications Linkages
• MCEETYA Good Practice Principles for Credit
  Transfer and Articulation from VET to Higher
  Education
• Australian Qualifications Framework
• ESOS Act and associated Code
• UNESCO/OECD Guidelines for Quality Provision in
  Cross-Border Higher Education
• Quality Audit Factors for NSAIs

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
24
Any issues or questions?
24
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
25
3. Themes
25
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
26
Themes

• Dictionary definitions of Theme
• A subject or topic on which a person speaks,
  writes or thinks
• 2a. An idea that recurs in or pervades a work of
  art or literature
• 2b. Theme song - A recurrent melody in a
  musical play or film
• 3. Theme park - An amusement park organised
  around a unifying idea!

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
27
Themes (contd)

• For the purposes of audit cycle 2, a theme is
  defined as a distinct unifying concept that is
  significant for an institutions strategic plan
  or academic profile. A theme can be either
• an area of academic outcomes within the core
  activities of teaching and learning, research and
  scholarship, including a broad discipline area,
  or
• an enabling area closely related to the
  achievement of academic outcomes

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
28
Themes (contd)

• A theme covers a range of sub-themes. These
  sub-themes must be interrelated or closely
  related enough to allow the auditee to provide a
  meaningful self-evaluation of the theme as a
  whole. The self-evaluation must provide more than
  just a description of a particular approach and
  its deployment it must include an evaluative
  component, including evidence of outcomes and
  effectiveness

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
29
Examples of Areas of Academic Outcome Themes

• Internationalisation
• Academic QA Curriculum and Assessment
• Academic QA Admissions and Progress
• Quality of Teaching

• Learning Outcomes
• Flexible or Blended Learning
• Research
• Research Training
• Engagement

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
30
Examples of Themes (contd)

• Enablers
• Academic Governance
• Student Experience
• Human Resources Workforce Planning Renewal
• Human Resources Career Development Performance
  Mgt.

• Disciplines
• Agriculture / Environmental Sciences
• Arts / Social Sciences
• Education
• Business, Management and Commerce
• Law
• IT
• Engineering Sciences
• Health Sciences
• Physical Sciences
• Biological Sciences
• Design/Creative Arts

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
31
Information for Themes

• Relevant strategic objectives relative to the
  theme. Data presented should include
• Planned targets and outcomes, with precise
  definitions
• Related measure (s) (qualitative or quantitative)
• Performance outcomes for past three years
• Policy framework, i.e. brief summary (or URLs) of
  policies, strategies and processes in place to
  support achievement of target outcome (including
  training and development)
• Brief commentary on whether target is on track to
  be achieved or not

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
32
Information for Themes (contd)

• Main strategies and processes in place, including
  processes to achieve alignment with relevant
  external reference points
• How the effectiveness of processes is monitored
• Benchmarking results and effect
• Review and evaluation mechanisms used, including
  risk management (see academic risk), and
  outcomes of these
• As appropriate, evidence of how academic
  standards related to the theme are set, monitored
  and evaluated or, for enabling areas, evidence
  of effectiveness in supporting academic outcomes
  and standards
• Priorities for improvement

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
33
Role of Risk Assessment

• What is the role of risk assessment in relation
  to the themes?
• Are there particular indicators of academic risk
  for each of the themes?
• - Risk factors in the external environment?
• - Risk factors in the province of own control?
• - In-house process or processes for assessing
  academic risks?

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
34
Stages in the Selection of Themes

• Approx 15 months from PF submission AUQA writes
  to institution for possible themes. Six weeks for
  response
• Approx 13.5 months from PF AUQA considers
  institutional proposal in light of most recent
  IAF and Progress Report confirms meeting date
• Approx 13 months from PF AUQA AD staff visit
  institution and discuss proposal, clarify scope,
  etc
• Approx 12 months from PF Approval of two themes
  by Chair of AUQA Board (AB)
• Institution advised, notification to next AB
  meeting, publication of themes on AUQA website

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
35
Any issues or questions?
35
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
36
4. Performance Portfolio
36
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
37
Portfolio Context

• Main changes since cycle one
• Greater emphasis on linkages of processes and
  outcomes
• Use of targets and measures (quantitative and
  qualitative)
• IAF measures
• Measures relative to each institutions
  objectives
• Benchmarking and its impact

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
38
Portfolio

• Attempt to reduce amount of documentation
• Guidance on core information required within
  Portfolio (as per sector feedback)
• Greater use of tables, charts, data
• Focus on period since cycle one
• Review of evidence through electronic media/web
  wherever possible
• 15,000 word limit (AUQA will request information
  on word count in institutions covering letter)

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
39
Key Issues for Reflection

• Will the structure allow an institution to
  express the extent of its achievement in relevant
  areas and themes?
• Are the guidelines on information to be included
  clear?

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
40
Chapter 1 Outcomes since Cycle 1

• Progress on cycle one AUQA Report
• Evidence of implementation of Recommendations and
  Affirmations
• Brief comments on those not yet implemented
• Summary of significant changes to Quality
  System/processes
• Major processes

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
41
Chapter 1 (contd)

• List of external and internal reviews/audits,
  including professional/3rd party accreditation
  since cycle one visit
• Table with reference to relevant documentation
• Main benchmarking activities and outcomes since
  last AUQA audit visit, with an emphasis on impact
  of benchmarking on outcomes

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
42
Chapters 2 3 Identified Themes

• Based on self-review
• Relevant strategic objectives, including targets
• Performance data for 3 years (as available)
• Brief commentary on progress and likelihood of
  achievement of targets
• Major policies/strategies/processes in place to
  support achievement of targets

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
43
Chapters 2 and 3 (contd)

• Monitoring processes used
• Benchmarking results and impact
• Review and evaluation mechanisms
• As appropriate, evidence of how academic
  standards related to the theme are set, monitored
  and evaluated (or effectiveness in supporting
  academic outcomes)
• Priorities for improvement
• Relevant external reference points
• assessment of achievement including National
  Protocols
• Processes in place to ensure alignment

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
44
Internationalisation Theme

• TL arrangements for international students
  onshore and offshore (including partner
  arrangements if relevant)
• Internationalisation of the curriculum
• Staff and student mobility
• Overall student experience
• Other international activities, such as research
  collaboration
• If relevant, information on transnational
  activities (see example table in handouts)

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
45
Any issues or questions?
45
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
46
5. Steps in the Audit Process
46
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
47
Steps in the Audit Process

• Selecting themes
• Appointing Panel
• Portfolio Submission
• Portfolio Meeting
• Preparatory Visit
• Audit Visit
• Audit Report

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
48
Selecting Themes

• Already initiated
• Early 2007 for the first group of auditees of 2008

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
49
Appointing Panel

• February 2007 onwards
• As in cycle 1
• Typically a panel of five members
• If one of the themes is a discipline area,
• one panel member from the Auditor Register with
  an appropriate disciplinary background or
• a standing advisor to the panel with
  disciplinary expertise
• Auditor training for cycle 2

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
50
Portfolio Meeting

• January 2008 onwards
• At AUQA to be attended by the audit panel
• The panel will
• consider the portfolio in the light of the cycle
  2 focus
• identify the additional information required
  (reduced documentation)
• identify the sampling and offshore audit, that
  are relevant to the themes
• finalise the overall structure of the audit
  program
• More briefing to the panel on cycle 2

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
51
Preparatory Visit

• From February 2008 onwards
• As in cycle 1
• Chair of the Panel and Audit Director
• To discuss the additional information required
  and the audit visit preparations with the
  institution

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
52
Audit Visit

• March 2008 onwards
• 3 to 4 days would be required
• The panel will need 3 days to hear about progress
  report / contextual matters as well as the two
  themes
• AUQA will continue the half-day for the panel to
  consider conclusions before the exit meeting
• The panel will speak to a similar range of groups
• May be fewer sessions compared to cycle 1
• The panel may run group interviews with fewer
  attendees
• There would be open sessions but for matters
  relevant to the themes
• If the theme is a discipline area, the panel
  would expect to speak to (or seek written input
  from) relevant discipline experts and groups of
  the institution

A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
53
Any issues or questions?
53
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
54
6. Audit Report Structure
54
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
55
Audit Report Considerations

• Cycle 1 reports intelligible only to the
  initiated?
• Separate reports for separate audiences?
  Labour-intensive
• Cycle 2 reports
• Assist different readers to find material of most
  interest
• Provide information for general audiences early
  in the report
• Sharpen focus

55
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
56
Audit Report Structure

• Part A Conclusions
• The Auditee
• Matters from Audit cycle 1
• Theme 1
• Theme 2
• Part B Analysis and Argument
• Part C Outcomes Assessment Data
• Part D Appendices Administrative

56
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
57
Part A Conclusions

• The Auditee brief factual profile
• Emphasis on achievement of standards and
  management of quality
• Conclusions in relation to external reference
  points and benchmarking
• Commendations, affirmations and recommendations
  included within the relevant section
• Essentially a summary of findings under three
  headings

57
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
58
Part B Analysis and Argument

• The body of the report, with more detail on how
  conclusions were reached
• Intended to assist the internal audience
• Refers to Part C data as required
• Similar to cycle 1 reports

58
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
59
Parts C and D Data and Appendices

• Part C will include outcomes data relevant to the
  themes, mainly in tables
• The examples show largely internal data but there
  will also be comparative data
• Part D will contain
• Description of the audit process
• Appendices as per cycle 1 reports
• External reference points

59
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
60
Any issues or questions?
60
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
61
www.auqa.edu.au
61
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y