Preparing for the QAC Audit

About This Presentation

Title:

Preparing for the QAC Audit

Description:

The following list illustrates the kind of questions which might be asked in ... The show was cancelled due to the strike', etc. ... – PowerPoint PPT presentation

Number of Views:144

Avg rating:3.0/5.0

Slides: 52

Provided by: lc195

Category:

more less

Transcript and Presenter's Notes

Title: Preparing for the QAC Audit

1
Preparing for the QAC Audit
2
Goals of UGCs previous TLQPR

To focus attention on teaching and learning as
the primary mission of Hong Kongs tertiary
institutions.
To assist institutions in their efforts to
improve the quality of teaching and learning.
To enable the UGC and the institutions to
discharge their obligation to maintain
accountability for the quality of teaching and
learning.

3
From TLQPR report on HKUST 2003

3.1 (iii) While the language of educational
quality processes has been taken on board at all
levels of the institution, and is generally taken
seriously, the processes and procedures signified
by that language are not yet embedded so as to
constitute a strong culture of search for
continuous improvement. The often conflicting
institutional priorities between research and
teaching need to be addressed, so that
discussion, evaluation and innovation regarding
teaching become as much a part of the culture of
HKUST as does research activity.

3.2.3 There is a need for tighter and more
obvious links between course / programme goals,
explicit student learning outcomes that derive
from these, and the design and implementation of
teaching and assessment activities that
rationally aim to achieve those learning
outcomes.

4.2 The particular methods emphasised in the
various units tended to reflect traditional
approaches within the particular discipline, and
overall the adjective traditional (and
generally competent) could validly be used to
characterise the general approach to teaching
across the institution as a whole.

6
Quality Assurance Council (QAC) Audit of HKUST

Date February 2010
Audit Manual http//www.ugc.edu.hk/eng/doc/qac/pu
blication/auditmanual.pdf

7
Scope of QAC Audit

The QACs audit activities cover all first degree
programmes and above, however funded, offered by
UGC-funded institutions (including their
continuing education arms and community
colleges).
This includes self-financing programmes, joint
programmes leading to a Hong Kong award, and the
teaching and learning aspects of research degree
programmes.

8
3.1 QACS approach to audit

The main objective of QAC quality audits is to
assure the quality of student learning in
UGC-funded institutions.
The audit examines whether the institution has
procedures in place appropriate for its stated
purposes, whether it pursues activities and
applies resources to achieve those purposes, and
whether there is verifiable evidence to show that
the purposes are being achieved.

A QAC audit is not a review against a predefined
set of standards. It does, however, oblige
institutions to articulate and justify the
standards they set for themselves, and
demonstrate how the standards are achieved.
The quality of student learning is the
centrepiece of audit. The QAC audits research and
managerial activities only in so far as they
affect the quality of teaching and learning.

10
Some characteristics of the audit model

(a) The quality of student learning is the
centrepiece.
(b) There is an emphasis on quality enhancement.
(c) An audit is viewed as a collaboration between
the QAC and the institution.
(e) The audit process involves institutional
self-evaluation, followed by peer review which
makes evidence-based findings.
(f) No attempts to make comparisons among
institutions.

11
Outcome-based approach

The audit process is particularly concerned with
the ways institutions articulate and measure the
student learning outcomes they expect or aspire
to.
The UGC encourages institutions to adopt an
outcome-based approach to student learning, and
the QAC audit process reinforces this objective.

Institutions, at the time of their first audit
between 2008 and 2011, will be at different
stages in their preparations for the four year
undergraduate degree and in their development of
an outcome-based approach to student learning.
This will be fully recognised by audit panels.
There will be no expectation or judgement of an
institutions progress in taking forward these
initiatives as part of the audit process.
1.2

13
3.2 Audit methodology

Approach-Deployment-Results-Improvement (ADRI
methodology)
1. (Approach) What is the institutions purpose?
2. (Deployment) How does the institution achieve
its purpose?
3. (Results) What evidence does the institution
have that its purpose is being achieved?
4. (Improvement) What processes are in place for
improvement?

14
1. What objectives is the institution trying to
achieve and on what basis does it set the
objectives?

The audit checks that the institution articulates
overall learning objectives which are consistent
with its mission and with the role statement
agreed with the UGC. The audit examines how these
overall objectives are set and justified, and the
extent to which they take cognizance of relevant
external reference points
The audit further examines how the overall
objectives are communicated throughout the
institution, and the extent to which they are
reflected in the desired learning outcomes of
individual programmes. The audit also examines
how objectives are set for each activity which
contributes to student learning

15
2. How does the institution go about achieving
its objectives?

The means of achieving the desired objectives
should be expressed in the institutions plans,
policies and procedures, and its activities and
deployment of resources should be geared towards
achieving its objectives.
The audit also checks that policies and
procedures are implemented widely and
consistently throughout the institution.

16
3. How does the institution measure achievement
of its objectives and what evidence is there that
the objectives are being achieved?

The audit examines the adequacy of the measures
(e.g. benchmarks, students achievements,
performance indicators) used by the institution
to demonstrate that it achieves its objectives in
each of the focus areas.
Some objectives may be directly measurable (e.g.
graduation rates) while others can only be
inferred from proxy evidence (e.g. certain
graduate attributes being inferred from employer
satisfaction).
The audit examines the institutions performance
against its own measures, and checks that the
institution systematically collects and analyses
evidence of its performance.

17
4. What processes are in place for improvement?

The audit examines whether the institution
analyses its performance in order to effect
improvement.
The audit checks that there are robust feedback
loops from evidence of performance back to
planning and implementation.

18
3.3 Standards and external reference points

there is public expectation that the standards
are in some sense appropriate and comparable with
standards elsewhere.
in setting its academic standards an
institution should take account of external
reference points such as public and employer
expectations, the requirements of professions,
and the standards of other institutions in Hong
Kong and elsewhere.

a QAC audit examines the adequacy of the
measures used by an institution to demonstrate
that it achieves its objectives.
An important element of this examination is the
extent to which the measures allow the
institution to place its achievement in a broader
context, through the use of benchmarks and other
comparative data.

20
3.4 The scope of audit Focus Areas

Since student learning is the focal point of the
audit system, audits examine all aspects of an
institutions activities which contribute to
student learning quality.
These activities range from planning and policy
development, through programme design, approval
and review, to teaching, assessment and student
support.

21
FOCUS AREAS REVIEW QUESTIONS

(from QAC Audit Manual, Annex A.3)
The following list illustrates the kind of
questions which might be asked in each of the
audit focus areas. The list is indicative rather
than prescriptive or exhaustive.

22
1. Articulation of appropriate objectives

o What overall objectives is the institution
trying to achieve?
o Are these appropriate to its agreed role?
o What student learning outcomes are expected,
and how are they measured? What are the
performance indicators, benchmarks or other
reference points?
o Are the desired objectives communicated and
understood throughout the institution?
o Do individual programmes specify desired
learning outcomes (e.g. graduate attributes)?
o Are programme objectives consistent with
institutional objectives?

23
2. Management, planning and accountability

o What is the organisational framework within
which teaching and learning is managed? Is it
effective? How does the institution know?
o What are the responsibilities and
accountabilities for delivering outcomes?
o How does the institution measure performance
against plan?
o What are the communication and reporting
mechanisms? Are they effective?
o How do students participate in planning and
management for learning?

24
3. Programme development and approval processes

o What are the criteria for introducing new
programmes? Are they consistent with the
institution s role?
o How are the rationale and expected outcomes for
new programmes articulated? Who approves them?
o What external input is there to programme
planning and development, including curriculum?
o How are planned learning experiences, including
out of class experiences, related to desired
learning outcomes?
o What processes are in place to check that
resources (e.g. staff, library, IT, laboratories)
are available, and that the delivery mode and
assessment are appropriate to the desired
outcomes?
o Is there adequate internal consultation (e.g.
to prevent duplication)?
o What processes govern the programme change
processes?
o How are programmes made up of courses, to
produce a coherent, integrated whole?

25
4. Programme monitoring and review

o How does the institution know that programmes
are delivering the desired student learning
outcomes? Does it use external references or
benchmarks?
o What are the frequency, scope and process of
programme review? What are the criteria for
programme continuation?
o What data is used in monitoring and review
(e.g. student demand and entry qualifications,
student feedback, employer feedback, employment
data, graduate destinations)?
o What external input is there?
o How does the institution make improvements to
the programme?

26
5. Curriculum design

o How does curriculum design relate to desired
student learning outcomes?
o What external input or reference points are
used?
o How are curricula systematically updated?

27
6. Programme delivery, including resources,
teaching mode, and the student learning
environment

o Are sufficient resources (e.g. staff, library,
IT, laboratories) applied to the achievement of
programme outcomes?
o Are the learning environment and the modes of
delivery geared to desired learning outcomes?
o How do student support services measure their
effectiveness? How do they ensure that their
activities are aligned to institutional goals?
How do they obtain and use student feedback?

28
7. Experiential and other out of classroom
learning

(e.g. leadership development, overseas exchange,
work-integrated learning, service learning)
o How does this form of learning contribute to
desired student learning outcomes?
o How is the learning experience designed? What
are the inputs?
o Are appropriate resources applied to delivering
and supporting the experience?
o How are the effects of the learning experience
evaluated?
o What processes lead to improvement?
o How is out of classroom learning integrated
with curricular learning? What evidence is there
of effective integration?

29
8. Assessment

o Does the institution have an assessment policy
which addresses issues such as equity, accuracy
and student workload? Is it widely implemented
and effective?
o Is assessment designed to evaluate and enhance
students achievement of desired learning
outcomes?
o How is out of classroom learning assessed?
o What processes are in place to detect and
handle plagiarism or fraud? Are they effective?
o What external checks or reference points are
used? Are they effective?

30
Example of traditional course objectives

Course Title Introduction to the Study of
Language (UG)
Objectives
Upon completion of this subject, students should
be able to
Be aware of both the diversity of language
systems and their fundamental similarities
Have a general idea of the main subfields of
language study morphology, syntax, phonetics,
phonology, semantics, pragmatics,
sociolinguistics, language acquisition
Have the basic concepts to study other specific
linguistic courses required by the program.

31
Example of traditional exam

Course Title Introduction to the Study of
Language (UG)
I. True/False (5 marks)
a) Lexical ambiguity refers to.
II. Multiple choice (10)
a) What is the study of the origin of words
called?...
III. Short questions (15)
a) What is the difference between pidgin and
creole.
IV. Label the diagram (20)
V. Essay question (25 25)
a) Discuss the connections between a peoples
language and their culture.

32
Assessment in OBE

In OBE, Assessment -- both continuous
(assignments, tests) and final (exams, term
papers) -- should directly measure the students
achievement of the Intended Learning Outcomes
The ILOs can serve as a kind of blueprint for
assessment questions and tasks.

33
Ex.1 Grammar of Modern English (PG)

Intended Learning Outcomes
Upon completing the course, the students should
be able to..
3. Demonstrate a proper understanding of the
nature of grammar and grammatical rules (and
how they differ from traditional prescriptive
rules), and the differences between written and
spoken grammar
4. Use an online databank to search for evidence
for the current state of the English language,
and draw well-supported generalizations and
conclusions concerning particular grammatical
structures.

34
Sample Assignment Assessment

Most popular books on English grammar and usage
(e.g. Fowlers famous Modern English Usage) adopt
an openly prescriptive approach. Now that we
have databanks at our disposal, we can find out
for ourselves how present-day users of English
actually use English, and how it differs from the
rules laid down by prescriptive grammarians.
Given below are three examples of such
prescriptive rules. Do a search on the Collins
WordbanksOnline for relevant data on each of the
three forms, and (i) decide whether there is any
empirical basis for the prescriptive rule, and
(ii) write an objective, accurate description of
how each form is actually used in English today.
Be sure to support your descriptions with data
extracted from the databank.

Traditional prescriptive rules
Example 1 DUE TO
Due to is wrongly used as an adverbial of reason
modifying a clause, in sentences like
Due to his age, he was unable to compete
The show was cancelled due to the strike, etc.
In such sentences, owing to should be used
instead
Owing to his age, he was unable to compete
The show was cancelled owing to the strike,
etc.
The proper use of due to is with a noun phrase
(NP1 is due to NP2), as in
His poor health is due to heavy smoking
The cancellation was due to the strike, etc.

36
Data on due to from Collins WordbanksOnline

137.95 yen. The dollar gained ground due to
increasing concern about the
officials say the heavy losses are due to the
extremely dangerous missions
of outside subcontractors. Another is due to
company pressure to speed up
impact outside the city, largely due to
television. Limited rioting
a very frustrating thing for all of us due to the
fact that you can't really
criticized by social workers are due to
overcrowding. Throughout the
itself. ltpgt The only worthwhile act, due to its
novelty value, was the group
mothers have had extra expenses due to the
General Election being held
Health WorksToday Offer lt/hgt ltpgt Due to the
overwhelming response to last
14-year-old and it is largely due to her
attitude to life. ltpgt I just
seen a thousand times before - largely due to the
iridescent cinematography of
cent less likely to develop problems due to the
fatty acids in fish oil. ltpgt
make up to pound 1,800 - partly due to
that Rolls. A nice car will
s professional future seems in doubt due to a
knee injury. Former Wimbledon
Goodwood defeat can be ignored due to the
false pace. ltpgt Doubts
foreign exchange) facilities available due to a)
adverse publicity b) previous
but perhaps more thanks were due to
Western Samoa for gifting such
wackiness. Hi there," says Phil. Due to
immense peer pressure I have
costs for the Japanese cars, this is due to the
fact that their parts prices

37
Ex.2 English as a World Language (PG)

Intended Learning Outcomes
By the end of the course, the students should be
able to
4. Understand and apply the conceptual and
methodological tools of World Englishes to the
analysis of the linguistic features of New
Englishes, as well as their social, cultural,
educational and political roles in their
respective societies.
5. Analyse and compare the linguistic and
socio-cultural features of some well-known modern
varieties of English, including British,
American, Australian, Indian, Singapore, Hong
Kong and China English
6. Discuss the linguistic, social, political,
cultural and educational issues arising from the
emergence of English as a world language, in a
logical and informed manner.

38
Term paper

1. In this course, we have studied some new
varieties of English (such as Singapore English
and Indian English), and analysed their
linguistic and socio-cultural features. Do you
think it is justifiable to claim that there
exists such a thing as Hong Kong English? In
other words, can the type of English spoken by
Hong Kong people be considered a variety of
English in its own right -- in the same sense
that Singapore English and Indian English are
varieties of English?
In trying to answer this question, you will first
need to explain the full meaning and implications
of the concept of a variety of English, and see
if Hong Kong English lives up to these
implications and expectations. You will also
need to be acquainted with some other
well-established varieties of English (like
Singapore and Indian English) in order to make
meaningful comparisons. Among the readings which
are relevant to this topic are. Do not merely
regurgitate materials from these readings, and be
aware that some of the views put forward by their
authors may be controversial (e.g. Boltons) and
require careful thought. Try to formulate your
own judgements and arguments on this issue.

39
9. Teaching quality and staff development

o How is teaching quality assessed?
o How is student feedback obtained and used?
o Are inexperienced staff offered an introduction
to teaching? Is it mandatory? Is it effective?
o What other teaching development activities or
opportunities are offered? How are they supported
by the institution? Are they effective? What is
the take-up rate?
o How is teaching informed by staff s research or
professional activities?
o What support is available to staff for the
effective delivery and assessment of in-class and
out of class learning?
o How is the teaching performance of individual
staff monitored and enhanced? How does teaching
performance affect promotion?

40
10. Student participation

o What opportunities are there for students to
participate in programme planning, monitoring and
review? Do relevant organisational structures
(e.g. department/ faculty committees) have
student members?
o Are there processes for systematically
obtaining and using student feedback on courses,
programmes and teaching?
o Are there processes for systematically
obtaining and using student feedback on support
services?

41
11. Activities specific to research degrees

o What learning outcomes does the institution
expect from research degree programmes? Are there
any external reference points?
o What criteria are used for admission to
research degree programmes? How are they
established and are they appropriate?
o What resources are provided to research
students? How does the institution know these are
sufficient?
o How does the institution appoint supervisors
and monitor the quality of supervision?
o How is the progress of research students
monitored? How are problems identified and
handled?
o What are the examination processes? What is the
external involvement?
o How does the institution foster and monitor an
appropriate intellectual climate for research
students?

42
3.5 Sample programme review

As part of a QAC audit, the panel examines in
depth a sample of programmes to seek evidence
that the institutions quality assurance
processes are in fact being implemented in
practice and that stated outcomes are being
achieved
In a sense, each selected programme is treated as
a test case of the institutions performance in
the focus areas
The number of programmes selected (normally no
more than three) is determined by the audit panel.

43
Selection of sample programmes

(Section 6.1) The panel selects programmes for
sample review (9 weeks before the Audit), using
criteria including the following
o relevance the extent to which a programme
may shed light on any major audit issues the
panel has identified
o distinctiveness the extent to which a
programme breaks new ground or employs innovative
modes of delivery
o diversity selection of programmes in diverse
areas (e.g. engineering and humanities) or of
diverse types (e.g. self-financing and publicly
funded) to provide distinctive samples to examine
o significance the size of a programme in
terms of staff, students or resources, or its
strategic importance to the institution s future
o recent review to examine whether a programme
which has been recently reviewed (by the
institution or by an external body) has adopted
any changes recommended
o coverage the extent to which the QAC wishes
to enhance its knowledge of a wide variety of
programme types

44
3.6 Institutional self-review

Audits are preceded by a period of institutional
self-review culminating in an Institutional
Submission to the audit panel.
A major output of the self-review is the
Institutional Submission which serves as a
principal source of information in the audit
The draft is submitted for feedback about 6
months before the audit visit, and the final
Institutional Submission is submitted about 3
months before the audit visit.

45
5.4 Self Review

As part of its preparation for audit, the
institution is required to review all of its
operations which contribute to the quality of
student learning.
This self-review is an opportunity for the
institution to analyse and reflect on its
objectives for student learning, on its
procedures, standards and benchmarks, and to
check that it has appropriate outcome measures in
place.

46
7. THE AUDIT VISIT 7.1 Overview

The purpose of the Audit Visit is to allow the
panel to test the material presented by the
institution through first-hand investigation and
personal interactions.
The visit allows the panel to clarify and
interpret the material it has been given, to
examine evidence, and through meetings with
staff, students and other stakeholders, to verify
that policies and procedures are carried out in
practice.

47
QACs activities during visit

interviews staff, students and other stakeholders
peruses documents requested by the panel
progressively reflects on and discusses the
written and verbal material so far presented
progressively refines findings and draft
recommendations
gives a brief oral presentation of its findings
during an exit meeting with the institutions
leaders
The audit visit lasts for 3 to 5 days

48
Appendices (i) HKUSTs Institutional Objectives

To give all students, undergraduate and
postgraduate alike, a broadly based university
experience that includes superior training in
their chosen fields of study a well-rounded
education that enhances the development of their
creativity, critical thinking, global outlook,
and cultural awareness a campus life that
prepares them to be community leaders and
lifelong learners.

49
(ii) ABC LIVE HKUSTs Graduate Attributes

Academic Excellence
An in-depth grasp of at least one area of
specialist or professional study, based on a
forward-looking and inquiry-driven curriculum.
Broad-based education
Intellectual breadth, flexibility, and curiosity,
including an understanding of the role of
rational, balanced inquiry and discussion, and a
grasp of basic values across the core disciplines
of science, social science, engineering and the
humanities.
Competencies and capacity building
High-end, transferable competence, including
analytical, critical, quantitative and
communications skills.

Leadership and teamwork
A capacity for leadership and teamwork, including
the ability to motivate others, to be responsible
and reliable, and to give and take direction and
constructive criticism.
International outlook
An international outlook, and an appreciation of
cultural diversity.
Vision and an operation to the future
Adaptability and flexibility, a passion for
learning, and the ability to develop clear,
forward-looking goals, and self-direction and
self-discipline
Ethical standards and compassion
Respect for others and high standards of personal
integrity
Compassion, and a readiness to contribute to the
community