FROM HERE TO THERE: BUT WHERE IS HERE AND WHERE IS THERE

1
FROM HERE TO THERE BUT WHERE IS HERE ANDWHERE
IS THERE?

• Howard Davies
• Professor and Associate Dean
• The Hong Kong Polytechnic University
• Co-convenor
• Hong Kong University Grants Committee
• Task Force on Outcome-based Education

2
Howard Davies Background

• Faculty of Business at The Hong Kong PolyU
• 220 full-time staff
• Heavy research focus UTDallas 66 and rising
• 2,500 BBA, 3,000 Masters, 200 DBA and PhD
• EQUIS accredited and AACSB Accreditation Plan
  accepted
• University heavily committed to the OBA
• UGC Task Force on Outcome-based Education
• Charged with promoting OBA
• 8 universities involved
• All faculties Humanities, Science and
  Engineering
• Closely coupled with a new Quality Assurance
  Council

3
Why The Whimsical Title?

• When we talk about Outcome-based Approach we
  know that we need to move from here to there.
• But we are often not very clear where here and
  there are
• Some colleagues say we are already there and
  always have been, some say the distance to be
  travelled is huge, and maybe not worth the time
  and effort

4
Topic for Today

• Conceptual and practical issues which arise in
  respect of the Outcome- based Approach.
• Defining outcome-based education
• Issues which arise with respect to the
  specification of outcomes/learning goals
• The alignment of content and pedagogy with
  outcomes/learning goals
• Collection and reporting of evidence for the OBA
• The challenge of the OBA construed as
  organizational change. 

5
The Outcome-based Approach?

• Entails
• A focus on defining and stating what students
  will be able to do at the end of the program
  learning outcomes
• The alignment of teaching and assessment with the
  achievement of learning goals
• The collection of evidence on the extent to which
  learning goals/outcomes are achieved
• Closing the loop by systematically taking
  follow up where improvements are needed
• Say what we will do and then do what we say

6
Whats New About It?

• Weve always done that! IS IT TRUE?
• YES. We have always assessed students on the
  content of our subjects
• Can they interpret a set of accounts? They have
  an exam in Accounting 101
• Do a piece of market research? They pass Market
  Research 201
• BUT NO. (Even within subjects -sometimes we
  actually assess only a small part of a subjects
  content one essay, one presentation and 3 exam
  questions out of 6 can mean only 3 topics
  assessed)

7
Whats New About It?

• AND NO. We have not generally assessed explicitly
  for more broadly written/generic program
  outcomes.
• Students will develop Communication Skills
• They do a lot of presentations!!! But we mark
  them for subject content, we dont explicitly
  teach or assess the Communication Skills element,
  we dont explain to students what we mean by good
  communication and we dont extract evidence on
  Communication Skills per se
• Program outcomes are key, but how many of your
  faculty members are REALLY aware of the program
  outcomes/learning goals?
• Program outcomes are written by Program Leaders,
  agreed by Program Teams, discussed at validation
  meetings and then forgotten?
• Program leaders say what we will do but the
  we who do it is a different we!!!

8
Step 1 Define Program Learning Outcomes

• How many to have?
• Where do they come from?

9
How many outcomes?

• In Business 4 to 10 is said to be the norm the
  fewer the better is commonly heard
• But one reason to define outcomes is to help
  students choose a program
• A significant tension here

10
How many outcomes?

• PARSIMONY has obvious advantages and is often
  recommended
• Easier to assure
• BUT if we opt for a very small number, everyone
  will have
• Critical and Creative Thinking/ Communication/
  Ethics/ Working with People/ Global Outlook
• AND those generic goals would be shared by
  Business, Engineering, Nursing and Humanities
  programs!!

11
How many outcomes?

• If Business/Engineering/Science/ Humanities
  students are told that the goals for their
  program are entirely generic, they are likely to
  be de-motivated.
• Students come to university to study something
  they are interested in. TO FORGET CONTENT IS TO
  THROW THE BABY OUT WITH THE BATHWATER!
• Entirely generic learning goals provide no
  guidance to program content, and no guidance for
  student choice

12
How many outcomes?

• You need to find a balance between
• defining goals which really express what your
  BUSINESS/ENGINEERING/SCIENCE program is trying to
  do, signalling the specific content, AND
• Having too many goals to assure effectively
• In our case, at PolyU, we have 13 goals for our
  BBA. Are we over-burdening ourselves? We hope
  not, because 6 or 7 of them are essentially
  business content goals, addressed and assessed
  in the normal way

13
Where do the outcomes come from?

• Your Faculty or Departments Mission, of course,
  which should be consonant with the Mission of
  your University.
• So we have different levels of outcomes, so that
  consistency and coverage are difficult to
  achieve
• University-level
• Program-level
• Subject-level
• Going from program level outcomes to subject
  level is difficult enough
• Do the subject goals simply repeat the program
  outcomes?
• Are faculty members able to write subject level
  goals which are achievable and at the same time
  reflect program goals? This is a skill which many
  do not have.
• Are program outcomes separable from subject
  outcomes? E.g Is critical thinking inherent in
  learning and applying concepts or is it something
  which should be separately assessed?

14
Step 2 Alignment

• A Curriculum Map and Assessment Plan, which
  identify where each Program Outcome is addressed
  and assessed
• In subjects
• In other activities
• Assessment tasks which are suited to the outcomes

15
Step 2 Alignment

• Easy to describe, difficult to do!!
• Are faculty members actually aware of program
  outcomes and assessment plans?
• Do they actually follow through with aligned
  assessments?
• Do they remember next time round?
• Are new staff aware of what to do?
• Do different faculty members teaching the same
  subject do the same thing?

16
Step 2 Alignment

• If the program outcomes to be met by a subject
  are not clearly set out as outcomes for that
  subject, in the paperwork which everybody uses
  THERE WILL BE SIGNIFICANT DRIFT!!
• Faculty members not used to specifying realistic
  and achievable learning goals
• Faculty members not used to seeing assessment
  tasks as directed onto specific learning
  outcomes.
• What about participation grades?
• What if students have a choice of essays which
  outcomes will be addressed?
• If an exam is said to cover many outcomes there
  need to be compulsory questions addressing those
  outcomes.

17
Step 3 Collecting Evidence

• The ideal evidence for each goal would be
• Located at the end of the program, but with
  mile-post tests along the way, to assess
  progress
• Involve all students sampling can have real
  problems!!
• Involve DIRECT tests of whether the student can
  do what the outcome intends they should do
• Have an element of external validity not just
  the professors and students say all is OK
• Be inexpensive to administer
• Be non-intrusive on the program, integrated with
  the content

18
Step 3 Collecting Evidence

• The evidence we usually have to hand includes
• Exam Board results for subjects OK for Outcomes
  which are directly addressed by subjects not
  helpful for generic outcomes- no external
  validity only graduation results are at the
  right time
• Student feedback results on subjects tells us
  if the students believe they have achieved in
  subjects
• Exit Surveys at the right time, and can ask
  about program outcomes, but students perceptions
  only
• In truth, these traditional sources have limited
  validity and are often indirect

19
Step 3 Collecting Evidence

• New types of evidence we might try
• Employer/Supervisor surveys but very difficult
  to do after graduation maybe for Internships
  during the program
• Subject-embedded assessment of generic outcomes
  reported separately from subject results use
  rubrics to help faculty members?
• Additional direct tests of ability
• Language tests
• Torrance test for creativity
• Defining Issues Test -2 on ethical reasoning
• Cross-cultural Adaptability Index for Global
  Outlook but is it testing what we want?
• Collegiate Learning Assessment maybe a godsend?

20
Step 4 Reporting and Closing the Loop

• How to draw together and summarize diverse
  sources of information, whose timing may be out
  of synch with reporting cycles?
• How to be sure that actions are followed up? So
  much effort may go into the reporting process
  that by the time it is complete, it is too late
  to DO anything!
• Faculty members are accustomed to paper
  exercises, not followed up

21
Organizational Changefor the OBA

• Most attempts to introduce educational
  innovations fail
• especially when they come from the top, and for
  accountability reasons
• faculty members are capable of infinite guile and
  infinite resistance when faced with a change they
  do not like or want
• Recognition of that problem is key and points to
  the need to initiate and facilitate
  organizational change
• Seeing the OBA as akin to a change in the
  assessment regulations and paperwork is a
  guarantee of failure

22
AoL as Organizational Change

• Ironic consent is common we fill in the
  papers, tick the boxes and get on with what we
  always did.
• Many faculty members see course documents as
  something which Associate Deans and Program
  Leaders do, nothing to do with them, which has
  been (almost) sustainable in the past
• Often a tradition of telling the good news and
  avoiding blame.
• In Asia, maybe especially, say what you do and
  do what you say is not so easy to do
• IF AoL IS TO SUCCEED IN ITS OBJECTIVES CHANGES IN
  ORGANIZATIONAL CULTURE ARE PROBABLY NEEDED

23
How to bring about the change?

• Dont try to go too fast! Expect to do everything
  3 times before you get it right. PERSIST!
• Secure LEGITIMACY amongst colleagues by pointing
  to top research schools who do it.
• Try to make sure the top people understand that a
  policy paper and a set of guidelines does not
  mean that anything at all has been DONE.
• Try to help the front-line teachers with the
  paperwork aspects and make sure they know they
  need to implement. Rubrics can help those who are
  nervous about grading generic goals
• Remember that the best is often the enemy of the
  good dont strive for perfection.
• Generate and maintain conversations within and
  across schools

24

• Good Luck and Thank You !