Jean Hughes

1

• Jean Hughes
• Office of the Vice President for Learning
  Innovation,
• DCU

2
Overview

• DCU Approach
• Framework Implementation Network
• Learning Outcomes
• General Assessment of Learning Outcomes
• Common Questions

3
DCU

• Already Modularised and Semesterised (1997)
• Module Descriptors use Learning Outcomes
• European Credit Transfer System (ECTS) well
  established
• But
• Mainly administrative exercise
• Academic Framework for Innovation (AFI) for
  Deep Curriculum Reform (3 year project)
• Bologna Compliance
• Demonstrable Alignment with NFQ
• Use of Learning Outcomes for real change in
  assessment practice
• Flexibility Designed in and through new Marks
  Standards

4
Framework Implementation Network

• Created by the IUA and the NQAI
• Membership
• 7 Universities
• NUI Colleges
• Purpose to collectively address outstanding
  issues relating to Bologna and the National
  Framework of Qualifications
• Three separate Working Groups
• Award Titling
• Disciplinary Learning Outcomes
• Assessment of Learning Outcomes

5
Key Issues

• Ensuring that Assessment, Teaching and Learning
  are aligned
• Moving from assessing content to assessing
  outcomes
• Ensuring all outcomes are assessed and all
  assessment instruments actually assess outcomes
• Avoiding over-assessment

6
Learning Outcomes

• Clearly identify what a learner can demonstrate
  as a result of successfully completing a part of
  a learning programme
• They describe student attainment which is
  demonstrable and assessable.

7
Benefits of Learning Outcomes

• Guide students in their learning - what is
  expected of them - in turn helping them to
  succeed in their studies.
• Help lecturers to focus on exactly what they want
  students to achieve in terms of both knowledge
  and skills.
• Provide a useful guide to inform employers about
  the general knowledge and understanding that a
  graduate will possess.

8
Challenges

• Traditional view
• inputs quality of learning outcomes
• Moving from content (Teaching) focus to Outcome
  (Learning) focus
• Aligning teaching, learning and assessment

9
Award and Module Outcomes

• NFQ describes generic, high level outcomes for
  awards at each level under 3 strands and 8
  substrands
• How do we demonstrate achievement of award
  outcomes?
• We dont assess at award level we assess at
  module level
• Need to ensure that module learning outcomes
  collectively deliver the award outcomes

10
Alignment of Programme and Module Outcomes
PO Programme Outcome Mod - Module
11
Assessing Learning Outcomes
12
Constructive Alignment
is the underpinning concept behind the current
requirements for programme specification,
declarations of intended learning outcomes and
assessment criteria, and the use of criterion
based assessment.
(Biggs, 1999)
13
Example Project Management
14
Student Perspective

• From our students point of view, assessment
  always defines the actual curriculum (Ramsden,
  1992)
• Biggs (2003)
• Teacher Perspectives
• Objectives Learning Outcomes
  Teaching Activities Assessment
• Assessment Learning Activities
  Outcomes
• Student Perspectives

Alignment
If curriculum is reflected in the assessment, the
teaching activities and the learner activities
are directed towards the same goal in preparing
for the assessment students will be learning the
curriculum
15
Assessment Impact(Boud, 1995)

• There is probably more bad practice and
  ignorance of significant issues in the area of
  assessment than in any other aspect of higher
  education.
• The effects of bad practice are far more potent
  than they are for any aspect of teaching.
  Students can, with difficulty, escape from the
  effects of poor teaching, they cannot (by
  definition if they want to graduate) escape the
  effects of poor assessment.

16
Assessment - Terminology

• Learning Outcomes
• What a student should be able to demonstrate
• (May imply the assessment criteria)
• Assessment Criteria
• The basis on which a judgement of the adequacy of
  the student work is made
• Assessment Methods
• The task(s) undertaken by the student writing
  an essay, answering an exam question etc- that is
  subject to assessment.
• Many assessment methods may be appropriate to the
  same outcomes e.g. Written, Oral, Visual,
  Project etc
• Must be valid and reliable Fit for purpose

17
Over-Assessment

• Typical with semesterisation and modularisation
• High risk with learning outcomes
• Tend to be quite granular and have several
  independent outcomes
• To avoid
• Try combining the assessment of more than one LO
• LOs only need to be assessed once
• Use assessments which inform/follow on from each
  other
• Check what you are assessing

18
Checking Assessment
?
?
?
?
?
?
?
?
Issues
What does assignment 3 assess?
How is Learning Outcome 2 assessed?
Are Learning Outcomes 1 and 4 over-assessed?
19
Refining Assessment
Perhaps design a new assessment 3 to re-balance?
?
?
?
?
?
?
?
?
?
?
20
Common Questions
21
Must all outcomes be assessed?

• Yes, but
• Not independently
• Look at how you write the learning outcomes
• Students are expected to be able to
• Write a business report
• Present orally
• Design a visual presentation
• versus
• Students are expected to be able to
• Communicate effectively using a combination of
  written reports, oral presentations and visual
  media
• This reduces the number of independent outcomes
  and also recognises the combination of
  communication mechanisms
• BUT
• For certain disciplines, for example languages,
  you may need these to be independently
  demonstrated

22
Example

• In a Business Module (1 outcome)
• Students are expected to be able to communicate
  effectively in a business context using
  appropriate communication tools including
  Written, Oral and Visual
• The outcome is communication, the tools are some
  means of demonstrating that outcome
• In a languages Module (4 outcomes)
• Students are expected to be able to demonstrate
  written proficiency in French
• Students are expected to be able to translate a
  piece of French text
• Students are expected to be able to comprehend a
  piece of French text
• Students are expected to be able to deliver an
  oral presentation in French

23
Must all outcomes be passed?

• In theory, yes, but
• Threshold Achievement
• Implies that ALL learning outcomes must be passed
  in order to pass a module
• Very strict interpretation of LO approach
• Leads to reductionist or Tick Box approaches
• Typical Achievement
• Learning outcomes are written (and assessed) with
  the typical or average student in mind
• Grading criteria will allocate marks to degree of
  achievement

24
But, we dont currently assess everything.

• Exams often require only 3 out of 5 questions
• Therefore we dont assess everything
• But
• Generally this refers to content
• Outcomes should be about demonstrating
  knowledge/understanding aided by knowledge of
  content, not content in its own right
• Assessment criteria and methods should satisfy
  you that students can demonstrate achievement of
  the outcomes
• Worst-case scenario no worse than current
  situation!

25
What if some outcomes are failed?

• Depends on the criticality of the outcome
• If particular outcomes must be passed then
  passing other outcomes should not be able to
  compensate
• Need to look at Threshold and Typical
  Outcomes
• Need to write outcomes carefully
• Need to ensure that supplemental/repeat
  assessment is also valid

26
What about repeats?

• Did they actually fail?
• Students will present orally on 20th century
  politics
• If they dont attend they fail.. Is this what
  you meant? or
• Students will effectively communicate on 21th
  century politics
• You may prioritise Oral assessment and allocate
  more marks, but this allows more choice of
  methods
• If a student has to repeat the supplemental
  assessment must be valid if different from the
  original
• Look at how you write the outcomes

27
Is it an Outcome, An Assessment or an Efficiency?

• Often mix up pedagogical (or efficiency)
  approaches with learning outcomes
• Example Group Working
• Learning Outcome
• Students will demonstrate effective group
  working skills
• Group working needs to be taught and assessed
• Cant assume that putting people in groups will
  automatically lead to effective group working!
• How does an individual student repeat?
• Students will be able to evaluate group dynamics
  and effective group working
• Would this suffice but give an alternative
  assessment approach for repeat students?
• Pedagogical Approach
• Group working is used to facilitate learning of a
  particular outcome
• Not assessing the group working assessing the
  actual outcome
• Efficiency
• Often used to reduce the volume of marking
• Need to be careful that it is pedagogically sound
• Not assessing group working

28
Making the Tacit Explicit

• Some disciplines rely on expertise of the
  individual lecturer
• I know a good piece of work when I see it
• You cant possibly express Creativity,
  Performance, Medieval English.. Using learning
  outcomes
• NOT about subjectivity expertise, tacit
  knowledge etc
• How do students know what to aim for?
• How do novice lecturers know about standards?
• How can we demonstrate consistency, transparency
  etc?

29
Example

• First, rewrite the outcome
• By the end of this module students will
  understand the importance of post-renaissance
  European art in the context of art history
• Redesigned
• By the end of this module students will be able
  to
• Evaluate and criticise post-renaissance European
  art within its historical context
• Date and identify key works of this specific
  period
• Demonstrate an understanding of the particular
  art-historical debates that have taken place
  about this period
• Show ability to argue why this particular place
  and time raise interesting issues about the
  discipline as a whole
• University of Sussex

30
Then, use a rubric

• Also called grading criteria or scoring guide
• A set of guidelines for marking or scoring
• Can be a list, chart, or guidelines
• They state all dimensions being assessed
• They contain a scale
• They assist the marker on rating assessments on
  the scale

31
Rubrics - 3 Types

• Checklists
• Indicating list of criteria
• Rating Scales
• Checklist with rating scale added
• Show degrees by which students meet criteria
• Holistic Scoring Guide
• Used where assignments can vary i.e. projects
• Narrative descriptions of characteristics of
  different grades of work
• Not list of criteria

32
Rubric Design
Achievement Levels
Criteria
Excellent
Good
Needs Work
Not acceptable
Objective 1
Accepted Minor Major
Rejected revision revision
Objective 2
Expert Advanced Intermediate Novice
Objective 3
6-5 4-3 2-1
0
33
Simple Checklist Rubric
34
Rubric is for a written Business Assignment
35
This assignment asked students to take a position
on a debateable issue regarding interpretation of
literature they had studied. The lecturer has
identified 3 major criteria Position, Support
and Acknowledgement of Alternative Points of View
and has weighted them different levels of
attainment of each criterion. Opposite is the
detailed rubric for the Position criterion.
36
In Summary

• Need to write learning outcomes so that they can
  be demonstrated and assessed
• Need to identify assessment criteria and
  appropriate assessment methods
• Need to be sure exactly what it is students need
  to demonstrate
• Making the tacit explicit Clearly written
  outcomes and the use of rubrics to articulate
  achievement

37
Questions?