Linking assessment to learning outcomes

1
Linking assessment to learning outcomes

• Cynthia Deane
• Galway-Mayo Institute of Technology
• 28 April 2008

2
Assessment

• A way of finding out what our students know and
  can do (UCC Portfolio, 2002)
• Assessment is the process of gathering and
  discussing information from multiple and diverse
  sources in order to develop a deep understanding
  of what students know, understand and can do with
  their knowledge as a result of their educational
  experiences
• (Huba and Freed, 2000)

3
Before we start

• What are your concerns about assessing learning
  outcomes?
• What challenges do you expect when you make
  changes in the assessment process?
• What do you want to achieve today?
• My task to help you come up with a good
  assessment plan for your courses.

4
Some questions for today

• Why is assessment such a big issue in higher
  education at the moment?
• How best can balance assessment FOR learning with
  assessment OF learning (formative and summative
  purposes)
• How do we make sure assessment is fit for
  purpose?
• What assessment techniques can we use to measure
  different types of learning outcomes?

5
More questions

• Where can we start if we want to improve the
  alignment between assessment and learning
  outcomes?
• How can we improve exams so that they test higher
  order skills?
• How can we make assessment criteria more
  explicit?

6
Later today

• Interrogating your assessment to check whether
  it follows good practice guidelines
• Identifying any changes needed to assessment
• Practice in linking assessment with learning
  outcomes
• Anything else to include?

7
Trends in assessment

• Traditional
• Examinations
• Lecturer-led
• Implicit criteria
• Product assessment
• Content
• Decontextualised
• Convergent
• Individual

• Changing approaches
• Course work
• Student-led
• Explicit criteria
• Process assessment
• Skills
• Contextualised
• Divergent
• Group
• Brown and Biggs

8
Assessment IS a big issue

• Influences
• Education theory
• National, international, Bologna
• Quality assurance, TL strategy
• Professional bodies
• Validation HETAC
• Diverse profiles

• Context
• Curriculum philosophy
• Qualifications frameworks
• Institutional policy/practice
• Subject/discipline
• Course/module
• Learners

9
Purposes of assessment

• Educational feedback, diagnosis, motivation,
  guidance, learning support
• Assessment FOR learning (formative)
• Managerial selection, grading, certification,
  progression, professional recognition,
  gate-keeping
• Assessment OF learning (summative)

10
Re-positioning assessment
11
Assessment principles summary

• Learner centred inclusive, diversity
• Linked to learning outcomes
• Performance of understanding
• Process matches purpose
• Range of modes, techniques, formats
• Transparent, fair and equitable to all users
• Valid, authentic and reliable

12
16 INDICATORS OF EFFECTIVE ASSESSMENT IN HIGHER
EDUCATIONA checklist for quality in student
assessment

• 1. Assessment is treated by staff and students as
  an integral component of the entire teaching and
  learning process.
• 2. The multiple roles of assessment are
  recognised. The powerful motivating effect of
  assessment requirements on students is understood
  and assessment tasks are designed to foster
  valued study habits.
• 3. There is a faculty/departmental policy that
  guides assessment practices. Subject assessment
  is integrated into an overall plan for course
  assessment.
• (Source Centre for the Study of Higher
  Education, Australia)

13
16 INDICATORS OF EFFECTIVE ASSESSMENT IN HIGHER
EDUCATIONA checklist for quality in student
assessment NB!!

• 4. There is a clear alignment between expected
  learning outcomes, what is taught and learnt, and
  the knowledge and skills assessed.
• 5. Assessment tasks assess the capacity to
  analyse and synthesise new information and
  concepts rather than simply recall information
  which has been presented.
• 6. A variety of assessment methods is employed so
  that the limitations of particular methods are
  minimised.
• 7. Assessment tasks are designed to assess
  relevant generic skills as well as
  subject-specific knowledge and skills.

14
16 INDICATORS OF EFFECTIVE ASSESSMENT IN HIGHER
EDUCATIONA checklist for quality in student
assessment

• 8. There is a steady progression in the
  complexity and demands of assessment requirements
  in the later years of courses.
• 9. There is provision for student choice and
  weighting in assessment tasks.
• 10. Student and staff workloads are considered in
  the scheduling and design of assessment tasks.
• 11. Excessive assessment is avoided. Assessment
  tasks are designed to sample student learning.

15
16 INDICATORS OF EFFECTIVE ASSESSMENT IN HIGHER
EDUCATIONA checklist for quality in student
assessment

• 12. Assessment tasks are weighted to balance the
  developmental (formative) and judgemental
  (summative) roles of assessment. Early
  low-stakes, low-weight assessment is used to
  provide students with feedback.

16
16 INDICATORS OF EFFECTIVE ASSESSMENT IN HIGHER
EDUCATIONA checklist for quality in student
assessment

• 13. Grades are calculated and reported on the
  basis of clearly articulated learning outcomes
  and criteria for levels of achievement.
• 14. Students receive explanatory and diagnostic
  feedback as well as grades.
• 15. Assessment tasks are checked to ensure there
  are no inherent biases that may disadvantage
  particular student groups.
• 16. Plagiarism is minimised through careful task
  design, explicit education and appropriate
  monitoring of academic honesty.

17
Assessing learning outcomes points to consider

• Learning outcomes statements of what a learner
  will know, understand, and/or be able to do at
  the end of a learning experience
• Specify the types of student performance that
  will provide evidence of learning
• (Assumption you have designed your courses using
  learning outcomes. You now want to find out
  whether students have achieved them. )

18
Backward design

• Identify desired learning outcomes
• What should learners know and be able to do?
• Determine acceptable evidence
• How will we know they have achieved desired
  outcomes? What measurement?
• Plan learning experiences and instruction
• What methods, materials, resources?

19
Assessment choices

• How best to measure the wide range of learning
  outcomes? Types of test items to include?
• Written, oral, practical, other techniques?
• Balance between formative and summative purposes?
• Continuous and/or terminal?
• How to manage large groups?
• Disciplinespecific approaches?

20
Jargon Modes of assessment

• Formal/informal
• Formative/summative
• Continuous/terminal
• Coursework/examination
• Process/product
• Criterion/norm referenced

21
Techniques of assessment

• Written tests, examinations, assignments
• Practical skills testing lab/workshop practice
• Oral interviews, various formats
• Aural listening tests
• Project work individual/group research/design
• Field work data collection and reporting
• Competence testing threshold standards
• Portfolio combination of techniques

22
Common techniques in HE

• Paper/thesis
• Project
• Product development
• Performance
• Exhibition
• Case study/critical incident

• Clinical evaluation
• Oral exam
• Interview
• Research assignment
• Portfolio
• Others??

23
Interrogating our assessment

• Have we included the right mix of learning
  outcomes in our modules? (cf Blooms Taxonomy,
  SOLO taxonomy, for example)
• How do we know if students have achieved the
  intended learning outcomes is there a good match
  between learning outcomes and assessment? What
  about grading?
• How can we improve assessment so that it tests
  the intended learning outcomes?

24
1. Evaluating learning outcomes

• Appropriate to the module/subject/level?
• Appropriate balance cognitive (higher/lower
  level), affective, psychomotor/performance?
• Written at threshold standard what is required
  to PASS?
• Clear, concise, readily understood?
• Observable, measurable performance i.e.
• can it be assessed?

25
The right mix of learning outcomes

• Typical learning outcomes in higher education
• Knowledge, comprehension, application, analysis,
  synthesis, evaluation (Bloom)
• Remembering, Understanding, Applying, Analysing,
  Evaluating, Creating (Anderson et al)
• Knowledge Skills and know-how Competence (NQAI)
• Problem solving
• Working alone and in teams personal and
  interpersonal skills
• Communications information literacy
• Which of these are included in your courses? How
  many are assessed?

26
Focus on cognitive domain

• Lower order thinking skills
• Knowledge Simple recall of previously learned
  material
• Comprehension Ability to make sense of the
  material
• Next level of thinking
• Application Ability to use learned material in a
  new situation with a minimum of help or direction

27
Cognitive domain 2

• Higher order thinking skills
• Analysis Break material into component parts so
  that its structure may be understood
• Synthesis Put parts together to form a plan new
  to the learner generalise
• Evaluation Judge the value of material based on
  certain criteria

28
2. Matching assessment to learning outcomes

• Typical assessment components
• Practical tests (range of approaches)
• Continuous assessment (various forms)
• Written examinations (various types of items)

29
Matching assessment to learning outcomes

• Unless the measuring instrument matches the
  thing to be measured, we havent got a chance of
  learning what we want to know (i.e. whether the
  learner has achieved the desired learning outcome)

30
Matching assessment to learning outcomes

• Unless the measuring instrument matches the
  thing to be measured, we havent got a chance of
  learning what we want to know (i.e. whether the
  learner has achieved the desired learning outcome)

31
The right assessment?

• Learning outcomes
• Identify signs and symptoms of MS
• Formulate food product
• Identify an area for research
• Demonstrate good presentation skills

• Assessment?
• Make a presentation to peers
• Prepare a 1000-word research proposal
• Lab-based project
• Multiple choice questions

32
Fit for purpose?

• Apply the unicycle test
• At the end of this module, the learner will be
  able to ride a unicycle 100m along a level paved
  street without falling off
• 1. Define unicycle
• 2. Write a short essay on the history of the
  unicycle
• 3. Name at least six parts of the unicycle
• 4. Describe your method of riding the unicycle

33
3. Improving assessment

• Achieve better balance between two main purposes
  of assessment
• Assessment FOR learning
• Formative feedback, diagnosis, motivation,
  guidance, learning support
• Assessment OF learning
• Summative selection, grading, certification,
  progression, professional recognition,
  gate-keeping

• Align assessment more effectively with learning
  outcomes
• Assess all major course outcomes
• Select valid and authentic assessment methods to
  test the relevant learning outcomes
• Ensure that different categories of learning
  outcomes are assessed
• Do not test any significant learning outcomes
  that are not explicitly stated

34
Standards and grading

• Write learning outcomes at threshold (Pass)
  standard achievement above this can be graded
  (honours, distinction etc)
• Learning outcomes tie down what is required to
  pass
• Remaining 60 allows scope for exploration of
  ideas, creativity, excellence
• Make clear what performance is needed to achieve
  higher grade levels specify criteria

35
Example

• Learning outcome
• design a page layout to a given brief
• Assessment task (practical test/project)
• lay out the given information as a book cover
  following the publishers brief
• Assessment Criteria (how the performance will be
  judged)
• Clarity of font
• Appropriate colour combinations
• Attractiveness of design
• Match with brief and budget

36
3. Improving assessment

• Align assessment more effectively with learning
  outcomes
• Assess all learning outcomes
• Select valid and authentic assessment methods to
  test learning outcomes
• Do not test learning outcomes that are not
  explicitly stated
• Be aware of backwash effect of assessment on
  learning
• Assess often, use variety of methods

37
A first step - improving exams

• Work with colleagues to draft questions
• Decide what you really want to test
• Dont keep measuring the same things
• Include data in questions reduce memory
• Show what assessment criteria will be
• Use stepped or structured items
• Make a clear marking scheme
• Give feedback to students and colleagues

38
Types of exams and items

• Open-book exams focus on what students can do
  with the information
• Open-notes exams topics can be given and
  prepared in advance
• Structured exams (eg MCQs, short answers, t/f,
  fill blanks) can test not just memory
• Essays subdivide question and show how marks
  allocated teach required skills eg mind maps,
  structure, organisation of material
• Case study material can be given in advance

39
Some other techniques

• Reviews, annotated bibliographies
• Reports lab/field work, research task
• Projects and assignments individual or group
• Portfolios
• Presentations
• Interview/viva
• Poster display/exhibition
• Performance/production of artefact
• Practical examinations
• Competence testing authentic or simulated

40
Large-class assessment

• Concept maps
• Venn diagrams
• Three-minute essay/paper
• Short answer exams
• Gobbets (chunks of content)
• Letter to a friend
• Cloze tests

41
Giving feedback to students

• Make it quick, clear and focussed
• Relate it to the assessment criteria and learning
  outcomes
• Use rubrics or formal marking schemes to show how
  well the requirements are met
• Steps in feedback
• Affirm what is done well
• Clarify ask questions about specific aspects
• Make suggestions for improvement
• Give guidance about what learner needs to do next

42
Fit for purpose? Assessing your assessment