personal, transferable skills

1
Assessing student employability

• personal, transferable skills?

2
Skills?

• Meta-analysis over 4000 reported effect sizes
  (involving an estimated 1237000 subjects, ranging
  from kindergarten to college students. His
  analysis
• posits the interaction of four aspects of
  human thought operating in most, if not all,
  situations (1) knowledge (2) the cognitive
  system (3) the metacognitive system (4) the
  self-system. (Marzano, 1998 8)

3
Skills and employability

• Understandings
• Skilful practices
• Efficacy beliefs and dispositions
• Metacognition
• Knight and Yorke, 2004

4
Beyond skills

• The Future of Higher Education (2003)
• Skills and other attainments, qualities,
  achievements
• 21st Century Skills
• anything is a skill?
• Handout 1. What employers want from new graduates

5
Personal?

• Distributed knowledge expertise as a social
  practice
• Social development of knowledge (Nonaka and
  Takauchi, 1995)
• Informal and non-formal learning (Eraut, 2004)
• Knowledge used with others (Wenger, 1998 Brown
  and Duguid, 2000)

6
Transferable?

• Situated cognition
• Tacit knowings (Sternberg et al., 2000)
• Transfer is hard a battle
• Employability as a transition issue? (Purcell and
  Elias, 2002 CIHE, 2003 Mason et al., 2003
  Pickering, 2004)

7
Complex outcomes of learning

• Complex rarely and inconsistently defined
• Advanced.
• Slow (10000, 5000, xxxx hours).
• Fuzzy (precision only at the expense of
  validity).

8
Folk beliefs about assessment

• Assessment ? measurement
• Problems with summative assessment (Knight 2002)
• Limits of number
• High-precision, high reliability
• Threats to validity
• Standards and meaning
• Communicating achievement
• Even assessment centres open to criticism (Brown
  and Hesketh, 2004)

9
Standards (1)

• The problem of standards
• Norm referencing and grading on the curve
• Criteria-referencing ? but whence the criteria?
• Taxonomies
• Practice-based ? subject benchmarks, professional
  associations, SEEC and NICATS
• Research-based ? but never Bloom Anderson and
  Krathwohl (Handout 2)
• Application of criteria local networks of
  approximate understanding

10
Standards (2)

• The rules of the game
• Number of (qualifying) modules
• Module assessment requirements ? coursework and
  examinations
• Weightings (cwexams level 78
  practice-basedacademic).
• Condonation and compensation.
• Sickness and special circumstances.
• Process standards and the limits of the external
  examiner system.

11
The cost of reliable assessment

• Reliability not inherently expensive
• Precisionreliability is expensive
• ? trade-offs between
• Validity
• Reliability (with precision)
• Usefulness
• Affordability

12
A second approach to assessment

• Low-stakes or formative purposes
• Create informed feedback on achievement
• In order to improve future achievement for
  example, SENLEF (http//www.ltsn.ac.uk/genericcent
  re/index.asp?id19681)
• Feed-forward
• Interactive task design

13
Claimsmaking

• HEIs cannot warrant all valued achievements.
• Students must make claims in respect of the
  others ? PDP?
• They need to
• Understand the languages of achievement
• Identify development needs/possibilities
• Make claims
• Support claims

14
The political economy of assessment

• What do we want to do?
• What can we afford to do?
• What are our priorities?

15
Differentiated assessment

• What is assessed?
• For what purposes?
• How?
• By whom?
• When?
• Blended assessment ? formative, low-stakes and
  summative, high-stakes.

16
Recap

• High-stakes, summative assessment has limited
  reach
• Reliabilityprecision is expensive
• Low-stakes, formative assessment is good for
  learning
• Low stakes can reach achievements that elude
  high-stakes approaches
• Blended assessment preferably planned at a
  programme level.

17
Assessing work-based learning

• Good wbl has
• Clear learning purposes
• Encourages reflection
• Uses formative assessment
• Often involves creation of a portfolio
• Gains academic credit
• (Blackwell et al. 2001)

18
What to assess? (1)

• Direct observation of performance
• Purpose
• What outcomes are to be directly judged?
• Which will be inferred?
• Will the criteria/indicators be known to all?
• Who will assess?
• Will it be unobtrusive?
• Will it be authentic?
• How will this assessment lead to judgements of
  competence?

19
What to assess? (2)

• Indirect judgements of workplace achievement ?
  portfolios, simulations, OSCEs, presentations,
  commentaries, tests.
• Issue of complexity the more complex the product
  (portfolios) the costlier to assess fully.
• Issue of authenticity
• Standardised tasks
• Artificiality

20
Assessing consistently

• Writing criteria.
• Understanding criteria.
• Differences in workplace opportunities.
• Differences between assessors, especially when
  workplace assessors are involved.
• Most problematic with high-stakes judgements.

21
Suggestions for work-based assessment (1)

• Have a few, clear goals for work-based learning.
• Restrict high-stakes assessment to pass / not
  pass judgements.
• Ensure criteria are clear and understood.
• Say 80 of cases clear pass, 10 clear fail.
  Concentrate resources on the borderline 10.
• Build copious, low-stakes formative assessment
  into the placement.
• Use finer-grained criteria for low-stakes purposes

22
Suggestions for work-based assessment (2)

• Require students to maintain a portfolio of
  claims to achievement, backed up by workplace
  evidence.
• Encourage peer support and peer formative
  assessment
• Either grade portfolio as pass/not pass or set a
  more finely-graded task that can only be done
  well with a good portfolio to hand.
• Review all resource (time) allocation decisions
  and invest at points of greatest leverage.

23
A programme focus

• Knight and Yorke, 2003, Chapter 2
• Value of programme plans
• Distinction formative and summative assessment
  purposes
• Skills plus (Knight and Yorke, 2004)
• Low pain, high gain
• Auditing (core modules and student trackways)
• Orchestrating
• Tuning (refining)
• Communicating ? teachers and students

24
Achieving change in assessment practice

• further research is need to understand
  our understanding of learning from work and the
  most appropriate ways of measuring its progress
  and of enhancing its development. (Hager, 2004
  257)

25
Questions

• Who has got in place procedures for assessing the
  achievements employers value?
• What scope is there, over the next two years, for
  adopting blended assessment approaches (2005/06)?