Forward Planning: Findings from a Roadmap for eAssessment

1
Forward Planning Findings from a Roadmap for
e-Assessment

• Denise Whitelock
• Institute of Educational Technology
• The Open University,
• e-mail d.m.whitelock_at_open.ac.uk

2
Background

• e-Assessment is of strategic importance to the UK
• Major global growth industry
• Recognised by JISCs e-Learning Programme
• Funded by JISC to produce a Roadmap

3
What is a roadmap?

• Obtain a consensus view or vision of the future
  landscape of e-assessment to decision makers
• Kostoff Schaller (2001)
• roadmapping provides a way to identify and
  evaluate and select strategic alternatives for
  the desired objective

4
Road mapping

• Technology push
• Requirements pull
• Layers
• Know why envisioning activities
• Know what policy and theoretical developments
• Know how - technological
• Time Dimension
• Adapted from Phaal, Farrukh et al, 2004

5
Adapting the Delphi Method

• Delphi questionnaire often used
  http//www.is.njit.edu/pubs/delphibook/index.html
  toc
• Consensus
• Reduced to lowest common denominator
• No priorities emerge
• This consultation will identify core and outlying
  issues avoid polarities

6
Application of the Delphi Method
7
Building the e-Assessment Roadmap

• Map current terrain
• Review current policy and initiatives
• Identify current practice
• Recognise practitioners concerns
• Check whether rhetoric matches reality
• Gauge stakeholders visions
• Recommendations for keeping on track

8
Roadmap framework
9
Sourcing the information for Stage 1

• Superinstitutional
• Policymakers HEFCE, SQA etc.
• Institutional
• HE, FE etc.
• Personal
• Tutors, students, developers, commercial companies

10
Superinstitutional Policies and Practices (1)

• All student centred
• Personalised learning/e-portfolios
• Instant feedback
• Learner control
• More choice

11
Superinstitutional Policies and Practices (2)

• e-Assessment will be rolled out in post-16
  education by 2009
• e-Assessment will make a significant contribution
  to reducing the assessment burden and improving
  the quality of assessment
• e-Assessment field trials should be taking place
  in at least two subjects per awarding body during
  2005
• 75 of key and basic skills tests will be
  delivered on screen by 2005
• N.B. 2009 deadline made explicit by QCA but not
  SQA

12
Institutional Drivers

• Increase in student retention
• Enhanced quality of feedback
• Flexibility for distance learning
• Coping with large student numbers
• Objectivity in marks
• More effective use of VLE

13
Institutional Vision

• More learner control
• Feedback
• Flexibility
• Embedding the e-learning within the institutional
  tools

14
Consulting Experts with a Survey

• Survey designed to probe the following
• Timings of policy implementation
• How e-Assessment can cut the burden of assessment
• Ways in which the quality of e-Assessment will be
  improved
• Implications of the vision of the policy
  documents
• Visions for the future
• Barriers to the visions

15
Experts

• Superinstitutional
• Representatives from SQA, HEFCE, BECTA etc.
• Institutional
• Directors of e-Learning Units and policy makers
  in HE and FE
• Personal
• Champions of e-Assessment

16
Survey Findings (1)

• Predicted timings
• Agree with 2010 deadline
• ICT accepted in all aspects of student experience
  within 2/4 years
• Tutors will have tools to assist with course
  design and providing electronic feedback to
  students within 2/4 years

17
Survey Findings (2)

• Cutting the burden
• Technological change means revamp of old practice
• Questions need to be more interactive
• Efficiencies is in results processing,
  transparency in grading, develop criterion
  referenced exams
• Introduce formative assessment
• Faster feedback
• Better plagiarism detection
• Production of essay marking tools
• Accessibility issues require higher profile

18
Survey Findings (3)

• Improving the quality of e-Assessment
• Offer more realistic assessments
• Cover wide range of curriculum
• Forces assessors to produce clearly defined
  objectives that enhance student learning
• Better regular feedback will assist students who
  usually under-perform
• More improvement seen in vocational sphere

19
Survey Findings (4)

• Implications for the vision set by the policy
  documents
• e-Assessment sites will be open 24/7
• More students will study University courses while
  at school
• Parents will over-pressurise students to take
  exams too early
• Over-use of on-demand testing does not always
  increase grades and produces lack of confidence
  in standards
• e-Assessment to play key role in personalised
  learning agenda

20
Policy implications

• 50 agreed students studying at own pace will
  result in higher grades and a lack of confidence
  in exam style
• Agreed resits should be regulated
• Agreed more breadth and depth of learning will be
  achieved through e-assessment
• 16 18 year olds will be studying some
  university courses while at school

21
Paperless examinations as recommended by CCEA
and Edexcel

• Authors need training
• Authors do not need same technical skills as
  technology partner
• Research into different question types sorely
  needed
• Good piloting essential

22
Further Expert Opinions

• Adaptation of existing standards regarding
  security (BS7799), quality (ISO9000), test
  delivery (BS7988) and item production (QTI).
  Commercial developers will need to subscribe to
  these standards
• Strong encouragement of open-source software
  licensing models
• An agreed level of interoperability based on
  national standards.

23
Visions Superorganisational

• Large-scale testing sites
• Test sites will deliver some paper and pencil
  components
• Prevalent from primary through to university and
  vocational qualifications
• High quality question banks
• No longer referred to as e-assessment

24
Visions Organisational

• e-Portfolios increase
• Formative, self assessment and e-portfolios main
  tools
• Random internal testing
• Confidence declared in systems by teacher,
  awarding bodies, students and general public
• More summative assessment in HE
• Flexible entry and examination

25
Visions Personal

• e-Portfolios exchanged maybe as microchips in
  business cards
• All e-assessment except for practicals
• e-Assessment seamlessly integrated into day to
  day practice
• Peer e-assessment
• Assessment for Learning
• More use of diagnostics

26
Barriers Superorganisational

• Customer attitudes
• Lack of public confidence
• Negative stories in the media
• Lack of integration of policies
• Key antenna of quality, accessibility,
  reliability and security are evaluated with pilot
  activities

27
Barriers Organisational

• Lack funding to encourage pilot and R D
• Lack of recognition in RAE
• Not engaging with in proven technology
• Lack infrastructure to pilot
• Lack of item banks
• VLE can push out champions

28
Barriers Personal

• Staff attitude
• Availability of resources
• Lack of IT skills in a pilot projects target
  student population

29
Barriers to adoption

• Reformation of methods
• Training and development
• Resources
• Technical
• Customer attitude
• Staff attitude
• Learner attitude
• VLE

30
Recommendations Superorganisational

• Clear policy directives
• Government agencies to support directions
• Standard system not acceptable to awarding bodies
• Standards for systems spelt out by QCA
• Initiatives to share expertise
• Recognition of training
• Encourage open source licensing models
• Co-ordinate dissemination at BECTA/JISC levels

31
Recommendations Organisational

• Tool development
• Examine role of VLEs on progress
• Follow SQAs lead COLA and SOLAR
• Find self and peer assessments
• Item banks
• Plagiarism
• Research into automatic text recognition
• Guidelines for best practice
• Accessibility

32
Recommendations Personal

• Training
• Recognition of effort/time
• Assist champions
• Fund studies of changes to job allocation and
  staff culture
• Appoint mentors
• Free text entry needed for student assessments

33
Research and Development

• VLE
• Evidence needed
• Learning objectives match e-assessments
• e-Assessment promoting learning control and
  reflective learning
• Pilot on demand testing
• Gauging emotional pressures as well as cognitive
  gains

34
Priorities

• Sort out authentication and plagiarism issues
• Train staff to devise e-assessments
• Build good test item banks following the lead of
  SQA
• Pass on good practice tips from current cohort of
  students to new learners using e-assessment for
  the first time
• Make more testing sites for summative assessment
  available and pass on good practice
• Support more research into automatic marking of
  e-assessment
• Address accessibility issues directly through
  superorganisational directives
• Public acceptance of high stake assessments needs
  to be achieved through dissemination of key
  findings using press releases, TV and press
  editorials and PR techniques

35
(No Transcript)
36
Summary

• Policy pressure main driver in England and Wales
• Affects FE more than HE
• More reliance than maybe anticipated on the
  commercial sector
• More use of formative e-Assessment
• More research funding into text recognition
  becoming acknowledged
• Andragogy rather than pedagogy?
• Roadmap final report http//www.jisc.ac.uk/elp_ass
  essment.html