Design, implementation and evaluation of a

1
Design, implementation and evaluation of a
'generic' ePortfolio the Newcastle experience.
Simon Cotterill, Tony McDonald, Paul Drummond,
Geoff Hammond School of Medical Education
Development University of Newcastle, UK
2
Background
SST (Student Support Tutoring)
1st Newcastle-Nottingham PARs project, 1998-2000

• Internet-PARs
• DfES funded
• PDP focus

• FDTL-4 project
• ePortfolios in Medicine
• Evidencing outcomes
• PDP

2nd Newcastle-Nottingham PARs project, 2000-02
3
Background

• An FDTL-4 project developing ePortfolios for
  undergraduate Medicine (2002-2005)
• Newcastle University (lead site)
• Leeds University
• Sheffield University
• Dundee University (for consultancy)

Project funded by
4
Aims
1. Personal Development Planning (PDP) "a
structured and supported process undertaken by an
individual to reflect upon their own learning,
performance and/or achievement and to plan for
their personal, educational and career
development Quality Assurance Agency for
Higher Education, 2001.
5
Aims
2. Supporting Life-Long Learning (LLL)

• b) Encouraging the development of LLL Skills
• Evidencing Learning Outcomes
• Transition to Independent learning
• Preparation for professional requirements
• - Appraisals
• - Assessment
• - Revalidation

6
Aims (technical)
3. Design a generic ePortfolio -transferable /
useful in different contexts
4. Integration with VLEs
7
Architecture
Generic framework
Developed with Open Source products Zope, MySQL
8
Generic framework Customisaton
Selecting tools by Course and Year
Course Admin view
Student view

• create context-specific
• tools via simple Web forms

9
Generic framework Customisaton
Configuring learning outcomes / skills sets
Course Admin view
10
Generic framework Sharing
11
Piloting Implementation The ePortfolio
developed at Newcastle has been applied in the
following contexts
2003/4 Undergraduate Medicine Contract Research
Staff (CPD) Generic Portfolio (project
demonstrator)
2004/5 Dentistry (Undergrad Vocational
Training) Undergraduate Biosciences St Andrews
University (Medicine)
12
Generic ePortfolio Public Demonstrator
httpwww.eportfolios.ac.uk
13
Medicine at Newcastle Integration with a bespoke
VLE
14
Common Tools Learning Diary (Medicine
Newcastle)
15
Context-specific Tools PDP for Year 4 Student
Selected Components
16
Common Tools Record courses (CRS)
17
Common Tools Learning Outcome / Skills (CRS)
18
Common Tools SWOT (Bioscience prototype)
19
Common Tools CV (St Andrews University)
20
Common Tools Tutor meetings (St Andrews
University)
21
Context-specific Tools Dentistry
Project involves 6 Dental schools and 2
Postgraduate Deaneries
22
Context-specific Tools Dentistry
Promoting Continuity LLL in Dentistry
23
Integration 2 Blackboard (via a tab)
24
Integration 2 Blackboard (via the tool bar)
25
Evaluation 1 Medicine (Years 12) Cotterill
SJ, Sarma S, McDonald AM, Bradley P

• Ethical approval / informed consent
• Focus groups Questionnaires
• Key issues
• Generally well received but what motivation ?
• not assessed
• not viewed by tutors
• (staff interface completed later)
• Need better clarity of purpose
• Wanted a more quirky / fun design !
• Learning diary - useful at first, less so over
  time
• Good to browse the Learning Outcomes

26

• Evaluation 2 Medicine (Year 4)
• PDP for student selected components
• Cotterill SJ, McDonald AM, Bradley P, Robinson R,
  Hammond GR

• Ethical approval / informed consent
• Questionnaires (before and after using the
  ePortfolio)
• Change in awareness of intended learning outcomes
  ?
• Factors influencing use of the ePortfolio
• Attitudes and perceptions of using the ePortfolio
• Student perception of impact of using the
  ePortfolio
• Evaluation of technical features and usability
• 186 students completed the portfolio (100
  compliance)
• 105 questionnaires completed (56 of cohort) at
  30/06/2004

27
Evaluation - positive
It encouraged me to really give thought to what
I wanted to achieve during the option, which was
especially useful as this was my first option. As
a result of the portfolio I think I got much more
out of the option than I would have otherwise.
It made me concentrate on creating aims at the
start of the option and allowed me to plan the
option with my supervisor in a defined way.
Overall it made my learning for the option more
organised and focused.
28
Evaluation - barriers problems
Most of the things I learned couldn't be
'measured/quantified' so I felt it was of little
value to try and invent a way in which they
could.
29
Building the ePortfolio was a useful learning
experience
80 thought it was a useful learning experience
30
At the end of the option I spent time considering
what I had learned from this option.
93 reflected on their learning after their SSC
31
Conclusions 1

• The generic ePortfolio
• Highly configurable
• Can be readily adapted for different contexts
• Existing tools have an emphasis on PDP
• -but the portfolio framework can serve other
  purposes
• Integration in VLEs
• Fully integrated into a VLE for Medicine
• Initial work to integrate with Blackboard has
  been successful

32
Conclusions 2

• Evaluation (in Medicine)
• ePortfolios implemented / piloted in 2003/4
• Demonstrated Feasibility Acceptability
• Most students perceive these as beneficial
• However, unless assessed / mandatory the
  motivation to complete is reduced.

33
Conclusions 3

• Educational Objectives (including PDP)
• Some evidence that PDP had a positive impact on
  planning and approach to learning (Year 4 SSC)
• Encouraged reflection in some students
• Good preparation (and practice) for LLL CPD
• However, long-term impact on learning is unknown
• Further research is essential
• Some (limited) evidence that PDP improves
  learning

A systematic map and synthesis review of the
effectiveness of personal development planning
for improving student learning. EPPI-Centre
(2002) http//eppi.ioe.ac.uk
34

• Additional Developments
• ePortfolios Extension Toolkit (ePET)
• -A JISC funded project to develop a Web
  services interface for the portfolio (by
  04/2005) http//www.eportfolios.ac.uk/ePET
• IAMSECT Shibboleth Single Sign-On project
• JISC funded project
• http//iamsect.ncl.ac.uk
• Further information on the ePortfolio
• http//www.eportfolios.ac.uk
• Simon Cotterill (S.J.Cotterill_at_ncl.ac.uk)

35
Implementation of ePortfolios - Humans are
important too !

• Buy-in of key stakeholders
• Learners
• Clarity of purpose ?
• Sense of Ownership ?
• Tutors Admin staff
• Extra work ?
• Senior Curriculum staff
• Integration with existing processes
• Training (all)
• Monitoring and evaluation

36
Selecting sub-sets of portfolio data for
different purposes
Portfolio for Presentation
Portfolio for Assessment
Portfolio for Application (job / promotion)
Portfolio for Accreditation/ Revalidation
Learners repository
Institutional Data
Portfolio for Appraisal
PDP (shared)
PDP / Reflective (private)