Evaluating the learner experience of elearning Rhona Sharpe rsharpebrookes'ac'uk

1
Evaluating the learner experience of e-learning
Rhona Sharpersharpe_at_brookes.ac.uk
2
Dfes Unified E-learning Strategy

• To be effective, e-learning needs more
  challenging standards of evaluation. We must
  develop a better understanding of what counts as
  effective e-learning and the conditions for
  successful local implementation.
• (Chapter 5, point 68)

3
JISC e-learning and pedagogy programme

• Aims to promote effective e-learning
• pedagogically sound, learner-focused and
  accessible
• JISC (2004) E-learning and pedagogy background
  information. http//www.jisc.ac.uk/elp_programme_i
  nfo.html

4

• LEX
• The aim of the study was to investigate learners
  current experiences and expectations of
  e-learning across the broad range of further,
  higher, adult, community and work-based learning.
• Learner XP
• explore how learners' experiences with e-learning
  differ in different learning and teaching
  contexts, working in collaboration with 4 HEA
  subject centres.

5
Sharpe, R., Benfield, G. Francis, R. (2006)
Implementing a university e-learning strategy
Levers for change within academic schools. ALT-J.
14 (2), 135 -151.
6
University of Wales, Swansea, Blackboard User
experience survey 2003
7
HEA review of blended e-learning

• Six of the seven institutions explicitly
  identified institutional level evaluation of
  blended learning as problematic, e.g.

8
HEA review of blended e-learning

• Six of the seven institutions explicitly
  identified institutional level evaluation of
  blended learning as problematic, e.g.
• One of the things thats been seriously lacking
  over the last five years at least, has been a
  lack of interest in evaluation, proper
  evaluation, not lip service evaluation like the
  student satisfaction surveys (Blackwater 2).
• We don't have any systematic, institution-wide,
  sufficiently detailed research into the student
  experience in my view (Longside 1).

9

• Jenny and Emma

10
The learner experience of e-learning (LEX)

• Linda Creanor Kathryn Trinder (Glasgow
  Caledonian University)
• Doug Gowan Carol Howells (The Open Learning
  Partnership)
• Interviews and focus groups with learners across
  the post-16 educational sectors in order to
  capture the role and impact of technology on
  their learning.

11
What some LEX learners have been saying

• To me its just learning, the fact its online as
  apposed to in a classroom is irrelevant, its
  just another way of accessing it... it strikes me
  as quite old fashioned and quite quaint, but
  talking to other people they're like 'oh wow! Its
  online! Its e-learning!' and I think it depends
  on where you're coming from what it means to you,
  but for me I just think of it as learning and I
  don't use the term.
• (Adult online learner)

12
What some LEX learners have been saying

• I'm addicted, it's the first thing I turn on in
  the morning before I even wake up and actually
  it's very, very bad. I think in the future people
  can't cope without their laptops. My main use
  My Space and Messenger and e-mail information
  gathering current affairs, news. I have alerts
  coming into me so I get information and then I
  use search engines for academic purposes.
• (Undergraduate Business student)

13
What some LEX learners have been saying

• and I was lying on the beach with my i-Pod
  and it just had been through so much, like you
  remember lying on beachwith i-Pod. You remember
  on the planewith i-Pod, so it was an emotional
  attachment, sad, but I loved that thing.
• (First year undergraduate student)

14

• Learner Experiences of e-Learning Exploring
  subject differences (LearnerXP)
• This project will explore how learners'
  experiences with e-learning differ in different
  learning and teaching contexts, working in
  collaboration with 4 HEA subject centres..
• Gráinne Conole, Open University
• Maarten de Laat, Exeter University
• Jonathan Darby, Southampton University
• Teresa Dillon, Polar Produce

15

• Medical student
• Today I used a laptop and projector to present
  my PowerPoint presentation. I used Word to
  prepare the handouts for my presentation Then I
  did a BMS e-module on Acne and Primary Care as
  part of a module Im doing at the moment.
• Computer science undergraduate
• You type in one word and you get all the
  information thats out there on it without having
  to look through different books.

16

• I use email to communicate with everyone,
  especially lecturers arranging meetings, asking
  questions about work and queries over assignments
  etc I write all my assignments using Word and to
  sort through the information I find, make notes
  of what I still need to do and spell check my
  emails that I'm sending to lecturers. Search
  engines are used to find news articles

17

• The first thing I do when given any piece of
  work is type it into a search engine! This gives
  me the opportunity to see how different people
  interpret the title. From there I can focus on
  one main idea and use the electronic resources to
  support my initial findings or indeed rule them
  out.

18
Are these useful sources of evaluative data?
19

• reading through all the feedback data from
  students and tutors is like standing at the
  apocryphal spaghetti junction and watching cars
  going every which way.
• Some students call for more group work others
  want none at all. .. Advice fumes the air.
• (Mason Weller, 2000)

20

• Teachers, peers, technology, course design and
  the learning environment are key variables that
  influence the learners' experience and success
• Atack, L. and Rankin, J. (2002). 'A descriptive
  study of registered nurses' experiences with
  web-based learning.' Journal of Advanced Nursing
  40(4), 457-465

21
Possible functions of evaluations

• Is it being used?
• How is it being used?
• Do the staff and students like it?
• How do the students experience it?
• Is it effective?
• Finding evidence for known situations
• Supporting pilot projects in each department
• Publicity and promotion of the service
• Identification and creation of case studies

22
Purposes of evaluation

• Formative To ensure that a product reflects the
  intentions of its designers AND meets the needs
  of the users
• Summative To test a product
• Integrative To integrate a product into a new
  environment
• Pragmatic To discover the unexpected
  benefitsilluminativeTo get a project funded
• Goodyear (2001) Effective networked learning in
  higher education notes and guidelines (p.
  37-40)

23
Research questions

• What strategies do e-learners use and what is
  effective?
• What beliefs and intentions do effective
  learners display?
• What might characterise effective learners in an
  e-learning context?
• How does e-learning relate to and contribute to
  the whole learning experience?
• How do learners manage to fit e-learning around
  their traditional learning activities?

24
Developing some questions

• Write down a few research questions
• Find 2 other people who have similar questions to
  you
• In groups of 3, build a set of primary and
  secondary questions.
• Be ready to feed back

25
(No Transcript)
26
Data collection methods

• Course Experience Questionnaire
• end of module feedback forms
• Survey (likert, open, closed)
• critical incident questionnaire
• Cognitive styles
• Learning styles
• open ended online reaction sheets
• online discussion
• coding of text transcripts
• web server usage/login logs
• test scores
• participant observation
• focus groups
• interviews

27
Data collection methods

• Individual face to face interviews (LEX)
  sampling, interview schedule, transcript analysis
• Audio diaries (Learner XP)

28
Using audiologs

• Learners from 4 discipline areas, 5 from each,
  phoned in up to daily over two weeks to report
• what technologies they used
• what they did with the technologies
• how they felt about the technologies they used
• Note Used Voipfone (www.voipfone.co.uk) which
  gives unlimited length messages saved as .wav
  files free 0560 number

29
Reflections on method

• Some learners excellent, others forget to call in
• Phoning in worked well
• Cant always catch every word
• Gives a real day in the life sense of how it
  was for learners
• Some surprising findings not captured by other
  methods
• come to 2pm paper session today for more details
• Captures frustrations well
• Will definitely use again

30

• Data triangulated with
• results of online survey (400 responses)
• face to face interviews with each discipline
  group
• sharing of preliminary results with learners in
  focus group setting (planned)

31
Triangulation

• An evaluation of the introduction of Personal
  Response Systems in first year engineering
  mechanics module at the University of
  Strathclyde
• five focus groups (each with 6 students) that met
  twice
• a critical incident questionnaire in the form of
  an A4 sheet with five questions and spaces for
  comments to record immediate experiences
• a 36 statement Likert scale survey derived from
  issues that emerged from the focus group data
• a focus group discussion with 6 staff.

Boyle, J. T. Nicol, D. J. (2003) Using
classroom communication systems to support
interaction and discussion in large class
settings. ALT-J. 11 (3), 43-57.
32
SOLE projecthttp//sole.ilrt.bris.ac.uk/

• The approach is designed to provide an in-depth
  set of case studies based on course modules
  across a range of subjects, with a wide range of
  data and collection methods.
• Student questionnaires(weeks 1 and 9)
• Student diaries (weeks 3 and 8)
• Transaction logging (throughout)
• Recording of interactions (throughout)
• Interviews with tutor (weeks 1 and 9)
• Interview with students (week 9)

33
Further resources

• LEX study http//www.jisc.ac.uk/elp_lex.html
• Learner XP study www.jisc.ac.uk/index.cfm?nameelp
  _learnerxp
• HEA Review of undergraduate blended e-learning
  available from HEA website from 1 Oct at
  http//www.heacademy.ac.uk
• OCSLD Online course on Researching and Evaluating
  e-Learning at http//www.brookes.ac.uk/services/oc
  sld/