Elearner narratives of experience: innovating evaluation Caroline Daly, Norbert Pachler, Jon Pickeri

1
E-learner narratives of experience innovating
evaluationCaroline Daly, Norbert Pachler, Jon
Pickering and Jeff Bezemer Institute of
Education, University of London

2
The Master of Teaching (MTeach)

• approximately 140 participants on various routes
  (post-induction experienced school-based,
  Institute for Ismaili Studies)
• based on the notion of shared knowledge
  construction at a distance through
  computer-mediated communication
• implies a corporate responsibility for developing
  teacher knowledge, reflected in the course
  infrastructure, module content, models of
  assessment and key players
• focuses on the art and science of teaching,
  requiring participants to examine closely their
  own classrooms as places of learning
• not preoccupied with the acquisition and critical
  analysis of established orthodoxies and
  epistemologies in the field, but concerned with
  understanding more deeply their practice in the
  context of a learning culture

3

• stresses the agentive dimension of teacher
  development, i.e. the reciprocal, symbiotic
  relationship between practice and intellectual
  reflection (scholarship of teaching)
• the knowledge base of the course is not located
  in the Institute of Education, nor in the set
  (digitised) readings, nor in the tutors but in
  the interactive environment of the electronic
  forums themselves
• use of portfolio assessments (containing
  evidence studies), nurturing an enquiry
  approach to understanding teaching and learning
  and raising levels of research literacy and
  practitioner research capacity
• develops research readiness in the early stages
  of the course, taking participants beyond the
  what works agenda, through an expanded
  understanding of critical classroom enquiry
• based on the notion of a professional learning
  academy, which has at its heart a number of
  linked online learning communities (MTeach tutor
  groups) consisting of programme participants,
  university tutors and some others

4

• requires participants to provide public accounts
  of several aspects of teaching, e.g. the
  establishment of learning goals, the management
  of learning environments, the design of
  materials, the evaluation of outcomes, the
  assessment of learning etc.
• much effort is expended on enquiries as a basis
  for productive professional conversations
  (talking teaching)

5
Pedagogical considerations

• the pedagogy of the course emerged over time
  through the shared spaces of interdisciplinary,
  cross-curricular and cross-phase planning
  meetings
• an antidote to a prescriptive initial teacher
  education curriculum which progressively
  incapacitates teacher educators and conceives of
  them as little more than delivery agents of
  government prescriptions (standards)
• leadership is conceived of outside of current
  managerial agendas, focussing instead on
  pedagogical leadership and leadership of
  learning
• the potential afforded by new technologies has
  significantly informed the design of the MTeach
  and its delivery mixed mode employing online
  networking as well as some traditional
  face-to-face modes
• course tutors have, by-and-large, not had a lot
  of prior experience in online tutoring and had to
  learn by doing how to create an environment
  conducive to the constructional and
  conversational discourses upon which the MTeach
  is predicated

6

• access to discourse transcripts documenting
  collaborative learning (i.e. the closed
  web-archive of contributions to online
  discussions)
• some technical and conceptual challenges around
  certain aspects of the particular online
  environment used (originally YahooGroups, now
  Moodle) and at perceptual-pedagogical level, for
  example around the effective use of threading and
  the development of strategies for monitoring the
  quantity and quality of participants
  contributions
• anxieties around ensuring that participants work
  meets with requisite (i.e. Masters level)
  standards course participants invariably
  contribute differently to electronic forums
  compared with formal seminar rooms, i.e. much of
  the work is conversational in nature and it
  involves participants continually in a discourse
  about their own learning and that of their peers
  with reference to relevant conceptual and
  theoretical frameworks
• the MTeach is characterised by a strong belief in
  social constructivist approaches to learning,
  i.e. the co-operative nature of knowledge
  building and pursuit

7

• template-based approach tasks as key cognitive
  tools (cf. social interactionism)
• templates encourage the on-the-fly recording of
  thoughts and impressions whilst examining
  learning materials and they provide a framework
  for information gathering, the stimulation of
  recall of prior knowledge and the guiding of
  knowledge construction
• problem-solving in orientation, encouraging
  MTeach participants to go meta about their
  teaching
• importance of three educational processes from
  divergence to convergence
• idea generating (and gathering)
• idea linking, organization, and intellectual
  progress and
• idea structuring (and convergence)

8

• case methodology generative potential of
  discourse potential of narrative or storying
  in learning (about teaching)
• MTeach participants to develop case studies which
  we see as a re-collected, re-told,
  re-experienced and re-flected version of direct
  experiences

9
Continuing Professional Development
accredited CPD
challenging orthodoxies
portfolios
scholarly teaching
Work-based learning
Criticality research literacy
critical incidents
practitioner research
socio-constructivist pedagogy
learning journals
online discussions
Collaborative learning
response partners
mixed mode
10
Rationale

• We need our course development to be informed by
  deeper understanding of our e-learners
  experiences and consider questions like
• what are (their perceptions of) the pedagogical
  advantages?
• does the use of technology create social
  barriers?
• Otherwise we are driven by other factors which
  are important, but can dominate our course design
  while we strive to keep ahead in the world of
  e-learning e.g.
• Keeping up with technology
• Widening participation
• Increasing student numbers
• Keeping costs down
• so that we feel like this

11
Why focus on e-learners experiences?

• The primacy of improving the student learning
  experience (HEA Strategic
• Plan 2005 2015) for overseas and home
  students.
• The inadequacies of evaluation based on
  satisfaction and exit models
• which consult students after study is
  completed.
• The gap between our intentions and the
  students experiences might be
• bigger than we thought (cf. the efficacy of
  designed-in narrative)they
• can cover up at a distance.
• We perceive coherence in our learning designs
  where the students may
• not. Our assumptions about what is going on
  can be wrong.
• We need a sustainable way of understanding
  our learners experiences
• when they are at a distance and
  encountering new ways of learning with
• technologies.
• We need ways which can help us know what the
  e-learning experiences on
• our courses are really like are our
  efforts worth it?!

12
Aims of the research project

• to identify e-learners experiences in the
  mixed-mode context
• to trial a new (embedded) approach to evaluation
• to analyse the implications for e-learning course
  development
• to see if, therefore, we should redesign our
  course in the light of what we found out about
  learners experiences

13
(No Transcript)
14
Why narrative evaluation methods?

• a way of capturing the learner voice
• a way of better understanding the realities of
  being an e-learner on our
• courses as a prime evaluation tool (cf.
  verisimilitude the intention was
• not to prove anything, not about
  logico-scientific ways of understanding,
• rather than to understand and adopt a stance
  which makes a difference
• to teaching)
• to provide ongoing information about the course
  while it is happening
• meaning located in place and time gain
  autobiographical insights
• to impact on the learners abilities to adapt
  to new learning contexts

15
What narrative evaluation methods did we use?

• individual, loosely-structured narrative
  interviews using postmodern techniques
  (Gubrium, 2003)
• naïve interviewer stance
• emplotment
• chronology
• without overspecifying the substance or the
  perspective of the talk (McCracken,1988)
• group narratives aimed at collective sense-making
  of experiences
• online forums as virtual focus groups (Bloor,
  2001) to elicit reflective group discussion on
  key issues
• online think aloud to elicit individual responses
  to mediating objects, e.g. images (Eraut, 2000)

16
Greenhalghs criteria for a good story in
narrative research
17
Analysis of narrative data

• inductive
• systematic
• inter-researcher coding
• continuous dialogue within a research team
• often involves stages of interpretation
• identification of themes in the first instance
  surface level, declaratory
• identification of sub-textual or latent themes
  and patterns
• development of categories revealing controlling
  principles, motivations, unarticulated
  experiences
• hypothesising about the meaning of the narratives
• development of vehicle by which the narrative
  data can be presented and findings communicated
  a diagram and sample cases using quotations
  from the narratives themselves

18
What participants told us about being an
e-learning community
it's a bit daunting posting work, without it
being checked first!! I'm sure I'll get used to
it
I am aware that other professionals are reading
my work before anyone has told me whether it is
any good or not.  Now, I know I'm a teacher and
I'm nearly 26 yrs old and I should probably have
grown up a bit by now but I hate getting things
wrong (!) and it feels a bit like being asked to
read your work aloud to the class before your
teacher has checked it.  Does anyone else know
what I mean ?
19
it's a big learning curve for me like this
before
Now that responses are being posted I am
beginning to feel more confident in this on-line
learning processand that a community is
beginning to emerge.
It certainly takes some getting used to, as I too
have not done anything like this before. I am
finding that when Ive read someones
correspondence I want to talk about it
immediately and explore the issues involved
20
The style inwhich a person writes I feel gives
an insight into their personality, as youread
their writing you can often build a sense of who
they are and thesituation they are coming from.
or perhaps not knowing the person who posted the
tasks could be more benefit to bringing a sense
of community as it allows us to take a more
non-judgemental view as we do not have the
baggage that people bring with them? Just a
thought.
21
E-learners experiences four types of newness
which affect participation for all

• These are the new areas of experience which the
  e-learners chose to talk about in their learner
  narratives
• the experience of managing work, life and
  learning with technology
• the experience of writing with fellow students in
  an online discussion in order to learn
• the experience of collaborative learning in an
  online tutor group
• the experience of social relationships with other
  e-learners

22
Learning about learning a positive outcome
for participants in a new approach to evaluation
based on learner narratives

• Most participants show considerable interest in
  how they learn with others and what
  impedes/supports this
• They assume responsibility for the conduct of the
  online forum
• They reflect on un/desirable features of online
  discussions
• There is increased commitment to the
  collaborative processes (e.g. responding to
  online partners, sharing files and documents)
• They take on teacherly roles
• They develop a respect for learner differences
• They create their own learning culture, shaping
  the context of their own learning and how it
  works at a distance
• Both novices and experienced e-learners gain from
  such activities and are extremely candid they
  want to tell

23
Second level of interpretation latent
categories of experience (getting used to
e-learning)
vulnerability security expertise
incompetence formality informality
newness continuum
24
Meta-learning

• An unexpected outcome has been the impact on the
  learners of discussing their learning and
  reflecting on how they learn within the
  collaborative online environment
• Considerable degree of interest in how they learn
  with others and what impedes/supports this
• Considerable commitment to the collaborative
  processes (e.g. response partners, sharing files
  and documents, detailed consideration of issues
  raised in online course discussions)
• They create a culture of use, shaping the
  context of their own learning and how it works at
  a distance
• Articulations are shaped by the social and
  physical environment in which they were produced
• among each other in tutor-prompted online
  commentaries
• with an interviewer who is not an online tutor
• at school, in a classroom, at IoE, with or
  without filming

25
Implications for programme teams

• There are benefits for learners of
  participating in narrative evaluation activities,
  
• by which they address explicitly their (new)
  experiences of learning with
• technologies.
• There are benefits for practitioners in gaining
  knowledge of unanticipated
• learner experiences, though these may not
  always lead to simple modifications
• of course design or learner tasks.
• There is an argument for course design to
  invest learner time in developing the
• abilities of participants to reflect on how
  they are learning. There is growing
• research evidence that this has a positive
  impact on what they learn.
• There is a need for meaningful evaluation
  activities which also support learning.
• Engaging with evaluation becomes a different
  type of experience for both tutors
• and students

26
(No Transcript)
27
Bibliography

• Bloor, M., Frankland, J., Robson, K., Thomas, M.
  (2001) Virtual Focus Groups Focus Groups in
  Social Research.
• Bruner, J. (1985) Narrative and Paradigmatic
  Modes of Thought in Eisner (ed) Learning and
  Teaching the Ways of Knowing Chicago University
  of Chicago Press
• Eraut, M. (2000) Non-formal learning, implicit
  learning and tacit knowledge in professional
  work in (ed.) Coffield, F. The necessity of
  informal learning Bristol Policy Press ESRC
  Learning Society Programme 12-31
• Greenhalgh, T. (2006) What seems to be the
  trouble? Stories in illness and healthcare
  Oxford Radcliff Publishing
• Greenhalgh, T., Russell, J. and Swinglehurst, D.
  (2005) Narrative methods in quality improvement
  research, Quality Safety in Health Care 14
  443-449.
• Gubrium, J. (2003) Postmodern Interviewing
  London Sage.
• McCracken, G. (1988) The Long Interview Newbury
  Park Sage.