Evaluating Student Learning Experiences

1
Evaluating Student Learning Experiences

• Michael ProsserHigher Education Academy

1
2
Academy Mission

• The Academys mission is to help institutions,
  discipline groups and all staff to provide the
  best possible learning experience for their
  students.

3

• Presentation based upon
• Over 20 years research into the student learning
  experience in higher education in the United
  Kingdom, Australia, Sweden and Hong Kong
• Experience in Australia with the Course
  Experience Questionnaire
• Interpretations of the results of evaluations of
  student learning experiences are not value or
  theory free
• Interpretations in terms of student satisfaction
• Interpretations in terms of student learning
  experiences

4
  Overview of the student learning perspective
Figure 1 Model of Student Learning
5

• STUDENT APPROACHES TO LEARNING
•  
• Surface Approach
• Intention to reproduce
• - rote memorise information needed for assessment
• - failure to distinguish principles from examples
• - treat tasks as external impositions
• - focus on discrete elements without integration
•  
• Deep Approach
• Intention to understand
• meaningfully memorise information for later use
• - relate new ideas to previous knowledge
• - relate concepts to everyday experiences
• relate evidence to conclusions 
• Approaches vary between tasks and modules

6
STUDENTS EPERIENCES OF THE LEARNING
CONTEXT Early research by Entwistle and Ramsden
(1983) using both interviews and questionnaires
identified a number of student experiences
relating to the way they approached their
studies Student experiences of Quality of
teaching including quality of feedback (NSS
Teaching, Assessment and Feedback)) Clearness of
goals of course and standards of assessment (NSS
Assessment and feedback) Workload so high that it
was not possible to understand everything Assessme
nt measuring reproduction and not
understanding were found to relate to how they
approach their studies and to learning outcomes
(exam results and other indicators)
7

• RELATIONSHIP BETWEEN COURSE EXPERIENCES AND
  APPROACHES
• Study of over 8000 students in first year
  subjects around Australia
• Amongst the data collected were students
  responses to
• contextualised Ramsden's Course Experience
  Questionnaire and a
• contextualised Biggs Study Process Questionnaire
• 1994-1996 Australian Research Council Academic
  Departments and the Quality of Teaching and
  Learning Paul Ramsden, Griffith University,
  Elaine Martin, RMIT, Michael Prosser, La Trobe
  University, Keith Trigwell, UTS

8
Approaches to Study Surface Approach 32. Although
I generally remember facts and details, I find it
difficult to fit them together into an overall
picture 35. The best way for me to understand
what technical terms mean is to remember the
textbook definitions   Deep Approach 28. I try to
relate ideas in this subject to those in other
subjects, wherever possible 34. In trying to
understand new ideas, I often try to relate them
to real life situations to which they might
apply.   Biggs Study Process Questionnaire
9
Student Experiences Of Learning Environment Good
Teaching 15. The staff made a real effort to
understand difficulties students might be having
with their work. Clear Goals and Standards 1. It
was always easy to know the standard of work
expected 6. I usually had a clear idea of where I
was going and what was expected of me in this
subject. Appropriate Workload 25. The sheer
volume of work in this subject meant that it
couldn't all be thoroughly comprehended
(-). Appropriate Assessment 8. To do well in this
subject, all you really need is a good memory
(-). Ramsdens Course Experience Questionnaire
10
Factor Analysis of Experiences of T L Context
and Approach to Study ____________________________
_____________________________________ Scale Fa
ctors ____________________________________
1 2 _________________________________________
________________________ Experiences of
Context Good teaching .80 Clear Goals and
Standards .67 Appropriate Workload -.69 Approp
riate Assessment -.65 Approach to
Study Surface Approach .81 Deep
Approach .73 ___________________________________
______________________________ Principal
Components, Variamax Rotation, n8837
11
 In each subject   1.   A deep approach is
associated with experiences that the teaching is
good and the goals and standards are clear (NSS
Teaching, Assessment and Feedback)   2.   A
surface approach is associated with experiences
that the workload is too high and assessment
tests reproduction That is, variation in
students experiences of the learning environment
within subjects is associated with the approaches
to study within subjects within subject
variation in perception is not measurement
error. 1994-1996 Australian Research Council
Academic Departments and the Quality of Teaching
and Learning Paul Ramsden, Griffith University,
Elaine Martin, RMIT, Michael Prosser, La Trobe
University, Keith Trigwell, UTS
12
(No Transcript)
13

• Comments on NSS
• Substantial amount of research linking results on
  surveys such as the NSS with student approaches
  to study and learning outcomes
• Student experiences are a function of both their
  prior experiences and understandings and the
  course design and teaching
• Do not necessarily improve student satisfaction
  by focusing on satisfaction - the individual
  items or scales need to better understand why
  they responded the way they
• Spread of results proportions responding in
  certain ways better than mean of responses
• Changes is scores over time of surveys such as
  the NSS represent broad cultural change
  departmental institutional experiences
• Can only expect small effect sizes in changes in
  scores over time - .2 of a SD if over 3 - 5
  years changes of the order of .1 to .2 points
  (3.5 to 3.6 or 3.7).
• Follow up with more qualitative studies
  interviews, open-ended written statements, focus
  groups to better understand experiences

14
  Overview of the student learning perspective
Figure 1 Model of Student Learning
15

• Recent Review of Research and Evaluation on
  Student Learning Experience in
• E-learning
• Rhona Sharpe, Greg Benfield, Ellen Lessner and
  Eta DeCicco,
• Much research on design of learning experiences
  teacher, course or program focused
• Much research on student observable behaviours
• Little or no research on the student learning
  experiences of those designs (intentions,
  perceptions, beliefs, outcomes etc)
• Little research on the student experience of the
  integration of online with face-to-face how
  e-learning relates to their whole learning
  experience and to their lives and jobs

16

• Learning Through Blended Discussion
• Peter Goodyear, Rob Ellis, Mike Prosser
• (acknowledge Australian Research Council Support)
• Work in Progress
• Student experience of learning through online and
  face-to face discussion
• Using a phenomenographic approach to analysing
  student learning experiences
• Key aspects of differences in the way something
  is experienced not rich, thick descriptions
• Second-order description and analysis not
  describing what it is but how it is experienced.
• Categories of description

17

• Investigate associations amongst
• how the students approached their discussion
  within blended learning context,
• how this variation was related to the quality of
  what they thought they were learning through
  discussion, and
• their performance in the course as a whole
• In-depth, semi structured interviews short
  open-ended questionnaire, assessment results
• Interviews 30 minutes, transcribed, main data
  for construction of categories and outcome space
• Short open-ended questionnaire same questions
  and leading questions for interview

18

• Questions
• What did you learn through discussion in your
  course? This includes all of the discussions that
  you were involved in in the course
• How did you approach engaging in face-to-face
  discussion in your course? What sorts of things
  did you do to engage in the discussions? Why did
  you use these strategies to engage
• How did you approach engaging in online
  discussion in your course? What sorts of things
  did you do to engage in the discussions? Why did
  you use these strategies to engage
• Sample
• 51 second your university undergraduate students
  studying social work in a large, research
  intensive metropolitan Australian University 19
  completed semi-structured interviews

19
Categories of Description of Conceptions of
Learning through Discussions
20
Categories of Description of Approaches to
Learning through Online Discussion
21

Chi-square24.2, phi.69, plt.001 Statistically
significant and large effect size No cases of
Fragmented conception of learning with Deep
approach to discussion on-pile Student
understanding of why they are engaging in
discussion fundamental to adopting deep
approaches to discussion online.
22
Relationship between Conceptions, Approaches and
Performance
23

• Focus on the student experience rather than
  satisfaction better understand their
  experiences
• Comprehensive and aligned set of student
  evaluation instrument designed and interpreted in
  terms of student experiences
• Conceptual model of teaching and learning based
  upon the student experience underlying all
  developments
• Follow up with more in depth qualitative studies
• Focus on the student experience