Using SETU data: some thoughts from the trenches

1
Using SETU datasome thoughts from the trenches

• Stuart Palmer
• Institute of Teaching and Learning

2
Dr Set-you

• Or, How I
• Learned to
• Stop Worrying
• and Love Like
• At Least
• Not Hate the
• Thing

Source http//en.wikipedia.org/wiki/ImageCastle_
Romeo.jpg
3
THE HITCH-HIKERS GUIDE TO SETU

• Dont panic!
• Mostly harmless

4
A context-free SETU comment

• I found that Stuart seemed to be complacent
  about this unit, his lectures where not worth
  turning up too, as it involved sitting there
  listening to him talk from the unit guide, which
  I did by myself at home anyway.

5

• Liberte

6

• Je Remasse

7

• Le Tour Eiffel

8

• Gendarmarie

9

• Animalerie

10

• Stonehenge

11
Student evaluation of teaching (SET)

• Work by Ramsden and others in the 1980s
  (re-)established a link between student
  perceptions of their learning environment and
• the level of their engagement with their
• studies (hence, generally,)
• the quality of their learning/outcomes

12
What is quality?

• ?

13
What is quality in a pen?
Source http//en.wikipedia.org/wiki/ImagePen.jpg
14
What is quality in a pen?

• Writes smoothly?
• Lasts a long time?
• Doesnt break?
• Good grip?
• Has a highlighter on one end?
• Doesnt smudge?
• Has a gold nib?
• Writes in space?

15
Student evaluation of teaching

• Reflecting the modern understanding of the
  multi-faceted nature of quality and the finding
  that it is the entirety of the university
  experience that contributes to the student
  course experience, virtually all authors
  examining the value of the SET process recognise
  that SET data are only one of many sources of
  information that should be called upon when
  evaluating teaching

16
Triangulation
Source http//en.wikipedia.org/wiki/Orthographic_
projection
17
Triangulation
18
Triangulation
19
Triangulation
20
Evaluation of teaching

• Other valuable data sources include
• objective measures of student learning (such as
  unit marks)
• reflective (self) assessment of teaching
  performance (what worked, what didnt)
• peer assessment of teaching
• student focus groups

21
Evaluation of teaching

• etc. maybe Facebook?

22
Interpreting SETU

• USA Today has come out with a new survey -
  apparently, three out of every four people make
  up 75 of the population
• David Letterman

23
Interpreting SETU

• a reasonable number of responses
• a reasonable overall response rate and
• a reasonably equal rate of responses
• between campuses
• provide some confidence that the survey sample
  is representative

24
Consistency

• a small standard deviation on a mean item rating
  estimate indicates that most respondents were in
  agreement
• a large positive difference between the
  proportion estimates of agreement and
  disagreement indicates that substantially more
  students agreed
• if the written open-ended comments reveal common,
  repeated themes, these are worthy of further
  investigation
• etc.

25
Discrepancy

• a small number of ratings that are different from
  the majority can influence the reported mean
  estimate
• a small number of adverse comments that seem at
  odds with the majority of written feedback are
  probably not worth losing a lot of sleep over
• a rating in one item that is significantly
  different from the others in the SETU results for
  the same unit might be an indicator worth
  investigating further
• etc.

26
Interpreting SETU in general

• SETU is only one measure of the quality of
  teaching
• SETU is a pretty blunt instrument
• Used in an rational manner, the quantitative and
  qualitative results from SETU can be an aid in
  the evaluation of teaching and learning

27
Interpreting SETU in general

• The larger the number of responses, the more
  reliable the data
• Trends (as long as they are real) can be
  informative

28
Mean SETU responses 2003-2004
1 The teaching staff of this unit stimulated
my interest in further learning. 2 The
teaching staff of this unit motivated me to do my
best work. 5 This unit was well taught. 6 I
had a clear idea of what was expected me in this
unit. 17 I would recommend this unit to other
students. 18 The use of on-line technologies in
this unit enhanced my learning experience.
29
Mean SETU responses 2003-2005
1 The teaching staff of this unit stimulated
my interest in further learning. 2 The
teaching staff of this unit motivated me to do my
best work. 5 This unit was well taught. 6 I
had a clear idea of what was expected me in this
unit. 17 I would recommend this unit to other
students. 18 The use of on-line technologies in
this unit enhanced my learning experience.
30
Mean SETU responses 2003-2005
1 The teaching staff of this unit stimulated
my interest in further learning. 2 The
teaching staff of this unit motivated me to do my
best work. 5 This unit was well taught. 6 I
had a clear idea of what was expected me in this
unit. 17 I would recommend this unit to other
students. 18 The use of on-line technologies in
this unit enhanced my learning experience.
31
Mean SETU responses 2003-2006
1 The teaching staff of this unit stimulated
my interest in further learning. 2 The
teaching staff of this unit motivated me to do my
best work. 5 This unit was well taught. 6 I
had a clear idea of what was expected me in this
unit. 17 I would recommend this unit to other
students. 18 The use of on-line technologies in
this unit enhanced my learning experience.
32
SETU teaching comments

• I found that Stuart seemed to be complacent
  about this unit, his lectures where not worth
  turning up too, as it involved sitting there
  listening to him talk from the unit guide, which
  I did by myself at home anyway.

33
SETU teaching comments

• An exceptionally hardworking teacher. Always
  responding promptly to enquiries. Thank you
  Stuart for your efforts.
• Excellent teaching of course work
• Excellent teaching, great feedback, showed
  interest in progress. Best I have come across.
• Good lecturer.
• Great course content and well delivered

34
SETU teaching comments

• It was good how Stuart used videos to illustrate
  things in the real world. His lectures are well
  structured as all the unit material is covered in
  lecturers which are always informative. It was
  nice to get the reminder email about assignment1
  and 3, this shows he has compassion towards the
  students.
• The lectures were good and didnt drag on too
  much.
• The teaching support offered as part of this unit
  was excellent.

35
SETU teaching comments

• This unit was very well taught. I could not fault
  any part of it.
• Very easy to listen to, and teaches well
• Very promptuos lecturer
• Well structured class.
• very interesting to listen to and knows how to
  teach. was very helpful to me when i had
  questions
• very well taught, easily understood, madde the
  unit enjoyable.

36
SETU teaching comments

• Yes stuart palmer is dedicated lecturer who is
  willing to help and puts an effort into helping
  students. His lectures cover alot of content with
  plenty of examples.
• Good uses of informations and extra material to
  help get a point across
• sometimes it feels very long to has class for 2
  hours.

37
Be careful of the prominent outlier
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
Using SETU

• The circumstances, context and influences on a
  particular unit are individual in nature
• Standard, boilerplate responses probably arent
  that useful
• SETU might offer some insights of value

45
Drowning by Numbers
46
Online PD materials
47
Online PD materials

• Online PD modules
• www.deakin.edu.au/itl/pd/tl-modules
• Enhancing teaching and learning
• www.deakin.edu.au/itl/pd/tl-modules/scholarly