Students Evaluation of Educational Quality SEEQ

1

Students Evaluation of Educational Quality (SEEQ)
Dr. Mary Benbow Associate Professor,Department of
Environment and Geography University of Manitoba
2
Resources to Examine Evaluations

• Centre for Educational Advancement at Curtin
  University, Australia.
• Students Rating Teaching by Mark Lawall
• Many, many peer-reviewed research articles
• A selection is provided in the bibliography

• My web site
• http//home.cc.umanitoba.ca/benbow/seeq.html

3
Using the SEEQ

• Student evaluations what we know
• Student Evaluation of Educational Quality (SEEQ)
• Using the SEEQ

4
Student Evaluation What We Know

• A lot of research (some apparently contradictory
  and/or inconsistent)
• Difficult to compare different instruments
• Many useful reviews
• Students Rating Teaching
• Other articles (e.g. Ory, 2001)

5
Development and Use of Student Evaluations

• Introduced in the 1960s
• Students sought a stronger voice
• Today viewed as one component of accountability
• Over 90 of institutions of higher education use
  student evaluations of some sort
• Comprehensive scheme for student evaluations
  began at the U of M in the academic year 1996-97
  (Policy 425)
• September 2002, Senate approved voluntary posting
  of SEEQ results on U of M website

6
Courses Requirements, Levels and Sizes

• Electives tend to get higher ratings than
  required course
• majors more favourable than minors
• Higher level courses tend to get higher ratings
• There are differences between the disciplines
• Class size has little effect overall

7
Instructors and Students

• Student ratings positively correlated to those of
  alumni
• positive correlation between students and
  colleagues
• Lower ratings for less experienced instructors
  and teaching assistants
• relationship between research productivity and
  ratings is either positive or nil
• Higher ratings where there is prior interest
• Slightly higher ratings where gender of
  instructor and students is the same

8
Administration and Use

• Diagnostic not prescriptive
• Need to follow uniform procedures
• higher evaluations if followed by a short speech
• higher also if not anonymous
• higher if the instructor stays in the room
• Most important not the only source of
  information concerning teaching

9
Student Evaluation of Educational Quality (SEEQ)

• Widely researched
• very high degree of reliability (r 0.88 - 0.97)
• Based on psychometric analysis (teaching not
  content)
• correlate well with a wide range of measures of
  learning outcome
• correlates well with instructors self ratings

10
The development of the SEEQ

• Developed Herbert Marsh of the University of
  Western Sydney in late 1970s
• Internationally recognised expert in
  psychometrics
• SEEQ has been exhaustively researched
• Statistical tests over 13 years, from
  approximately 50,000 courses and almost 1 million
  students

11
SEEQ Factors Questions developed from interviews
with faculty and students about what constitutes
effective teaching

• Learning
• Organization
• Individual Rapport
• Breadth
• Assignments

• Enthusiasm
• Group Interaction
• Examinations
• Overall Ratings

Teaching as a multi-dimensional activity
12
SEEQ Questions

• Formative questions
• Identifies specific aspects of each factor
• Organization
• Instructors explanations were clear
• Summative overall questions
• 30. Compared with other courses I have had at the
  U. of M., I would say this course is
• 31. Compared with other instructors I have had at
  the U. of M., I would say this instructor is
• 32. As an overall rating, I would say this
  instructor is

13
Student Evaluation of Educational Quality
Using the SEEQ

• Little research on summative use
• Ratings are a reliable source of data
• Data must then be examined and evaluated
• Examine the data to create assessment
• Followed by decisions and examination of change

14
Using the SEEQ

• Understanding your priorities
• Undertake a self-evaluation of teaching
• Looking at your evaluations
• Identify solutions
• Making changes and smart decisions

15
Your Priorities

• Related to your Teaching Philosophy
• Describes the principles that direct your
  teaching style and activities
• Identify which factors and elements are most
  important to you
• Are your priorities consistent those of your
  discipline, department or institution?
• Can you establish a reason for your priorities
• Prior to examining your evaluations, it is
  important to assess which matter most to you

16
Self-Rating of Teaching

• The SEEQ correlates well with self ratings and
  trained observer ratings
• Useful to consider our assessment of teaching
• Collect evidence of teaching and assemble in a
  teaching dossier or portfolio
• Consider our assessment using a self-rating form
  based on the SEEQ
• Most of us await student reactions to our
  courses with both eagerness and trepidation
  (Delucchi and Pelowski, 2000)

17
Identify the most common responses you expect
from your class
18
Self-Analysis Looking at your course evaluations

• Are there any surprising responses?
• Which surprises are most relevant?
• Do the written comments reveal any more
  information?
• You can also use your statistical summaries of
  evaluation data (but differences of lt0.05 are not
  considered significant)

19
Analysis framework

• What special features reflect the subject, the
  students or other contextual factors?
• Eliminate any factors that are not appropriate to
  your intentions in teaching this class.
• What are the main strengths of your teaching?
• Which are the main weaknesses in your teaching?
• How do these findings differ from your
  Self-Rating Survey

20
Other Useful Data on the SEEQ

• Students Enrolled and Students Responding can
  indicate attendance (e.g. 62)
• Expected Grade can be compared to the actual
  grade distributions
• Reason for taking course can reveal how many are
  required to take a course

21
Theres always one! ?

• Look at the response sheets and identify which
  responses are the outliers
• Look at their written comments
• Look at the Student and Course Characteristics
• Are the outliers only in one question or subject
  area? (e.g. use of humour)
• Are they relatable (Level of interest (37)
  Learning questions)?

• What should we do about that one or two
  reponse(s) that stand out, perhaps anomalously?

22
Using the SEEQ Making changes

• Using your priorities and self rating decide
  where changes would be most useful
• Need to identify sources of information
• Choose information sources to suit your teaching
  and learning style
• Start small (quick, easy, cheap) before investing
  in large, long-term changes

23
Select key areas for improvement

• Which are the two or three areas that received
  the lowest ratings?
• What strategies for improvement could you use
  that would be appropriate to your particular
  situation?
• Could the Tips to Improve Academic Teaching help
  you in this?
• Would further personal guidance or investigation
  be needed to help you improve your knowledge and
  skills?

24
Making Changes

• Start small before investing in long-term changes
• Why go it alone?
• Could there be some changes that could be
  accomplished by group action?
• It takes time to create a new habit
• When you try something new, it may feel
  uncomfortable at first
• Give yourself reminders to try some new
  techniques or ideas

25
A good place to start are the Tips to Improve
Academic Teaching organized in 8 sections
referring to each factor
12. Instructor gave lectures that facilitated
taking notes Strongly Strongly
Disagree Disagree Neutral Agree Agree 9
18 3 13 7
This question comprises part of the Organization
factor choose Organization Clarity SEEQ Factor
3
26
Statistical Analysis of SEEQ Results

• SEEQ data cannot be analyzed or summarized
  statistically by administrators
• But, individuals can analyze their own data
• The Statistics Advisory Group recommends the
  following techniques
• Can be undertaken by hand or by using a computer
  spread sheet

27
Averages, Medians and Grouped Questions

• Find the average response for each question
• Find the median response for each question
• Find the average for each subject grouping or
  FACTOR (e.g. average response for all four
  questions under Learning)

28

• 1. I have found the course intellectually
  challenging and stimulating
• N/A 0
• 1 Strongly Disagree 0
• 2 Disagree 0
• 3 Neutral 3
• 4 Agree 20
• 5 Strongly Agree 2
• Average (01)(02)(33)(204)(25) 99/25
  3.96
• Median 4

29

• 1. I have found the course intellectually
  challenging and stimulating
• 2. I have learned something which I consider
  valuable
• 3. My interest in the subject has increased as a
  consequence of this course
• 4. I have learned and understood the subject
  materials of this course

Av. 3.96 Med. 4.0
Av. 4.16 Med. 4.0
Av. 4.08 Med. 4.0
Av. 4.24 Med. 4.0
Learning factor Average 4.11, Median 4.0
30
You may wish to provide summaries to indicate
improvements in your teaching
31

• You can also graph changes in averages (in a
  teacher dossier, include an explanation)

32
Additional Questions in the SEEQ

• At Curtin University (Perth, Western Australia)
• Staff engaged in the 1995 pilot of SEEQ were
  concerned that it might not accommodate forms of
  teaching and learning other than conventional
  face-to-face lecturing
• Teaching staff across the disciplines have
  subsequently developed factors and questions to
  complement the SEEQ form.

33
SEEQ Questions Examples The computer
laboratory

• 1. The laboratory was an essential part of the
  unit.
• 2. Laboratory-practical exercises reinforced the
  main points of the lecture.
• 3. By applying theoretical concepts in the
  laboratory, my understanding of the material
  improved.
• 4. The time and effort required for
  computer-based laboratory work was reasonable.

34
Written Comments

• Type them up
• Include course title, number, year, number of
  students, etc.
• What additional depth do they provide for the
  question responses?
• e.g. if you score well in Organization the
  comments may bring up a clear outline, good
  notes, web pages, etc.
• Note Not to be used for tenure or promotion

35
Students Evaluation of Educational Quality
(SEEQ) Conclusions

• Use your priorities and self rating to decide
  where changes would be most useful
• Innovations to improve teaching are everywhere
• Use the library, NetDoc or Google Scholar to find
  great information and ideas
• The SEEQs are not the only source of information
  about your teaching
• Develop your teaching dossier
• Develop your teaching philosophy and goals