Please silence cell phones.

1
Please silence cell phones.

• THANK YOU!

2
Challenges and Advantages of Online Class
Evaluations
Shohreh Bozorgmehri Manager, Instructional Web
Technologies Briandy Walden Web Developer,
Instructional Web Technologies
3
TOPICS

1. BACKGROUND
2. CHALLENGES
3. TRANSITION
4. ADVANTAGES RESULTS
5. FACTS FIGURES
6. FUTURE
7. CONCLUSION
8. QUESTIONS

4
BACKGROUND
Inspiration

• Paper
• Online
• Version 1 Instructor-initiated
• Version 2 Department Administrative Tool for
  Evaluations (D.A.T.E.)

5
BACKGROUND

• Paper Process
• Purchase Scantron forms
• Organize distribute packets
• Fill in forms
• Collect return forms
• Scan forms
• Analyze results
• Distribute results to instructors

6
BACKGROUND
Academic Senate Role

• EEE final evaluations are based on the
  Campus-wide Teaching Evaluation Form (CTEF)
  approved by the academic senate committee, the
  Council on Student Experience (CSE).
• Senate proposal
• http//www.senate.uci.edu/9_IrvineManual/3ASMAppen
  dices/Appendix11.html
• http//www.senate.uci.edu/9_IrvineManual/3ASMAppen
  dices/Appendix11App2.html

7
BACKGROUND
First Online Evaluations

• Instructor initiated
• Campus-wide Teaching Evaluation Form (CTEF)
• Secure and anonymous
• Fixed one time email announcement
• Automated analysis of results

8
BACKGROUND
Online evaluations had initially been driven by
faculty requests. With broader interest, the
needs of entire Schools or Departments became the
new driving force behind the direction of the
online evaluation tools evolution.
9
BACKGROUND

• Engineering was the first school to use EEEs
  tool school-wide
• Entire school switched to online Final
  Evaluations
• Simple Administrative Web page was developed
• Schools evaluations fully online by Fall 2003
• Learned from experience and incorporated feedback
  into version 2

10
CHALLENGES

• Significant Culture change
• Concern of response rate decrease
• Students would become responsible for taking and
  submitting evaluations on own time and outside of
  class
• Duration changed from 15 minutes inside the
  classroom to what individual students chose on
  their own
• Instructors would need to diligently notify and
  remind students in order to receive desirable
  submissions
• Departments would need to communicate to faculty
  and/or directly enrolled students
• ENCOURAGEMENT COMMUNICATION BECAME VITAL

11
CHALLENGES

• Evaluation requirements
• CTEF format
• Anonymity
• Department Customizability
• Cooperation from academic units
• Willingness to take part in a potentially
  difficult transition
• Strong history of good support and cooperation
  helped convince Departments that their needs
  would be met, and any unexpected issues would be
  addressed
• Instructor Student participation
• Notification of students encouragement to
  participate required for best results

12
TRANSITION PROCESS OVERVIEW

• TOOL FEATURES
• ACTIVATION
• ANNOUNCEMENT
• MONITORING
• RESULTS
• WEB INTERFACE
• STUDENT EXPERIENCE

13
TRANSITION
Tool Features

• Departments steered development with requests
• Optional addition of four custom questions
• Scantron-like results file
• Printable report with page breaks
• Response rate monitoring tools
• More department control over instructor options
• Email reminders for multiple evaluations combined
  into one
• Only non-respondents receive reminders

14
TRANSITION

• Activation
• Administrators
• Create and upload CSV files containing instructor
  IDs and course codes
• Specify the bounds of instructor control over
  evaluation windows

15
TRANSITION

• Announcement
• Administrators compose, schedule, and send email
  announcements and/or reminders using a simple
  online form through the Web interface
• No limit to the number of announcements sent
• Only non-respondents do receive reminders
• Reminder emails can have different subject-line
  or body each time

16
TRANSITION

• Monitoring
• Administrators monitor rate of evaluation
  submissions
• Constant monitoring helps department
  administrators decide when to send announcements
  or reminder emails

17
TRANSITION

• Results
• Speed
• Administrators, evaluated instructors, and
  teaching assistants view and download results
  files in desired format(s) after the Registrars
  Final Grade Submission Deadline.
• Flexible Formats
• Comma Separated Value (CSV)
• Print-friendly report
• Scantron compatible

18
TRANSITION

• Error check
• Prevents user errors from adversely affecting the
  process
• Instructor abilities
• Preserves option to allow or disallow instructors
  from changing evaluation dates/times, or
  canceling evaluations
• Retains midterm evaluation options for all
  instructors, regardless of Department-activated
  final evaluations

The D.A.T.E. Web Interface makes Department
activation of final evaluations for multiple
instructors easy and efficient.
19
TRANSITION

• Online form
• Requires secure authentication
• Available anywhere, at any time during the
  evaluation window
• Quick and easy multiple choice questions
• Provides space for typed open-ended responses
• Incentives
• Respondent List makes incentives possible
• Anonymity preserved
• Names cannot be associated with responses
• List only available if 5 or more responses
  received (minimum increasable at department
  discretion)

20
ADVANTAGES RESULTS

• Save Time 90 savings common
• Save Money 2000/yr for Scantron (PhySci)
• Add Customizability Add 4 custom questions
• Unexpected side-effect
• With online evaluations, students have begun to
  submit more thorough and detailed comments,
  presumably because they are under no time
  pressure, and are able to type, rather than
  handwrite, their comments.
• Because comments are already typed, they can be
  distributed and read more easily and quickly.
• Results are compiled and ready for distribution
  and review in matter of hours rather than months.
  
• Faculty are enabled to powerfully and immediately
  reflect the evaluation results to the upcoming
  class.
• Some departments more rapidly are able to renew
  or take away TAs or lecturers appointments.
• DEFINITE IMPACT ON QUALITY OF TEACHING

21
ADVANTAGES RESULTS

• EEEs Evaluations tool is the fastest growing
  tool in the EEE course management system toolbox

22
FACTS FIGURES
1 2 3 4 5

1. SurveyMonkey
2. QuestBack
3. Zoomerang
4. Paper Evaluations
5. D.A.T.E.

Supported
Unsupported
23
FACTS FIGURES
24
FACTS FIGURES
Submissions
Engineering
Physical Sciences
Biological Sciences
Classes
25
FACTS FIGURES
Projected Department participation for Fall,
2005
26
FACTS FIGURES
Numbers in parentheses represent percentage of
time saved per department
27
FUTURE

• More customizability
• New survey engine will power tools
• Evaluation tool
• Survey tool
• Quiz tool
• More mobility
• Ability to share Evaluation forms with others
• Re-use frequently used questions answers

28
CONCLUSION

• D.A.T.E.s strength and key success come from the
  solid partnership our team has created and
  maintained with a wide range of senior academic
  faculty, committees, and campus units.

29
QUESTIONS?
http//eee.uci.edu/iwt/uccsc2005/
For questions, please contact
Shohreh Bozorgmehri shohreh_at_uci.edu Briandy Walden bwalden_at_uci.edu
or eee_at_uci.edu