An Online Course Assessment Tool (OCAT) for Faculty Peer-assessment or Self-assessment Speaker: Claire DeCristofaro, MD Assoc Prof, WCU Nursing, MS(N) Program eLearning Faculty Fellow, Coulter Faculty Center Western Carolina University, Cullowhee,

1
An Online Course Assessment Tool (OCAT)for
Faculty Peer-assessment or Self-assessment
Speaker Claire DeCristofaro, MDAssoc Prof,
WCU Nursing, MS(N) Program eLearning Faculty
Fellow, Coulter Faculty CenterWestern Carolina
University, Cullowhee, NC Sloan-C
ConferenceNovember 2007
2
Coulter Faculty Center eLearning Faculty
Fellowsat Western Carolina University

• eLearning Faculty Fellows
• John LeBaron (Distinguished Professor,
  Education)
• Claire DeCristofaro
• (Assoc Prof, Nursing)
• Dixie McGinty
• (Assoc Prof, Education)
• Mary Teslow
• (Asst Prof, Health Sciences)

3
Why perform peer assessment?

• Provide constructive feedback regarding teaching
  effectiveness
• Stimulate instructional improvement
• Promote faculty development
• Provide opportunities for peer support

4
Online Course Assessment Tools

• Open source resources rubrics
• Universities (course reviews)
• Quality Matters, Sloan Consortium
• Different formats, priorities, methods
• Our tool
• Blends recurring themes
• Includes narrative summaries
• Accommodates varied disciplines
• Accommodates varied student cohorts
• Includes instructor feedback

5
OCAT Reliability Testing

• IRB-approved research project to determine
  inter-rater reliability
• Tool used to assess existing courses drawn from
  multiple disciplines
• Courses used for reliability testing selected
  because instructors are experienced online
  teachers or instructors have an interest in
  online teaching
• Reliability project included peer assessors drawn
  from faculty across disciplines and from
  instructional design staff at the faculty center

6
Peer Assessors

• Will have been trained in the use of the tool
• Will honor the confidential nature of this
  activity
• Will maintain confidentiality of results

7
Overview of the Tool

• Interactive PDF document
• Process Data
• First five sections cover specifed domains
• Course instructor assessment
• Each domain includes assessment of
• Elements of course design
• Elements of teaching
• Last two sections
• Peer Assessor summary narratives
• Instructor response to assessment process
  assessment results

8
Process Data

• Instructor (rank department)
• Course (code, section, term)
• Peer Assessor (rank, department)
• Format of CMS
• Dates request, assessment, completion
• Number of students in attendance

9
Performance Levels for Rated Elements

• Scoring for listed elements
• Exemplary exceeds the element
• Standard meets the element
• Developing falls beneath the element statement
• Not Applicable not relevant to the course
• Includes Narrative Commentary boxes for each
  section
• Getting Dinged ??

10
Course Instructor Domains

• Course Overview Organization
• Learner Objectives Competencies
• Resources Materials
• Learner Interaction
• Learner Assessment

11
Final Narrative Summaries

• 6. Peer Assessor narrative commentary
• Additional comments on strengths of course
• Additional comments regarding recommendations for
  improvement
• Recommendations for improvement
• 7. Instructor response narrative commentary
• Suggestions for altering peer assessment process
• Suggestions for revising the peer assessment
  process

12
Confidential Peer Assessment Process (1)

• Faculty requests peer assessment of online course
  through the CFC
• CFC Director arranges peer assessment (selects
  peer assessor)
• Trained peer assessors meet with instructor
  before beginning assessment

13
Confidential Peer Assessment Process (2)

• Peer assessor completes sections 1- 6 (peer
  assessor sections)
• Peer assessor meets again with instructor and
  section 7 is completed (instructor feedback
  section)
• Final, completed document is sent to instructor
  and also kept in confidential file at the faculty
  center

14
Use of OCAT for faculty self-assessment

• The peer assessment instrument itself will also
  be available online for faculty
• The tool can be used for self-assessment as a
  faculty development tool
• Additionally, the tool can be used as a guide
  for course development, especially for faculty
  new to the online teaching environment

15
Faculty Feedback on OCAT

• OCAT has been presented to several university
  workshops
• Faculty have had the opportunity to
• use the tool to evaluate their own courses
• use the tool in evaluating courses volunteered by
  experienced online teachers
• Faculty response overall positive tool
  provides
• a guide to effective online teaching practices
• a template for course development
• a structured guide for ongoing improvement of
  existing courses for serial re-evaluations over
  time

16
Domain 1Course Overview and Organization
17
A. Elements of Course Design

• Navigational instructions make the organization
  of the course easy to understand and are
  transparent at the beginning of course
• Course introduction that includes guidance on the
  structure of the course is provided
• Course design demonstrates user-friendly
  presentation style
• Minimum student technology competencies/skills
  for the course are stated, if applicable
• Course-specific technology requirements are
  stated, if applicable
• Description and method of accessing the
  institutions academic support system is defined
• The institutions support services and resources
  (e.g. library) are described and linked
  appropriately
• Course instructions articulate or link to
  tutorials and resources that answer basic
  questions related to research, writing,
  technology, etc.
• Course design item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

18
B. Elements of Teaching

• The instructor
• provides an introduction and provides credentials
  related to this course
• provides the opportunity for student introduction
• provides the opportunity to practice and master
  the technologies needed for the course
• assures that learners are ready to undertake
  assigned tasks
• states the broader scholarly/practical context
  for learning objectives/goals
• Instructional item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

19
Domain 2Learner Objectives and Competencies
20
A. Elements of Course Design

• Learning Objectives/Goals
• are stated for each unit/module
• are clear and easy to understnd
• describe outcomes that are assessable
• address content mastery
• address critical thinking skills
• address core learning skills
• Course design item(s) emerging from peer
  discussion not included in the list above (write
  in box below)

21
B. Elements of Teaching

• The instructor
• provides information to students on how to meet
  the learning objectives/goals
• explains how to meet the learning
  objectives/goals in a manner that is clear and
  easy to understand
• fosters integration with prior learning, if
  applicable
• Instructional item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

22
Domain 3Resources Materials
23
A. Elements of Course Design

• Resources and Materials
• address learning objectives/goals (e.g. course
  reference materials, glossaries, library
  resources, relevant web resources,
  discipline-specific resources
• are appropriatly authenticated and cited
• are current and/or timely
• are prepared by qualified sources
• are presented in a format appropriate to the
  online environment
• are easily accessible to the student
• state the purpose related to the course (e.g.
  core material, supplemental, historical,
  required/optional)
• have active links to electronic resources
• include the use of electronic library scholarly
  resources, if applicable
• address diverse learning styles (e.g.
  Multimedia, graphics, audio, video, text chats,
  voice chats
• Course design item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

24
B. Elements of Teaching

• The instructor
• Demonstrates knowledge of the discipline
• Incorporates strategies that promote the
  progressive learner construction of knowledge
  (e.g. use of units/modules that are thematically
  connected and have a logical progression)
• Encourages students to independently augment the
  course resource base through independent or
  collective research, if applicable
• Instructional item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

25
Domain 4Learner Interaction
26
A. Elements of Course Design

• The course design prompts the instructor to be
  present, active and engaged with the students
  (e.g. provides tools necessary)
• Netiquette instructions/recommendations provided
  regarding emails and discussion postings
• Guidelines for collaboration are defined, if
  applicable
• Guidelines for communication/accountability are
  defined
• Course design item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

27
B. Elements of Teaching

• The instructor
• Is aware of student progress (e.g. assigned
  tasks) and any barriers to progression
• Fosters interaction among constituencies inside
  and outside the course as appropriate (e.g.
  student-student, student-instructor, and with
  external persons or agencies
• Embeds learning activities (student-content
  interaction) that promote the achievement of
  stated objectives and learning outcomes
• Selects communication strategies to communicate
  appropriately with the class as a whole, student
  groups, and individuals
• Respects student privacy as necessary
• Promotes independent and shared student research,
  if applicable
• Provides content-based interaction as needed
• Provides timely responses to student queries
• Offers announcements tailored to the progress of
  the class
• Models a communication style that demonstrates a
  positive tone
• Instructional item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

28
Domain 5Learner Assessment
29
A. Elements of Course Design

• Assessments are frequent enough to provide
  formative feedback
• Rubrics are provided to define assessment
  criteria, if applicable
• Methods of submitted assignments are appropriate
  to the online learning environment
• Course design item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

30
B. Elements of Teaching

• The instructor
• Clearly describes assignments (student
  discussion, participation, and projects)
• Provides formative assessment/progress feedback
• Provides summative assessment feedback
• Provides timely assessment feedback
• Instructional item(s) emerging from peer
  discussion not included in the list above (type
  in box below)

31
Section 6Peer Assessor Commentary Final
Narrative Summaries
1. Additional comments regarding the strengths
of this course and teaching (type in box
below)2. Additional comments regarding
recommendations for improvement (type in box
below)
32
Section 7Instructor Response Final Narrative
Summaries

• Instructor response to the results of this
  assessment (type in box below)
• Suggestions for revising the peer assessment
  process (type in box below)

33
The Dynamic OCAT

• Current length of document in order to allow
  extensive narrative summaries in each section
• Modifications planned based on implementation and
  feedback
• modify assessment process data collection
• modify OCAT itself

34
Thank youComments? Questions?You are also
welcome to emaildecristofaro_at_email.wcu.edu