Moving from Periphery to Priority: Peer Review of Teaching1

1
Moving from Periphery to Priority Peer Review of
Teaching-1

• Nancy Chism
• IUPUI
• POD Conference, 2007

2
Why This Session?

• A very important yet neglected topic
• Developers are especially equipped to help with
  this work
• You are the person on your campus
• Im getting tired

3
Our Goals

• Following this session, I hope that you will
• Have a working definition of peer review
• Have expanded your idea of what peer review
  encompasses
• Be able to cite reasons for the importance of
  peer review as well as resistance to it
• Understand the importance of aligning criteria,
  evidence and standards
• Possess new ideas and resources for doing work in
  this area

4
Format

• Workshops within a workshop
• Workshop models
• How I facilitate work on peer review of teaching
  (blue, brown, maroon)
• Our workshop
• How you can design and facilitate discussions of
  peer review (green)

5
Models

• Consultation with committees working on peer
  review systems
• Workshop to introduce the idea (blue)
• Workshop to outline design options (brown)
• Workshop to prepare reviewers (maroon)

6
I. Workshop to Introduce the IdeaCommon core for
other workshops (abbreviated or not)

• Facilitate discussion of basic concepts
• Engage participants to expand upon ideas
  presented
• Help participants try the ideas out for size in
  their own environment
• Encourage action planning for the future

7
Introduction Components

• Definition of peer review of teaching
• Expansion of concept of peer review
• Reasons why peer review is important
• Acknowledgement of challenges to peer review
• Examples of formative and summative peer review
  of teaching

8
Illustration-Introductory Workshop

• Definition of Peer Review

9
Effective Peer Review of Teaching

• Informed peer judgment about faculty teaching for
  either fostering the improvement of the people
  being reviewed (formative) or making personnel
  decisions in their cases (summative)

10
Effective Peer Review of Teaching

• Informed peer judgment about faculty teaching for
  either fostering the improvement of the people
  being reviewed (formative) or making personnel
  decisions in their cases (summative)

11
Elements of Effective Peer Review of Teaching

• Systematic process
• Peer
• Judgment
• Feedback/Documentation

12
Illustration Introductory Workshop

• Expansion of Concept

13
Peer Review Can Involve

• Different Purposes (formative and summative)
• Different Methods (observation, analysis of
  documents, dialogue, journaling)
• Varieties of Teaching Roles (advising, teaching
  scholarship and leadership, classroom/
  clinical/studio teaching)
• Levels of review (evidence from person being
  reviewed, students, other peers)

14
Illustration Introductory Workshop

• Rationale for Peer Review of Teaching

15
Value of Peer Review of Teaching

• For the Individual
• Promotes growth in teaching
• Is reciprocal in nature both reviewer and person
  being reviewed can benefit
• For the Academic Unit
• Creates community of practice
• Increases group awareness about curriculum
• Increases value of teaching

16
Increased Value of Teaching

• Shulmans idea of Community Property emphasizes
  that teaching will not be rewarded adequately
  until it is open to public inspection
• Becomes more like research
• Standards of excellence shared
• Peer review affirms quality
• Shulman, L. (1993). Teaching as community
  property Putting an end to pedagogical solitude.
  Change, 25 (6), 6-7.

17
Rationale for Peer Review of Teaching

• Only peers can judge some aspects of teaching
  effectiveness, such as currency and accuracy of
  information
• Inherent in the notion of professional is the
  responsibility for monitoring of standards by
  peers

18
Arguments for Peer Reviewof Teaching

• Acknowledges appropriate complexity of teaching
• Multiple sources and multiple kinds of
  information are needed for sound evaluation
• Responds to pressure to implement from
  accrediting organizations

19
Illustration Introductory Workshop

• Reservations about Peer Review of Teaching

20
Reservations about Peer Review

• Time commitment
• Norms of privacy and egalitarianism of teachers
• Reduction of risk-taking in teaching
• Problems in defining a peer
• Lack of clarity about standards

21
Reservations about Peer Review (2)

• Concerns about skills of reviewers
• Possibility that interpersonal or
  intradepartmental conflicts might contaminate
  process
• Legal and interpersonal vulnerability of reviewer

22
How Can We Address Reservations?

• Establish system that
• Is clearly understood
• Is consistent
• Is practical
• Provides reviewer preparation
• Is embedded in larger culture of dialogue about
  teaching and reward for teaching

23
Illustration Introductory Workshop

• Examples of Peer Review
• Handout 1

24
Some Examples of Formative Use

• Peer mentor programs
• Teaching circles or faculty learning communities
• Team teaching
• Collaborative research on teaching
• Materials library

25
Examples of Summative Peer Review of Teaching

• Hiring Decisions-Pedagogical Colloquia
• Annual Reviews and Reappointments
• Promotion and Tenure
• Merit and Awards Decisions
• Post-tenure Review

26
National Examples of Formative Peer Review

• New Jersey Master Teacher Program
• Peer Review of Teaching Portfolio Project
• Example at Texas AM http//cte.tamu.edu/programs
  /prp.php
• Former AAHE Project (materials from Stylus)
• Participating schools Alverno, Georgetown,
  IUPUI, Kent State, Northwestern Stanford,
  Syracuse, Temple, UCSC, Georgia, Michigan,
  Nebraska, UNC-Charlotte, Wisconsin, Wyoming,
  Xavier (La.)

27
National Examples of Summative Peer Review Systems

• Usually department or school specific
• Examples of General Campus Guidelines
• University of North Carolina http//www.ncsu.edu/p
  rovost/peer_review/references.html
• University of Wisconsin-Madison-Delineation of
  peer review of teaching policy http//www.provost.
  wisc.edu/archives/ccae/MOO/

28
Stopping Point 1

• Looking back over the slides for this workshop
• What other advantages of peer review might you
  add?
• What other reservations?
• How would you structure this information
  differently?
• How would you engage the participants during this
  workshop?

29
Moving from Periphery to Priority Peer Review of
Teaching-2

• Nancy Chism
• IUPUI
• POD Conference, 2007

30
II. Workshop to Outline Design Options

• Offered to people who will serve on design teams
  or administrators who would like to support work
  on peer review
• Engage participants in thinking through design
  decisions and policy options
• Accomplish only initial engagement and require
  follow up at local level

31
Illustration Design Workshop

• Building Blocks for the System
• Handout 2, also see tables in Ch. 3 of book

32
The Basis for the System

• Criteria aspects of teaching to be evaluated
• Evidence artifacts that can be examined for
  indications of the quality of performance on each
  aspect
• Standards differences between levels of
  performance

33
Importance of Foundation

• Articulation of criteria, evidence, and standards
  are basic to conversations about both improvement
  and personnel decisions.
• Systems must be aligned.
• Culture of teaching must be established and
  strengthened.

34
Criteria What Is Important?

• Categories
• Examples advising, lecturing, designing courses,
  leadership for teaching
• Characteristics
• Examples (for lecturing) enthusiasm, clarity,
  organization, interpersonal rapport, command of
  content
• High vs. low inference judgments

35
Some Common Categories

• Subject matter knowledge
• Instructional design
• Presentation
• Student engagement
• Assessment
• Mentoring of others
• Leadership on teaching
• Scholarship of teaching and learning

36
What Criteria Are Important to You?

• Lets try listing a criterion on a planning grid
• What is the main category?
• What are some characteristics of desired
  performance in this area?

37
Peer Evidence Where Should We Look?

• Kinds of Evidence
• Classroom/Clinic/Studio Observation Record
• Instructional Materials
• Instructors Records
• Records Provided by Others

38
Classroom Observation Record

• Narrative notes
• Checklist
• Summary letter or report
• Videotape or virtual classroom downloads

39
Instructional Materials

• Syllabi
• Course packets, bibliographies, textbooks
• Multimedia course materials
• Course handouts
• Tests and student assignments
• Graded student work

40
Instructors Record

• Report of scholarship of teaching efforts (as
  well as printed products, if any)
• Report of contributions to teaching quality
  within the department
• List of courses taught and details
• List of nonclassroom teaching activities
• Portfolio or reflections on teaching growth
• Teaching philosophy statement

41
Records Provided by Others

• Student Products
• Student evaluations of teaching
• Administrative reviews
• Alumni satisfaction data
• Student tracking data
• Conference evaluations of presentations on
  teaching topics
• Reviews of publications on teaching topics

42
Evidence Back to the Grid

• What are some appropriate sources of evidence?
• Choose one criterion and think of evidence for
  judging performance on that criterion

43
Standards

• What levels of performance are outstanding,
  meritorious (excellent, effective in your
  system)?
• Choose one criterion and think of distinguishing
  characteristics for each level

44
Illustration Policy Issues
45
Designing a Peer Review of Teaching Policy-1

• Gathering the development team
• Deciding at what level the plan will be used
  (program, dept., school/college, institution),
  how much variation is allowed
• Defining purpose (balance between formative and
  summative)

46
Designing a Peer Review of Teaching System-2

• Deciding population and timeline who will get
  reviewed and when? Who will do the reviews?
• Negotiating and communicating plan How will the
  plan be introduced and reinforced?
• Providing resources How will reviewers be
  prepared for this work? What recognition will
  they receive for doing it?
• Revising periodically What mechanism will be in
  place for revision?

47
Defining Who Should Be Reviewed

• Everyone who teaches?
• All probationary faculty?
• Only tenure-track faculty?
• Only tenure-track faculty eligible for PT?

48
Defining When and How Often Reviews Should Occur

• Schedules dependent on purpose?
• (Mentoring/coaching vs. personnel decisions)
• Schedules dependent on type of faculty
  appointment?
• Schedules dependent on stage of faculty career?

49
Defining Who Does the Reviews

• Special content knowledge peer?
• At higher rank?
• Teaching methods peer?
• Teams of peers?
• Inside or outside program/department?
• Everybody but you?
• Multiple reviewers and points in time are
  essential for reliability

50
Defining What Aspects and Format to Use in Reviews

• Matched with criteria and standards in grid
• Forms for peer reviewers to use in assessing
  classroom performance, classroom materials, etc.
• Samples in Peer Review of Teaching
• Based on consensus within the academic
  unit/campus on criteria and characteristics

51
Defining How Results Should be Shared

• In a meeting?
• Through written communication (with or without
  response)?
• Document for private communication or inclusion
  in dossier?
• Prompt, personal feedback is necessary for
  development purposes standard reporting format
  for personnel purposes

52
Choices for Negotiating and Communicating the Plan

• Group brainstorming?
• Draft to faculty for revision?
• Administrative edict?
• Allow choices within plan (do either A or B)
• Insert in formal policy document such as pattern
  of administration

53
Integrated System The Eportfolio

• Professional development planning
• Goal setting and tracking
• Recording of accomplishments
• Attachment of evidence
• Reflective practice
• Commentary on practice
• Formative and summative peer review
• Review of performance

54
Eportfolio Systems

• Consist of a database with friendly user
  interfaces
• Are the main storage vehicle for information
  about faculty members careers
• Are able to retrieve and display information in
  various formats (for annual review, promotion and
  tenure dossiers, awards nominations)

55
Illustration Summary
56
What Does a System of Peer Review Offer an
Administrator?

• Way to promote culture of teaching and learning
• Talk about teaching and learning
• Socialization of new faculty resocialization of
  senior faculty
• Shared responsibility for support
• Way to articulate model of effective teaching as
  goal for faculty
• Way to provide support for personnel decisions

57
Elements of a Good System

• Is based on faculty input about what constitutes
  good teaching on this campus
• Is specific and realistic
• Is supported by administration and faculty
• Proposes instruments or other resources
• Offers preparation for both reviewers and those
  being reviewed

58
Elements of a Good System (2)

• Results in a review that is fair in both content
  and process
• Uses methods that reveal the philosophy and
  thinking that forms the foundation for the
  faculty member's teaching
• Calls for gathering information on more than
  classroom teaching
• Is updated periodically

59
Why Peer Review Sometimes Fails

• No plan, poor plan, overambitious plan
• Lack of buy-in by faculty and/or administrators
• Not integrated into other performance systems
  (PT, annual report, awards)
• Lack of model for reviewers
• History of poorly-done reviews

60
How to Help Design a Good System

• Participateboth faculty and administrators
• Make sure system focuses on fullness of teaching
• Provide for opportunities for faculty to
  understand how to use the system well
• Understand that it will not be perfect but aim
  high

61
Stopping Point 2Workshop on Systems Design

• How can participants be helped in reaching some
  consensus on criteria?
• How can one facilitate work on standards?
• How can the process decision questions be
  facilitated?

62
Moving from Periphery to Priority Peer Review of
Teaching-3

• Nancy Chism
• IUPUI
• POD Conference, 2007

63
III. Workshop to Prepare Reviewers

• Intended for sites ready to implement a system
• Can also be used to demystify the process to
  faculty
• Model process with local materials or my materials

64
Illustration Workshop for Preparing Reviewers

• Trying a Case
• Handouts 3-8 Syllabus, Statement, Forms

65
Lets Try a Case Chem 101

• Classroom Observation
• Review of Syllabus
• A Look at Instructors Comments

66
Steps in Classroom Observation Preobservation
Exchange

• Establishes context
• Nature of course and background
• Typicality of students, lesson
• Unearths teachers concerns
• What particular focus should be taken?
  (especially important in formative reviews)
• How should the observer act during class?

67
Those Being Reviewed Should

• Tell the reviewer what they most want to know
  (especially for formative review)
• Articulate to the reviewer assumptions about the
  teaching activities under review
• Provide the reviewer with explanatory documents
  or verbal explanations

68
Considerations during Observation

• Unobtrusiveness
• Where should observer sit?
• Should observer be introduced?
• Method of recording information
• Notes
• Checklists or rating forms
• Diagrams or coding systems

69
Using Review Forms

• Checklists
• Rating Forms
• Narrative Response Forms
• Which Do You Prefer?

70
Postobservation

• Schedule promptly, if in person, or complete
  written feedback quickly
• Provide opportunity for self-evaluation in
  formative feedback sessions
• Focus the conversation on the thinking behind the
  teaching as well as observed behaviors

71
Now for Materials

• Read the syllabus for Chem 101
• How do you respond to it
• As a document (does it meet your expectations for
  a good syllabus?)
• As a course (does it sound like it is your idea
  of a good course?)

72
Using a Checklist

• Use the checklist to record your judgments about
  the course
• Did it add anything to your first analysis?
• Do you think that the use of a checklist such as
  this would increase consistency across peers?
• How would you change the checklist to make it
  more appropriate for your campus?

73
Another Layer

• Read the instructors commentary on the Chem 101
• Does it help you to do a better job of peer
  review? If so, how?

74
More Layers

• Instructor can also provide
• Verbal or print information on course context
  (nature of students, fit in curriculum, etc.)
• Vignette of classroom experience (writing about
  incident, taped or live observation opportunity)
• Examples of other syllabi in the field

75
Yet More Layers

• Instructor can also provide
• Samples of student work
• Student comments or ratings
• Other course materials
• Handouts
• Graded papers
• Notes on communications with students

76
Illustration Workshop on Preparing Reviewers

• Summary Advice
• Handout 9

77
Preparing for Peer Review

• As Reviewer
• Learn how to use unit processes
• Review evidence carefully with standards and
  methods articulated
• As Person under Review
• Self-initiate when necessary
• Use reflection in documenting and selecting
  evidence

78
Responding Helpfully

• Prepare yourself to care
• Put yourself in the others shoes
• Be reflective and systematic as a reviewer
• (Dont shoot from the hip)
• Embed your comments in data from the course
• Refer to specific instances

79
Giving Formative Feedback

• Use descriptive language rather than evaluative
  as much as possible
• Be responsive to what peer has listed as own
  concerns about teaching
• Check to make sure peer has heard your message in
  the way you intended
• Be forward-looking in your thrustwhat can be
  improved over what went wrong

80
Receiving Peer Feedback

• Make sure you take advantage of opportunities for
  feedback
• Try to avoid defensive reactions by considering
  feedback within the context of inquiry about
  teaching
• Check your understanding of what is being said

81
Receiving Peer Feedback (2)

• Enlist the reviewers help in thinking about
  steps to take toward improvement
• Record your thoughts about the review for future
  use and reflection
• Aim for reciprocal benefitshow can this exchange
  help the reviewer as well?
• Aim for system changeshow can the peer review
  process at your campus be improved?

82
For Summative Purposes

• Make sure that you complete the standard forms
  promptly and completely
• Cite concrete examples when possible
• Keep your notes in case there are questions later
• Suggest revisions to the process if you think
  there are shortcomings

83
Final Stopping PointReviewer Preparation Workshop

• Can other artifacts/documents be useful in
  portraying the person under review?
• How close a context match should be made with the
  person portrayed?
• How can the facilitator keep this positive and
  constructive?
• Should participants construct summary advice?

84
Giving Away the X-rays

• Please use as helpful
• Make a careful diagnosis before applying
• Do no harm