NCATE Professional Development WebConference Series for Program Review

1
NCATE Professional Development
Web-Conference Seriesfor Program Review

• Program Reviewer General Session
• Process and Guidelines for the Spring 2007 Review

Deidre Alves M.Ed. March 7, 2007
2
Welcome Reviewers!

• Please remember to set-up your audio
• 145pm- First audio participant mic
  sound check
• 155pm- Final audio participant mic
  sound check
• 200pm- Session begins

3
Session Agenda

• PART A---Intro System Operations
• 1. Meet the participants
• 2. Becoming an active participant at todays
  conference
• Intro to Eluminate web-conferencing product
  features
• 2. Conference Goals
• 3. System Overview
• Current System
• PRS
• 4. Live Demo of the systems
• Current System
• PRS

4

• PART B---Team Operations
• 5. The Recognition Report Template
• 6. Effective Writing Strategies
• 7. Reviewer Attitude Tone
• 8. Program and Unit Relationship
• 9. Communication with Your Team
• 10. Timelines and the rolling review
• 11. Q A
• 12. Reflection
• 13. Conclusion

5

• Meet the Participants

6
Anna Devito- NASPE
7
Sharon Cramer-CEC
8
Todd Curless- AECT
9
Sumita Bhattacharyya-NSTA
10
Kathryn Davis-NASPE
11
Mike Hall-NCTM
12
Sally Sentner-ACEI
13
Introduction to Features on Elluminate

• Audio check
• Polling feature
• Synchronous text message feature
• On-air feature

14
Becoming an Active Participant at Todays
Conference

• Synchronous text message feature
• On-air feature
• Applause feature
• Polling feature

15
Interactive Icons
16
(No Transcript)
17
Conference Goals

• to provide information on how to evaluate and
  submit final recognition reports using 2
  interface systems (Current PRS)
• to provide reviewers with effective writing
  strategies and examples for use in completing the
  final recognition report
• to provide professional development training to
  serve the needs of program reviewers
• to provide a forum to enhance collaboration among
  NCATE/SPA reviewers

18

• Part A
• System Operations

19
System Overview

The system interface allows reviewers to access
program reports and post final recognition
decisions This semester NCATE will be utilizing
2 systems Current PRS
20

• NCATE launches the new system interface
• -(PRS)-
• Spring 2007

21

• PRS
• Program Review System

22

• PRS URL
• http//prs.ncate.org

23

• Remember that this semester
• NCATE will utilize both the current system and
  the PRS

24

• This will be the LAST semester utilizing our
  current system as PRS becomes the official
  interface for all reviews beginning Fall 2007

25

• Regardless of which system you will use (current
  or PRS), you will first need to access the
  password protected site and log-in
• URL
• ID
• PASSWORD

26
URL

• URL (current system)
• http//notes.ncate.org/progrev/progreports07.nsf
• URL (PRS)
• http//prs.ncate.org

27

• ID and PASSWORD
• Sent via email

28

• Once on the site (current or PRS)
• You will see your name and program assignments

29

• The log-in screen for the current system will
  look like this

30
(No Transcript)
31

• The log-in screen for the PRS will look like
  this

32

• The log-in screen for the PRS will look like
  this

33
(No Transcript)
34

• After log-in on the current system you will see
  your assignments listed as follows

35
(No Transcript)
36
(No Transcript)
37

• On the PRS, it will look like this

38
(No Transcript)
39

• Advantages of PRS
• On-line application
• Field sensitive
• User-friendly
• Character counts
• Streamlined
• Feedback email loop

40

• Simple steps PRS or Current
• Access Program Report from the institution
• Access SPA Program Recognition Report template
• Current System (save template to desktop)
• PRS (template is retrieved by clicking on the
  pen icon under reviewer for the assignment-and
  is entered entirely on-line (content is exactly
  the same)
• Submit the completed SPA Program Recognition
  Report
• Current system ---attach
• PRS--- simply click submit

41

• All reviewers on the team must submit individual
  reports as well as contribute to the final report
• Lead reviewers know to complete a final report
  which brings together the findings of all team
  members

42

43

• Resources for reviewers
• www.ncate.org (Program Reviewers)
• NCATE Guidelines for Reviewers
• SPA Template
• SPA Worksheets (where applicable)
• Archived Webconferences

44

• Program Report
• vs.
• Program Recognition Report

45
Live Demo
We are going to walk you through the
process of accessing the Program Report and
submitting the final Program Recognition Report
on both systems.
46

• We will first walk you through the process using
  our current system

47

• Then, we will walk you through the
• process using the new PRS interface

48

• Lets go to the current system

49

• Some items to note

50

• All reviewers on the team should complete
  individual reports and contribute to the final
  report (lead submits final)

51

• If the report requires revision by the
  institution in the next semester, the same team
  is assigned

52

• If assigned a revised report
• The Program Report and SPA Recognition Report
  have been made available at

53

• URL (old archived reports)
• http//notes.ncate.org/progrev/progreportf05.nsf
• -or-
• http//notes.ncate.org/progrev/progreports05.nsf
• -or-
• http//notes.ncate.org/progrev/progreports06.nsf
• -or-
• http//notes.ncate.org/progrev/progreportf06.nsf

54

• Any questions up to this point

55

• Part B
• Team Operations

56

• We have talked about the PROCESS of accessing the
  Program Report and Program Recognition Report---
• Lets talk about the CONTENT of the Program
  Recognition Report

57
The SPA Program Recognition Report Template

• Templates available at www.ncate.org
• Click program reviewers then resources for
  reviewing
• Template format is consistent across SPAs

58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61

• Examples of well-written Summary of Strength
  statements
• emphasis on reflective practitioner throughout
  education coursework
• use of an action research project that focuses
  candidates on their effect on student learning
• beginnings of a comprehensive program
  assessment system that when refined should
  provide useful, current information on candidate
  success for improving the program and tracking
  candidate progress

62

• Characteristics of a well written Summary of
  Strength statement
• Evidence based
• Clear
• Objective
• Finds a strength even in a weak program

63
(No Transcript)
64

• Met- comment not required
• Not Met- comment is required

65

• Examples of well-written comments for an unmet
  standard
• Standard 2.4. Candidate use practices designed
  to assist students in developing habits of
  critical thinking and judgment. NM
• The assessment cited for this standard has
  insufficient specific reference to critical
  thinking and judgment rather it is focused on
  planning lessons that improve student
  understanding through the use of various reading
  processes.

66

• There are no references to media or technologies
  in cited assessments. This standard does not
  focus on texts.
• The program cites assessment 5, the rubric and
  data table based on the state assessment, as
  providing evidence that this standard has been
  met. However, reviewers found no mention of
  building family and community relationships in
  this assessment tool. Also, reviewers found no
  scoring guides or rubrics in the information
  provided to help understand how a candidate is
  rated as distinguished, proficient or
  developing. Again, there was no information
  specific to the relationship of the state
  assessment and the NAEYC standards, including
  standard 2

67

• Why was this comment well written?
• CLEAR
• OBJECTIVE STATEMENTS
• RELATES DIRECTLY TO THE STANDARD
• DIRECT

68
(No Transcript)
69

• An example of a well-written Part C.1-
  Candidates knowledge of content
• The institution indicated assessments to
  address areas of content knowledge essential for
  elementary teachers. However, evidence from
  Assessment 2 was not available other
  assessments provided only limited data for some
  areas of content knowledge (i.e., science and
  social studies) and no evidence was available to
  indicate knowledge of the arts, health education,
  and physical education.

70

• An example of a well-written Part C.2-
  Candidates ability to understand and apply
  pedagogical and professional content knowledge,
  skills, and dispositions
• The institution indicated Assessments 3, 4,
  and 5 address ACEIs standards applicable to
  pedagogical knowledge and skills. However, the
  elements of the scoring rubrics and the candidate
  data presented only indirectly or narrowly
  addressed most of these standards. It was
  difficult for reviewers to see relationships from
  the information and candidate data presented that
  was clearly directed toward and aligned with the
  applicable standards.

71

• An example of a well written Part C.3- Candidate
  effects on P-12 student learning

There was no evidence to indicate candidates
collect data on students for the purpose of
subsequent specific planning and implementation
of instruction.
72

• More Examples
• C.1Candidates knowledge of content
• The Evidence of Meeting Standards chart is
  particularly helpful in making connections
  between the assessments and the standards.
• Assessments show multiple ways in which content
  is covered in the program.
• C.2Candidates ability to understand and apply
  pedagogical and professional content knowledge,
  skills, and dispositions
• All cited assessments give an opportunity for
  pre-service teachers to begin to learn the
  language of teaching.
• Effectively developed assessments in this area
  make the program strong.
• C.3Candidate effects on P-12 student learning
• Analysis of student learning is clearly covered
  in the assessments. Many of the projects are
  cross-referenced and cover content, pedagogy as
  well as student learning impact standards.

73

• Well-written statements for part C-
• CLEAR/ DIRECT
• INFORMATIVE
• OBJECTIVE
• STATEMENTS not PRESCRIPTIONS

74
(No Transcript)
75

• Example of a well-written Part D
• Evidence that assessment results are evaluated
  and applied to the improvement of candidate
  performance and strengthening of the program (as
  discussed in Section V of the program report)
• Supportive evidence is clear for all
  assessments and there is a procedure in place for
  the evaluation and application of that evidence
  for the improvement of candidate performance and
  strengthening of the program.

76

• The program appears to understand that it has
  considerable work to do some significant
  weaknesses have been identified and the program
  indicates it knows it must work on these. From
  this realization can come significant change and
  strengthening of the program. This review should
  assist in the process as well.
• The university has provided evidence that they
  have made adjustments to their program based on
  data collected from the first cohort of
  candidates. In one example from the Contextual
  Statement, an analysis was made of the match
  between content expectations of the program and
  of candidates performance on the Praxis II exam
  as a way to determine monitor trends that may
  indicate the need for adjustment to the program.

77

• A well-written Part D comment
• Evidence based
• Active on-going evaluation of data
• Improvement driven proof

78
(No Transcript)
79
(No Transcript)
80

• C O N D I T I O N S
• (only for Rec with Conditions rating)

81

• This report functions as a contract between
  NCATE/SPA and the Institution.
• In order for an Institution to retain Recognition
  after a Recognized with Conditions decision has
  been granted

82

• THE CONDITIONS BOX
• MUST BE FILLED OUT !

83

• Write in the SPECIFIC conditions that need to
  be met by the institution on the next report
  submission.

84

• The revised team that looks at this when it is
  re-submitted will zero in on this conditions
  section to determine whether they were met or not
• High Stakes!

85

• An example of a well-written CONDITION
• Assessments 1 and 2 The program needs to
  address candidate knowledge of content by
  standard (academic discipline) and by category
  (sub scores) on the Praxis II exam. They must
  also present aggregated candidate data (grades)
  in a table or chart of candidate scores from high
  to low for each standard.
• Assessment 5 Candidate impact on student
  learning needs to be addressed by the degree to
  which secondary students learned from candidate
  instruction. The unit plan assignment could well
  yield such data if the candidate taught it in a
  secondary classroom and conducted something as
  simple as a pre and post test on the unit.
• Standard 3.2 The program must identify the
  instructor(s) of the Social Studies Methods
  course(s) and indicate their backgrounds in
  social studies education or in one of the
  disciplines.
• Data from all assessments for the next year must
  be collected and analyzed.
• Additional procedures for evaluating post
  baccalaureate candidates need to be implemented
  that ensure standards and indicators are met.
• Concerns cited under Standard 1 must be
  addressed.

86

• Characteristics of a well-written Conditions
  statement
• CLEAR
• PRECISE
• OBJECTIVE
• STANDARDS BASED
• EVIDENCE BASED

87
The Decision

• Program Report Decisions
• Nationally Recognized
• Nationally Recognized w/Conditions
• Not Nationally Recognized

88

• Data
• Standards
• Assessments (narrative, scoring guide (rubric)
• Alignment (standards, assessments, scoring guide,
  data)
• Tone
• Sections A-G

89

• Assessments should bring to life the
  standards---standards in action
• Standards should not decorate the assessment

90

• SPA reviewers will make one of the following
  decisions based on your program report
• The program is nationally recognized.
• The program is nationally recognized with
  conditions.
• Insufficient data
• Insufficient alignment
• Poor assessment, scoring guides, etc
• 80 rule
• The program is not nationally recognized a new
  program report may be submitted.

91

• Data Rule
• Until Spring 07, the following rule will apply
  If the program has at least one semesters data
  on at least five assessments and everything else
  is appropriate then a reviewer can recognize the
  program.
• If less than five then the reviewer should
  recognize with conditions.

92
A note about tone

• Reviewer attitude in report decisions is
  critical---
• Remember that you want to be as helpful as
  possible to the institutionstating the problem
  in clear objective languageavoid a God-like tone

93

• Reviewers are asked to accept the integrity of
  the program and review it for the purpose of
  determining if the information submitted meets
  the SPA standards
• Do not try to redesign the programs
• Objectively evaluate the QUALITY of the
  performance assessments and rubrics with respect
  to the alignment w/ SPA standards
• POSITIVE PERSPECTIVE
• Should include strengths and weaknesses
• Adapted from IRA- Guidelines for IRA/NCATE
  Program Reviewers October 4, 2006

94

• The BIG question!

95

• DOES THIS PROGRAM MEET THE SPA
  STANDARDS
• ?

96
Program and Unit Relationship

• How does my Program Recognition Decision fit
  into the Accreditation Process?

97

• The BOE (Board of Examiners)-
• will use these final report findings (across
  programs) at an institution to evaluate whether
  NCATE Standard 1 and 2 have been met

98
Communication With Your Team

• Lead makes contact
• Team sets work schedule
• Conference call capability

99
TIMELINESUMMARY

The Rolling Review Institution deadline
SPA deadline Review Team
Audit Team Initial reports
2/1 4/1 6/1 Revised reports
4/15 6/1 7/1 Late Revised Reports
5/15 7/1 8/1
100
NCATE Deadline

• Final Posting to Institution
• By 6/30 Initial Submissions
• By 7/31 Revised Submissions
• By 9/1 Latest Revised
  Submissions

101
Q A

• To pose a question
• Click hand icon
• Push talk icon to begin and end audio
• Please state name and SPA affiliation
• Questions will be taken in the order they
  are received

102
Reflection
After attending this conference, reviewers
recognize Interface differences --- current
PRS Why do I need to know both? Log-in and
assignment retrieval procedures (both
systems) Accessing submitting the Program
Recognition Template (both
systems) Timelines reviewer
roles Effective writing
strategies report
evaluation

103

• Web-Conference Archive
• http//www.ncate.org/programreview/progRevWebConf
  erences.asp?ch37

104
Conclusion

• Next Professional Development Web Event
• March 8, 2007 at 200pm (eastern)
• PRS Tutorial

105
BRIEF SURVEY

• A- Extremely
• B- Somewhat
• C- Not at all
• 1. Overall, how helpful was the Elluminate Web
  Conference Presentation?
• 2. How useful was the Eluminate Web conference
  product?
• 3. How clear was the information presented at the
  Web conference?
• 4. How comfortable do you feel with the NCATE
  Reviewer process after attending the
  conference?
• 5. How organized was the correspondence in
  getting you ready to attend the conference
  (emails, e-flyers, confirmations, configuration
  information)?

106
Program Review Staff

• NCATE Contact Number
  202.466.7496_____
• Margie Crutchfield Ph.D.
• Associate Vice President, Program Review
  (margie_at_ncate.org)
• Deidre Alves M.Ed.
• Accreditation Associate, Program Review
  (deidre_at_ncate.org)
• Jaye Bishop
• Accreditation Associate, Program Review
  (jaye_at_ncate.org)
• Monique Thomason, Accreditation Assistant,
  Program Review (monique_at_ncate.org)