NCATE Program Review Process

1
NCATE Program Review Process

• Margaret D. Crutchfield, Ph.D.
• September 2006
• margie_at_ncate.org

2
A brief review.

• Unit School, College, or Department of
  Education plus other entities on campus that
  prepare school personnel
• Program Specific Discipline Area

3
(No Transcript)
4

• SPA Specialized Professional Association

5
NCATE has standards for the following programs

• Early Childhood Education
• Educational Communications and Instructional
  Technology
• Educational Computing and Technology
• Educational Leadership
• Elementary Education (Initial)
• English as Second Language Education (Initial)
• English Language Arts Education (Initial)
• Foreign Languages Education (Initial)

• Health Education (Initial)
• Mathematics Education (Initial)
• Middle School Education
• Physical Education
• Reading Education
• School Library Media
• School Psychology
• Science Education (Initial)
• Social Studies Education (Initial)
• Special Education
• Technology Education (Initial)

6

• What about programs accredited by other
  accrediting bodies (e.g. ASHA, CACREP, etc.)?
• What about programs for which NCATE does not have
  standards and are not accredited by another
  agency?

7
How Do Program Reports Fit into the Unit
Accreditation Review?

• Program recognition reports are important part of
  evidence for meeting NCATE Standard 1
• A unit can be accredited without receiving
  national recognition for all programs

8
Purpose of the Program Review

• Determine whether or not the program has in place
  a limited number (6-8) of comprehensive
  assessments that demonstrate candidate mastery of
  the SPA standards.
• Candidate performance on these assessments is
  appropriate to demonstrate mastery.
• Provide information for unit to use to respond to
  Unit Standard 1

9
NCATE Unit Standard 1

• Candidates preparing to work in schools as
  teachers or other school personnel know the
  content of their fields, demonstrate professional
  and pedagogical knowledge, skills, and
  dispositions and apply them so that students
  learn. Assessments indicate that candidates meet
  professional, state, and institutional standards.

10
Required Assessments

• State licensure exam for program area (if
  availableotherwise another content based
  assessment)
• Content Assessment
• Assessment of Planning (e.g., unit plan)
• Student teaching/internship assessment
• Assessment of candidate impact on student
  learning or providing a supporting learning
  environment
• A 6th assessment is required, type is optional

11
Required program assessments

• Content
• State licensure exam for program area (if
  availableotherwise another content based
  assessment)
• Content Assessment
• Professional and Pedagogical Knowledge Skills and
  Dispositions
• Assessment of Planning (e.g., unit plan)
• Student teaching/internship assessment

12

• P 12 Student Learning
• Assessment of candidate impact on student
  learning or providing a supporting learning
  environment
• Program is required to have at least one more
  assessment, can have a maximum of 8 to
  demonstrate candidate mastery of SPA standards

13
Assessment 1 - Licensure data

• State test required for licensure in the content
  area
• 80 of completers must meet or exceed the state
  pass score

14
Assessment 2 Another content-based assessment

• Grades (although this can be tricky)
• Comprehensive exam
• Content-based portfolio assessment
• Case studies action research

15
Assessment 3 Ability to plan

• Assessment of ability to plan instruction
• (classroom-based programs)
• e.g., unit or lesson plan assignment
• Assessment of ability to fulfill specified
• professional roles (advanced programs)
• e.g., needs assessment project

16
Assessment 4 Clinical Practice

• Student teacher/internship/practicum evaluation
• What if it is a unit-wide (not discipline-specific
  ) assessment?

17
Three effective strategies

• SPA-specific addendum
• SPA-specific rubrics
• Coding assessment
• AND including information in narrative about
  context

18
(No Transcript)
19
Assessment 5 Impact on Student Learning

• Impact on Student Learning (Initial Teachers)
• Teacher work sample or other classroom-based
  project
• Ability of Candidate to Provide Supportive
  Learning Environment (Other School Personnel)
• Action research
• Employer or other external surveys (ELCC)

20
Acceptable rubric for Unit Standard 1 expects
candidates to

• Assess student learning
• Be able to use assessments in instruction
• Develop meaningful learning experiences for
  students based on their developmental levels
  prior experience.

21
Candidates can

• judge students prior learning,
• plan and teach lessons,
• assess student learning,
• reflect on student learning, and
• make adjustments to their teaching to improve
  learning.

22

• Must have at least one more assessment (6) but
  assessment type is optional unless SPA has
  SPA-specific requirements
• SPA specific information is included in each
  program report form
• Each program can have up to eight assessments

23
(No Transcript)
24
What is an assessment?

• For the purposes of the program report, an
  assessment is an evaluated activity or
  requirement by which a program determines that
  specific outcomes or standards have been mastered
  by a candidate.
• The assessment package consists of the
  assessment instrument, scoring guide, and
  candidate data derived from the assessment.

25
What isnt an assessment

• A syllabus (although it might be used as an
  assessment instrument for a course grade)
• A list of program objectives or expected
  candidate outcomes
• State or professional standards by which a
  program is designed/assessed

26
What might the assessment instrument look like?

• Examples
• A sample of a comprehensive test
• An evaluation form (student teaching, survey)
• A course outline or outlines (if grades are used
  as the assessment)
• A description of the assignment (unit plan,
  teacher work sample)

27
What Criteria Are Used to Evaluate Assessments?

• Assessments are aligned to standards
• Assessments reflect the rigor of standards
• Assessments are free from bias
• Scoring guides clearly identify levels of
  proficiency
• Data provide evidence of candidate mastery of
  standards

28
What about the scoring guide?

• The scoring guide should
• Include explicit statements of proficiencies
  candidates are expected to demonstrate in their
  responses.
• Be constructed so that different levels of
  candidate proficiency are clearly distinguished.

29
What isnt a scoring guide

• A Excellent
• B Good
• C Satisfactory
• etc.

• Journal 10 pts
• Essay 50 pts
• Attendance 20 pts
• Final exam 20 pts

30
(No Transcript)
31
More on Scoring Guides

• The assessment/data are only as good as the
  scoring guide.
• If possible, the scoring guide should be specific
  to the discipline and aligned to the standards.
• Scoring guide should be correlated to and
  appropriate for the assessment.

32
Presenting Your Data

• Data tables should be self-contained documents
  readable and user-friendly
• Shown the parts of the sum, not (just) the sum of
  the parts!
• Data should be presented by categories used in
  the scoring guides

33
What Is Not Useful Data

• Candidate GPA in the math methods course for
  the past three years is 3.27.

• All of our candidates must score at the
  proficient level or above on this assessment in
  order to advance to student teaching.

34
Characteristics of Well-Presented Data
Specify academic year or cohort group, e.g.
2002-3003 candidates admitted spring 2003
Note Provide legends for abbreviations,
acronyms, etc.
35
Disaggregate Data for

• Different levels of a program (e.g. baccalaureate
  and Initial Masters)
• Different sites at which a complete program is
  offered
• Different cohorts or grading periods

36
How Much Data for Program Reports?

• At least through reports submitted in spring
  2007, one semester of data for five assessments
• In fall 2007 and spring 2008, one year of data
  will be required
• Fall 2008 and spring 2009, two years of data
• Three years of data are optimal not necessary
  to provide more

37
Program review decisions

• SPA reviewers will make one of the following
  decisions based on your program report
• The program is nationally recognized.
• The program is nationally recognized with
  conditions.
• Insufficient data
• Insufficient alignment
• Poor assessment, scoring guides, etc
• 80 rule
• The program is not nationally recognized a new
  program report may be submitted.

38
When do you submit reports?

• HIGHLY recommend submitting reports 12 months
  prior to site visit
• February 1 or September 15
• Can submit up to 2 years prior to visit

39
What else has changed?

• Not required to place key assessments at specific
  transition points
• Do not submit examples of candidate work

40
Who to Contact at NCATE

• For questions related to overall process,
  submission requirements, etc.
• Margie Crutchfield, margie_at_ncate.org
• Jaye Bishop, jaye_at_ncate.org
• Deidre Alves, deidre_at_ncate.org
• Technical problems
• Frank Huang, frank_at_ncate.org

41
Resources on NCATE web site

• www.ncate.org, click on Institutions, click on
  Program Reviews
• Program Report Forms, Guidelines, Instructions,
  Examples of Assessments, Examples of Report
  Sections
• www.ncate.org, click on Program Reviewers
• All training materials for program reviewers