Montana Comprehensive Assessment System

1
Montana Comprehensive Assessment System
MontCAS
Montana Office of Public Instruction Linda
McCulloch, Superintendent May 2006
2

• Welcome
• Panelists
• Sharon Houle, Measured Progress
• Carol Taylor-Cann, Riverside Publishing
• Bob Runkel, OPIGAP Students
• Lynn Hinch, OPIEnglish Language Proficiency
• Donna ONeill, OPI--NAEP Coordinator
• Linda Peterson, OPIScience Standards
• Moderator, Judy Snow, OPI--Assessment

3
Agenda

• MontCAS Overview
• Judy Snow
• MontCASUpdates on Other Assessments
• Panelists
• MontCAS Issues and Details
• Judy Snow

4
MontCAS Overview

• Assessment 101
• MontCAS Phases 1 and 2

5
Assessment 101

• Standardized Test
• Norm-referenced test
• Criterion-referenced test
• Performance levels
• Alternate Assessment
• LEP, ELL, ELP

• NCLB
• No Child Left Behind
• Adequate Yearly Progress (AYP)
• Administrative Rules of Montana
• ARM
• Board of Public Education (BPE)

6
MontCAS

• Phase 1
• Norm-referenced test (NRT)the IowasRiverside
  Publishing Company
• Aligned to national standards
• Grades 4, 8, 11
• Reading, language arts, math, social studies,
  science
• Multiple choice
• Alternate assessment scales
• Funded by the state

• Phase 2
• Criterion-referenced test (CRT)Measured Progress
• Aligned to Montana Content Standards
• Grades 3-8, and 10 (science in spring 2008)
• Reading and math
• Multiple choice, math short answer and
  constructed response
• Evidence based alternate
• Funded by NCLB

7

• Other AssessmentsPanelists

8
Bob RunkelModified CRT
9
GAP Students

• In between CRT-Alternate and CRT
• Modified performance standards
• Feasibility Study
• Grade 5 students
• Identified as special education students for the
  grade 4 CRT in spring 2006
• Pilot math CRT based on modified performance
  standards
• Tentative Window
• September 11-19, 2006

10
Lynn Hinch English Language Proficiency Test
11
ELP

• English Language Proficiency
• Required by NCLB
• Grades K-12
• Test Development through Mountain West Consortium
  (MWAC)
• Date TBA

12
NO CHILD LEFT BEHIND REQUIREMENTS

• Both Titles I and III require an annual
  assessment of English language proficiency of all
  K - 12 limited English proficient students
• Assessment must be aligned to state English
  language proficiency standards
• Assessment must report a separate score for
  listening, speaking, reading, and writing

13
TITLE III REQUIREMENTS

• States must establish Annual Measurable
  Achievement Objectives (AMAOs) that determine how
  LEP students are making progress in learning
  English, and whether students are attaining
  proficiency in English.

14
Status of English Language Proficiency Assessment

• Mountain West Assessment Consortium
• Test Development and Delivery
• RFP Process
• March '06 Request to the Department of Education
  for timeline extension

15
Donna ONeillNAEP
16
NAEP

• National Assessment of Educational Progress
• NAEP--Nations Report Card
• Odd-numbered Years
• Grades 4 and 8, Reading and Math
• Required Sample

17
All About NAEP

• What is NAEP?
• Why is it important to Montana?
• Who takes the NAEP Assessment?
• What are the requirements for NAEP?
• New developments in Science Writing
• What is the timeline for the NAEP Assessment?

18
NAEPNational Assessment

• The Nations Report Card
• Measures academic achievement
• Reports to the public what students know and can
  do
• average scale scores
• proficiency levels
• Formats vary by grade
• multiple choice
• constructed-response
• Links performance to educational variables
• Teacher background information
• School characteristics
• NAEP is confidential

19
Montana
20
The Nations Report Card
21
Montana
22
Who?

• Random sample
• 2,000 students per
• Subject
• Grade
• Grades 4, 8 12
• Students with disabilities
• Limited-English proficient

23
NAEP Requirements

• Linked to Title I funding
• Schools
• Districts
• NAEP is designed
• Minimal disruption of instruction
• 90 to 120 minutes of each students time
• Teachers are encouraged to observe
• Field staff are many Montana teachers

24
New Developments

• Science Release
• May
• Writing (8, 12)
• Pilot and Special Studies
• National Indian Education Study (NIES)
• Grades 4, 8
• NAEP data will be linked to survey data
• Student, Teacher and Principal Questionnaires

25
NAEP Timeline
26
Key Dates

• Secure cooperation of districts and schools
• Notify districts in early May
• Notify schools of selection during May
• Notify schools of assessment dates in June
• Maintain communication with schools throughout
  the process

27
Handouts

• Packet of Information
• Facts About Montana Education
• Historical information on results
• NAEP Questions Tool
• NAEP Data Explorer (researchers)
• http//nces.ed.gov/nationsreportcard/

28
Sharon HouleMeasured Progress
29
CRT Science Test

• Grades 4, 8, and 10
• Spring 2008
• Aligned to Revised Science Standards

30
Criterion-Referenced Test Where are we going?

• Science--grades 4, 8 and 10
• May 31, 2006
• National science content and bias reviews
• July 25
• Montana science content and bias reviews
• Spring 2007
• Field test
• Spring 2008
• First AdministrationCRT and CRT-Alternate

31
Linda Peterson

• ACCREDITATION DIVISION
• STANDARDS REVISION PROCESS

32
The Five-Year Comprehensive Education Plan
(5YCEP) Building a Continuous Improvement
Process for Montana Schools
33
Montana Board of Public Education

• CHAPTER 54
• CONTENT AND PERFORMANCE STANDARDS
• 10.54.2503 STANDARDS REVIEW SCHEDULE (1)
  Montana's content and performance standards shall
  be reviewed and revised on a five-year cycle
  beginning July 1, 2005.

34
Montana Board of Public Education

• Standards Revision 2005-2010
• Purpose
• Assure Montana citizens that its public schools
  are providing all children of our great state
  with challenging academic expectations
• Revised standards clearly and consistently
  identify what students should know, understand
  and be able to do
• Revised standards provide a framework to help
  guide local curriculum and instruction

35
Montana Board of Public EducationCriteria to
Guide Standards Revision

• Standards will be
• academic in nature and content specific.
• challenging and rigorous.
• clear, understandable and free of jargon.
• measurable.
• address diversity, specifically fulfilling the
  commitment to implementing 20-1-501, Indian
  Education for All.

36
(No Transcript)
37
K-12 Science Standards

• Montana Standards Framework
• Content Standards
• Benchmarks - Grades 4 and 8 and upon graduation
• Performance Descriptors

38
K-12 Science Standards

• Performance Descriptors
• Advanced superior performance
• Proficient solid academic competency
• Nearing Proficiency partial mastery of
  prerequisite knowledge and skills
• Novice beginning to attain prerequisite
  knowledge and skills

39
K-12 Science Standards

• Content Standard 1
• Students design, conduct, evaluate, and
  communicate processes and results of scientific
  investigations, and demonstrate thinking skills
  associated with this procedural knowledge

• Benchmarks
• Grade 4 (3) - describe and communicate the
  results of scientific investigations
• Grade 8 (3) - review, communicate and defend
  results of investigations
• Upon Graduation review evidence, communicate
  and defend results, and recognize that the
  results of a scientific investigation are always
  open to revision by further investigation

40
Performance DescriptorsContent Standard 1

• Proficient
• Grade 4 with direction, completes a simple
  investigation with identified variables, using
  appropriate tools and communicates results
• Grade 8 identifies and communicates testable
  questions, plans and conducts experimental
  investigations and communicates results
• Upon Graduation generates testable questions,
  constructs a plan for a controlled investigation,
  makes logical inferences based on observations,
  accurately interprets data by identifying the
  strengths and weaknesses in an investigation
  design, and communicates results.

41
K-12 Science Standards

• Content Standard 3
• Students demonstrate knowledge of
  characteristics, structures, and function of
  living things, the process and diversity of life,
  and how lining organisms interact with each other
  and their environment, and demonstrate thinking
  skills associated with this knowledge.

• Benchmarks
• Grade 4 create and use a classification system
  to group a variety of plants and animals
  according to their similarities and differences,
  (e.g., American Indian medicinal plants, American
  Indian dwellings)
• Grade 8 create and use a basic classification
  scheme to identify plants and animals, (e.g.,
  classification scheme based on American Indian
  use of indigenous plants for medicinal, food and
  other uses)
• Upon Graduation recognize, generate and apply
  biological classification schemes to infer and
  discuss the degree of divergence using
  ecosystems, (e.g., American Indian food
  collections systems)

42
Performance DescriptorsContent Standard 3

• Proficient
• Grade 4 identifies attributes of biotic
  (living) things and abiotic (non-living) objects,
  including classification based on similarities
  and differences, basic structure and function,
  processes of each system
• Grade 8 identifies and classifies biotic things
  and abiotic objects through application of common
  classification schemes identifies the
  interdependence of life and the environment and
  explains how characteristics of living things
  change because of the environment
• Upon Graduation organizes, classifies, and
  describes interactions of the biotic and abiotic
  parts of the biosphere as well as the natural
  history of interactions of life on Earth and uses
  these skills to solve related novel (to the
  student) problems

43
K-12 Science Standards Proposed Time Line

• March May Gather Public Comments
• May 11 Present Update to BPE
• May 15 Presentation and Discussion with MACIE
• May June Writing Team Work Sessions
• June Prepare draft standards for notice of
  hearing

44
K-12 Science Standards Proposed Time Line

• July Present to BPE draft Notice of Hearing
  and proposed time line
• August Notice of Hearing posted
• October Public Hearing
• November Final BPE Action Anticipated
• 2007 Implementation, Outreach, and
  Dissemination

45
K-12 Science Standards Revisions

• Linda Vrooman Peterson
• Accreditation Division Administrator
• Office of Public Instruction
• lvpeterson_at_mt.gov
• [phone]
• Draft K-12 Science Standards link
• http//www.opi.mt.gov/pdf/standards/ScienceStandar
  dsMarch06.pdf

46
Assessment Issues and Details

• Standardized Administration
• Accurate Data
• Technical Adequacy
• Impact on Instruction

47
Standard Administration Accurate Reliable
Data

• Participation
• Training
• Test Security
• Special Populations
• Accommodations
• Alternate Assessments

48
Participation All Means All!

• Phase 1 Options
• No accommodations
• If in IEP, 504, or LEP plan
• Standard or non standard accommodations
• Alternate Assessment Scale
• Combination

• Phase 2 Options
• No accommodations
• Standard accommodations
• If in IEP or 504 plan
• Non standard accommodations
• CRT-Alternate

49
Training so that . . .

• everyone understands the procedures for
  administration.
• the test is administered in a comparable way in
  all locations across the state.
• quality control procedures are utilized when
  returning test materials.

50
Test Security
All test items and responses to those items in
MontCAS are secure materials and may not be
copied or duplicated in any way CRT materials
may not be retained in the school after testing
is completed. It is very important to review
test security information with test
administrators during test administration
training.
51
OPI Test Security

• Any concern about breaches in test security or
  noncompliance with test administration procedures
  must be reported immediately to the principal and
  system test coordinator and to the State
  Assessment Director.   
• OPI Guidelines and Procedures for Test Security
• This OPI publication outlines procedures for
  reporting testing irregularities and should be
  made available to system superintendents,
  principals, and test administrators.

52
Accommodations
Standard accommodations do not change what we
expect students to know and be able to do. Test
accommodations are based on individual student
needs and classroom assessment practice they are
not for the use by an entire classroom or
category of students. They should not give
students unfair advantages rather they are meant
to remove barriers that may exist due to a
students learning style or disability.
53
Standard and Non StandardAccommodations

• Standard
• Do not change the intent/content of the
  assessment
• Phase 1 Only by IEP, 504, or LEP plan
• Phase 2 By IEP, 504, LEP, or regular routine in
  the classroom

• Non Standard
• Change the content/intent of the assessment
• Phase 1 Only by IEP,504, or LEP plan
• Phase 2 Only by IEP or 504 plan

54
Accurate Data

• the information collected is of high quality
• demographic information
• enrollment information
• disability information
• decisions related to the use of accommodations
  and the alternate assessment (CRT-Alternate)

55
2006 Data Check Points

• OPI CRT Testing Cycle Enrollment Count
• March 6, 2006 Window Opens
• April 13, 2006 Data due to County
  Superintendent
• April 20, 2006 Data due to OPI
• Measured Progress CRT Online Demographic Data
  Verification
• April 24 May 19, 2006
• OPI Comparability StudyJune 2006
• OPI Iowa Tests (NRT) Disaggregated Data Review
• July 24 - August 4 , 2006

56
Technical Quality and Expertise

• Test Development
• Technical Advisory Committee (TAC)
• Technical Manuals
• CRT and CRT-Alternate Standard Setting

57
Montana Performance Levels

• Advanced
• Proficient
• Nearing Proficiency
• Novice

58
Standard Setting
Standard Setting

• Standard setting is the process of determining
  cut points for performance levels for the CRT and
  the CRT-Alternate.
• Panels of Montana educators
• Facilitated by Measured Progress

59
2006 Standard Setting Details

• For more information http//www.opi.mt.gov/asses
  sment/Phase2.htmlTD
• To apply
• http//iservices.measuredprogress.org
• Travel, meals (those not provided during the
  sessions), and lodging reimbursements
• Renewal units and honoraria

60
2006 Standard SettingWhere and When

• Red Lion Colonial Inn in Helena
• June 21-22
• CRT, Reading and Math, grades 4, 8, and 10
• June 20-22
• CRT-Alternate, Reading and Math, grades 3-8 and
  10
• July 26-27
• CRT, Reading and Math, grades, 3, 5, 6, and 7

61
Impact on Instruction

• Reporting
• Data Interpretation
• Tools

62
Reporting and Impact on Instruction

• Phase 1NRT
• NPR
• NCE
• Stanine
• Performance Level
• Phase 2CRT
• Scaled Score
• Performance Level for each content area
• Raw Score
• Number of points for each standard

63
Iowa Tests Interpretive Guide

• 2005
• http//www.opi.mt.gov/PDF/Assessment/NRT/05InterpG
  uide.pdf
• Relationship of stanines, percentile ranks, and
  normal curve equivalents (p. 5)
• Comparison of NRT Scores (p. 6)
• Performance Profile (p. 12)
• Individual Customized Skills Report (p. 16)

64
CRT Results Interpretation

• 2005 Interpretive Guide
• http//www.opi.mt.gov/PDF/Assessment/CRT/05CRTCRTa
  ltInterpGuide.pdf
• Roster and Item-Level Report (p. 6)
• Released Items
• Items that scored are released.
• http//www.opi.mt.gov/Assessment/index.html
• iAnalyze
• A secure website where data gathered from the CRT
  assessments are reported

65

• Judy Snow
• State Assessment Director
• Office of Public Instruction
• jsnow_at_mt.gov
• 406-444-3656
• Assessment link
• http//www.opi.mt.gov/Assessment/index.html
• JUMP link
• http//www.opi.mt.gov/Assessment/JUMP.html