Notes''

1
Notes..

• This presentation was developed by Vermont Data
  Consortium and is intended to serve as a generic
  template for anyone who is in the position of
  presenting their NECAP data to their school
  board, community, teachers or administrators.
• It is intentionally very large (too large) for
  most presentations. Decide upon your audience.
  Think about your own school district and tailor
  it to your own needs. Make it your own. Change
  the font size for your presentation needs, add
  local information. How much time do you have?
  What are the most essential questions or
  statements you would like to make? Think about
  what type of dialogue you would like to
  facilitate. Are there any local initiatives you
  would like to discuss using this data? How can
  it be connected to action planning or budgets?
  Are you presenting at an SU or local district
  site?
• Less is more. Keep it focused. Cut out those
  areas of less importance for your presentation
  and keep those critical pieces. You can always
  choose to include more on your website or a
  handout.

2
Notes..

• Look for the red font! Red font indicates a note
  from me to you, helpful advice, questions to
  consider, areas where you may want to do some
  tailoring.
• Finally, if you have any questions or get stuck,
  please contact
• Amy Cole at amy_at_amylcole.com
• John Ferreira at jferrara_at_vdcmembers.org
• Michael Hock at michael.hock_at_state.vt.us
• or visit the VDC website at www.vermontdata.org

3
ltSchool Districtgt2006 NECAP Assessment Results

• ltName of presenter(s)gt
• ltPresentation dategt

4
Organization of this presentation

• This presentation is divided into four sections
• Introduction and background about NECAP and how a
  local assessment plan works with a quick overview
  of 2005 results.
• More detailed results, how they are interpreted
  and how they connect to local programs.
• Communication, action planning and next steps.
  How the results are presented to parents in a
  student report? Commonly asked questions. What
  do we do with the information now? What are our
  next steps?
• QA

5
1. Introduction What is NECAP?

• The New England Common Assessment Program is the
  result of a collaboration among New Hampshire,
  Rhode Island and Vermont to build a set of
  assessments for grades 3-8 in order to meet the
  requirements of No Child Left Behind.
• NECAP assesses reading and math in grades 3-8 and
  writing in grades 5 and 8.
• The NECAP was piloted in October 2004 then fully
  implemented for the first time in October 2005
  and again in October 2006.

6
What is NECAP?

• NECAP replaces New Standards Reference Exam
  (Grades 4, 8 and 10) in the Vermont State
  Assessment and Accountability Plan.
• Vermont continues to administer the Grade 2
  Developmental Reading Assessment (DRA).
• A high school NECAP assessment will be
  administered in grade 11 officially for the first
  time in October 2007.
• A Science NECAP will be administered in grades 5,
  8 and 11 officially for the first time in May
  2008.

7
What does NECAP measure?
NECAP test items are aligned to the Vermont Grade
Expectations and are designed to measure student
performance in the following strands.

• Reading (grades 3-8)
• Word ID/Vocabulary
• Level of Reading Comprehension Initial
  Understanding and Analysis and Interpretation
• Type of text Literary and Informational

• Math (grades 3-8)
• Number and Operations
• Geometry and Measurement
• Functions and Algebra
• Data, Statistics and Probability
• Writing (grades 5 and 8)
• Structure of Language and Writing Conventions
• Short Responses
• Extended Response

8
How to use student data

• Triangulation of state, local and classroom
  assessment

Classroom Performance and Assessment
Student performance - content and skills
State Assessments
Common Local Assessments
9
For example.

• reading logs
• literature circles
• journals
• reading screenings
• running records

Student performance in Reading
Primary Observation Assessment (POA), Qualitative
Reading Inventory (QRI)
DRA, NECAP Reading Assessment
10
How to misuse data

• By not looking at the data accurately. Ex Are
  they the same kids?
• Taking small bits of data and drawing large
  assumptions. Look at the size of the group!
• Using only state datawhat does your local and
  classroom data tell you? Triangulate,
  triangulate, triangulate!
• Not looking at a long time period. Dont jump to
  conclusion after one year. Develop a set of
  questions and concerns. Then develop a plan to
  look at it carefully over time to see if your
  questions or concerns lead to any valid
  conclusions.
• Not involving all of the right people in the
  discussion. It is critical that we find ways to
  support teachers in looking at and making sense
  of all of the available student assessment data.
  

11
Important Changes in NECAP this year

• Out of level testing There are some students
  who have been evaluated as unable to test at
  their own grade level due to a disability. This
  tends to be approximately 2-3 of each districts
  students. In 2005 schools were permitted to test
  these students out of grade level (OTL) so that
  the students could still participate in NECAP in
  a meaningful way. Schools received two reports
  (1) all students who tested at grade level and
  (2) a separate report for students who tested out
  of grade level.

12
Important Changes in NECAP this year

• What has changed? The US Department of Education
  required that students who were designated (OTL)
  now be included in the general public reports.
  This will mean that all school districts will now
  have an additional 2-3 of students in their
  average percentages that were not included in the
  general reports last year. It should be expected
  that in this first year the percent of students
  who fall in the substantially below level will
  increase between 2005 and 2006 as we transition
  between these two practices.

13
NECAP Test Design

• NECAP Testing window
• Three weeks in October
• Grades 3-8 (Approximately 6 hours)
• Three reading sections (approx. 1 hr. each)
• Three math sections (approx. 1 hr. each)
• Grades 5 and 8 (an approximate additional 2
  hours)
• An additional two writing sections (approx. 1 hr.
  each)

14
NECAP Test Design

• Multiple choice
• Multiple-choice items are efficient for testing a
  broad array of content in a relatively short time
  span. (1 point)
• Short answer
• These open-ended items ask students to generate a
  short response to a question. (1 or 2 points)
• Constructed response
• This is a more complex item type that requires
  students to give a longer response to items
  related to a reading passage or solve multi-step
  mathematics problems. (4 points)
• Extended-response writing prompts
• These are topics or questions designed to prompt
  students to respond in writing. Students compose
  a response to the writing prompt. (12 points)

15
Quick Overview of 2006 Results

• Suggestion Put a clean graph of just the of
  students meeting and exceeding standards for
  reading, writing and math.
• How?
• As a VDC member you can construct this graph
  using the PC Analyzer. However, this query will
  be included in the Shared Favorites under EDW
  Templates. You simply need to open it up and
  drag your own school into the query.

16
How 2006 Compare with 2005 Results?

• Suggestion If you want to compare whole school
  scores from year to year then PLEASE dont forget
  to remind them about the changes in OTL and the
  mobility factor.
• Simple way.Put a clean graph of just the of
  students meeting and exceeding standards for
  reading, writing and math between 2005 and 2006.
• You may also want to show a more accurate
  comparison by providing data on only those
  students who were tested in both years.
• Questions to consider Do you want to show
  scaled scores or percentages? of students in
  each proficiency level? of students at
  proficiency and above?
• How?
• As a VDC member you can construct this graph
  using the PC Analyzer. However, this query will
  be included in the Shared Favorites under EDW
  Templates. You simply need to open it up and
  drag your own school into the query.

17
What about cohorts?

• Cohorts is a term which describes individual
  groups of students (by grade level, by NCLB
  categories, etc.) We can track groups of
  students over time or look for trends within one
  set time. (Ex how students of poverty perform
  in early grades or how one group of 3rd grade
  students perform the next year in 4th grade,
  etc.)
• However, it is very important that we are looking
  at the same students in both scenarios.

18
Have we learned anything by looking at cohorts?

• Suggestion.examine a cohort or student
  subcategory (poverty, ELL, students will
  disability, etc.) which is of particular
  interest in your district and show how NECAP can
  be used to track progress of these groups.
• You could do a whole school comparison of a
  subcategory in 2005 and then 2006 or showing a
  table with of students meeting the standard
  moving up a grade level along the table.
  Caution! Be sure they are the same kids! As
  with looking at any group over time, be sure to
  take into account mobility and the changes in
  OTL. Also, please note that it is premature to
  draw any significant theories using only two
  years of data.
• How?
• As a VDC member you can construct this graph
  using the PC Analyzer. However, this query will
  be included in the Shared Favorites under EDW
  Templates. You simply need to open it up and
  drag your own school into the query. Caution!
  If you are going to build a query with specific
  grade levels, dont forget to pull the grade
  level over from the student section, not the
  organization section.

19
2. Detailed Results

• The second part of this presentation will
• provide detailed results in math, reading and
  writing
• Provide more details about the test itself

20
Math Overview

• The NECAP mathematics test at grades 3 and 4
• 44 multiple-choice items,
• 13 one-point short-answer items, and
• 13 two-point short-answer items.
• The NECAP mathematics tests at grades 5 through
  8
• 41 multiple-choice items,
• 9 one-point short-answer items,
• 9 two-point short-answer items, and
• 6 constructed response items.

21
Example Math Tasks Grade 8
Content strand Geometry and Measurement VT
Grade Expectation 7-2 Item type Multiple
Choice Points 1 Correct answer C
22
Example Math Tasks Grade 8
Content strand Function and Algebra VT Grade
Expectation 7-1 Item type Short
answer Points 2
23
Example Math Tasks Grade 8
Content strand Number and Operations VT Grade
Expectation 7-2
Item type Constructed Response Points 4
24
Local results in Math by Grade Levels
Suggestion.Insert graph here showing each grade
level for math. Horizontal or vertical
columns. For additional context you could
include the average of combined grades so that
you can see lower and higher averages per grade
level. You could do the same for the total VT
as well. How? As a VDC member you can
construct this graph using the PC Analyzer.
However, this query will be included in the
Shared Favorites under EDW Templates. You
simply need to open it up and drag your own
school into the query. Dont forget to pull the
grade level over from the student section,
not the organization section.
25
Local results in Math by Subcategories

• Suggestion.Insert graph here showing school
  performance by subcategory, particularly if it is
  helpful in addressing any particular action plan
  item or initiative and evaluating the effect of
  these actions/programs.
• How? If you want to use a graph, use the query
  from VDC shared favorites. If you want to show
  the table found on the Measured Progress report
  by following these steps
• Go to the school district report where you can
  find your subcategories. This is a PDF file.
• Make sure you are at 100 image (not 75 or 50.)
• Find the table with subcategories.
• Take a picture of it by touching the camera
  icon on your PDF tool bar then dragging a box
  around the image you would like to copy.
• Go to this slide and paste it on.
• Resize. Try to resize it to fit your needs. Be
  careful though! You can make it very fuzzy if
  you begin to change the image too much.
• If you are tempted to compare subcategories
  from year to year please read the next slides
  cautions first!

26
Local results in Math Over Time
Suggestion. If you would like to compare
performance in the whole school or from one grade
level from year to year you can either make a
graph or copy an image from the new NECAP
report which does just that. CAUTION Please
remember that these are not the same students!
1) OTL students are included in the reports this
year and werent last year. (2) All schools
experience some mobility. How? There are
several ways you can do this. As a VDC member you
can easily run a query to include the students
who were actually tested in both years. This
will eliminate the OTL and those who moved in
after 2005 testing for the most accurate cohort
comparison.
27
What have we learned?
Students who fell in the lowest two proficiency
levels tended to score low on the short answer
and constructed response items. X of these
students were two math problems away from meeting
proficiency level. Several students were within
2 points of moving into the partially proficient
level (one or two raw item points). Several of
these students have IEPs or are ELL students.
28
What have we learned?
Students who fell in the top two proficiency
levels tended to show strength in expressing
answers on short answer and constructed response
items. X students were four points away from
Distinction (two problems or one constructed
response item).
29
Reading Overview

• The NECAP reading test at all grades
• 42 multiple-choice items and
• 9 constructed response items
• Literary passages
• Modern narratives diary entries drama poetry
  biographies essays excerpts from novels short
  stories and traditional narratives such as
  fables, tall tales, myths, and folktales.
• Informational passages
• Factual texts often dealing with the areas of
  science and social studies.
• Taken from sources such as newspapers, magazines,
  and excerpts from books and also include
  directions, manuals, or recipes.
• Word identification and vocabulary skills
• Tested, primarily through multiple-choice items,
  at each grade level.

30
Example Reading Tasks Grade 8
Content strand Informational/Analysis and
Interpretation VT Grade Expectation 7-8 Item
type Constructed Response Points 4
31
Example Reading Tasks Grade 8
Content strand Word ID/Vocabulary VT Grade
Expectation 7-2 Item type Multiple
Choice Points 1 Correct answer C
32
Local results in Reading by Grade Levels
Suggestion.Insert graph here showing each grade
level for math. Horizontal or vertical
columns. For additional context you could
include the average of combined grades so that
you can see lower and higher averages per grade
level. You could do the same for the total VT
as well. How? As a VDC member you can
construct this graph using the PC Analyzer.
However, this query will be included in the
Shared Favorites under EDW Templates. You
simply need to open it up and drag your own
school into the query. Dont forget to pull the
grade level over from the student section,
not the organization section.
33
Local results in Reading by Subcategories

• Suggestion.Insert graph here showing school
  performance by subcategory, particularly if it is
  helpful in addressing any particular action plan
  item or initiative and evaluating the effect of
  these actions/programs.
• How? If you want to use a graph, use the query
  from VDC shared favorites. If you want to show
  the table found on the Measured Progress report
  by following these steps
• Go to the school district report where you can
  find your subcategories. This is a PDF file.
• Make sure you are at 100 image (not 75 or 50.)
• Find the table with subcategories.
• Take a picture of it by touching the camera
  icon on your PDF tool bar then dragging a box
  around the image you would like to copy.
• Go to this slide and paste it on.
• Resize. Try to resize it to fit your needs. Be
  careful though! You can make it very fuzzy if
  you begin to change the image too much.
• If you are tempted to compare subcategories
  from year to year please read the next slides
  cautions first!

34
Local results in Reading Over Time
Suggestion. If you would like to compare
performance in the whole school or from one grade
level from year to year you can either make a
graph or copy an image from the new NECAP
report which does just that. CAUTION Please
remember that these are not the same students!
1) OTL students are included in the reports this
year and werent last year. (2) All schools
experience some mobility. How? There are
several ways you can do this. As a VDC member you
can easily run a query to include the students
who were actually tested in both years. This
will eliminate the OTL and those who moved in
after 2005 testing for the most accurate cohort
comparison.
35
What have we learned?
Students who fell in the lowest two proficiency
levels tended to score low on the short answer
and constructed response items. X of these
students were two math problems away from meeting
proficiency level. Several students were within
2 points of moving into the partially proficient
level (one or two raw item points). Several of
these students have IEPs or are ELL students.
36
What have we learned?
Students who fell in the top two proficiency
levels tended to show strength in expressing
answers on short answer and constructed response
items. X students were four points away from
Distinction (two problems or one constructed
response item).
37
Writing Overview

• NECAP assesses students writing skills directly
  through the use of writing prompts, or topics, to
  which students respond.
• Structures and conventions of language are
  assessed through multiple-choice items and
  throughout the students writing.
• The NECAP writing test at grade 5 included
• 12 multiple-choice items,
• 3 constructed response items,
• 3 one-point short-answer items, and
• one extended-response writing prompt.
• Each of the constructed response items and the
  writing prompt addressed a different genre of
  writing.
• Writing in Response to Literary Text
• Writing in Response to Informational Text
• Narratives
• Informational Writing (Report/Procedure at grade
  5 and Persuasive at grade 8)

38
Writing Overview

• The content standards in writing identify four
  major genres
• Writing in Response to Literary Text
• Writing in Response to Informational Text
• Narratives
• Informational Writing (Report/Procedure at grade
  5 and Persuasive at grade 8)
• Each year, all four genres of writing are
  assessed in the writing portion of the NECAP
  test. In addition, structures and conventions of
  language are assessed through multiple-choice
  items and throughout the students writing.

39
Example Writing Tasks Grade 8
Content strand Short Responses VT Grade
Expectation 7-5 and 7-8 Item type Constructed
Response Depth of Knowledge 2 Points 4
40
Example Writing Tasks Grade 8
Content strand Structures of Language and
Writing Conventions VT Grade Expectation
7-9 Item type Multiple Choice Depth of
Knowledge 1 Points 1 Correct answer A
41
Local results in Writing by Grade Levels
Suggestion.Insert graph here showing each grade
level for math. Horizontal or vertical
columns. For additional context you could
include the average of combined grades so that
you can see lower and higher averages per grade
level. You could do the same for the total VT
as well. How? As a VDC member you can
construct this graph using the PC Analyzer.
However, this query will be included in the
Shared Favorites under EDW Templates. You
simply need to open it up and drag your own
school into the query. Dont forget to pull the
grade level over from the student section,
not the organization section.
42
Local results in Writing by Subcategories

• Suggestion.Insert graph here showing school
  performance by subcategory, particularly if it is
  helpful in addressing any particular action plan
  item or initiative and evaluating the effect of
  these actions/programs.
• How? If you want to use a graph, use the query
  from VDC shared favorites. If you want to show
  the table found on the Measured Progress report
  by following these steps
• Go to the school district report where you can
  find your subcategories. This is a PDF file.
• Make sure you are at 100 image (not 75 or 50.)
• Find the table with subcategories.
• Take a picture of it by touching the camera
  icon on your PDF tool bar then dragging a box
  around the image you would like to copy.
• Go to this slide and paste it on.
• Resize. Try to resize it to fit your needs. Be
  careful though! You can make it very fuzzy if
  you begin to change the image too much.
• If you are tempted to compare subcategories
  from year to year please read the next slides
  cautions first!

43
Local results in Writing Over Time
Suggestion. If you would like to compare
performance in the whole school or from one grade
level from year to year you can either make a
graph or copy an image from the new NECAP
report which does just that. CAUTION Please
remember that these are not the same students!
1) OTL students are included in the reports this
year and werent last year. (2) All schools
experience some mobility. How? There are
several ways you can do this. As a VDC member you
can easily run a query to include the students
who were actually tested in both years. This
will eliminate the OTL and those who moved in
after 2005 testing for the most accurate cohort
comparison.
44
What have we learned?
Students who fell in the lowest two proficiency
levels tended to score low on the short answer
and constructed response items. X of these
students were two math problems away from meeting
proficiency level. Several students were within
2 points of moving into the partially proficient
level (one or two raw item points). Several of
these students have IEPs or are ELL students.
45
What have we learned?
Students who fell in the top two proficiency
levels tended to show strength in expressing
answers on short answer and constructed response
items. X students were four points away from
Distinction (two problems or one constructed
response item).
46
3. Communicating results and Taking Action

• The Student Report
• Commonly Asked Questions
• Action Planning and Next Steps

47
How do I read the NECAP Student Report?

• The NECAP Student Report is divided into three
  sections.
• Students Achievement Level and Score
• Students Achievement Level Compared to Other
  Students by School, District, and State
• Students Performance in Content Area
  Subcategories

48
Student Report
49
Student Report
50
Student Report
51
How are we responding to this new data?

• Providing training and support to our teachers
  with regards to how to support parents in
  understanding their childs results.
• Participating in state-wide NECAP trainings.
• Providing individual team and classroom student
  results in the hands of current teachers and
  teachers who taught the students the previous
  year.
• Working with administrators in looking at
  school-wide and team-level disaggregated data
  such as gender, free and reduced lunch, ELL and
  students with disabilities.
• Looking at students who fell into each
  proficiency level and determining whether these
  are students who are currently on some type of
  plan such as an IEP, 504, EST or other
  intervention plans.
• Working as curriculum teams in looking at how our
  students performed on individual content strands
• Comparing NECAP with local assessment data in
  order to draw conclusions regarding curriculum
  and instruction target areas.

52
How do we use it to evaluate programs?

• Suggestionfocus on one or two initiatives and
  see if NECAP can provide any data to look
  critically at how those initiatives are working.

53
Commonly asked questions

• Q I noticed that the percent of students who
  achieve the standard on the grade 2 DRA is about
  10 higher than the 3rd grade reading NECAP. Why
  is that?
• A Quite simply, they are different assessments.
  The 10 trend is common across VT and should not
  be of significant concern. The DRA is a
  one-on-one assessment (narrow in focus and
  provides deeper information.) NECAP is a
  whole-group assessment which assesses more
  components of reading but with less depth. Both
  sets of scores are valid and inform our
  instruction differently. Teachers work together
  to look at what the two assessments and what
  they measure and then use the data accordingly to
  adjust instruction within their classes.

54
Commonly asked questions

• Q Can the NECAP tell me if my child has
  progressed from one year to the next?
• A Yes in that you can compare your childs
  scaled score (ex 440) to this years raw
  score (ex 540). The first digit (4) reflects
  the grade level assessed. If your childs score
  increases by about 100 you can assume that they
  progress about one grade level from the last
  assessment. Keep in mind that this assessment is
  not the only important measurement of growth.
  These are single snapshots of your child on one
  given day. In addition, not all students show
  their abilities well on assessments.

55
Commonly asked questions

• Q Can we look at whole groups and whether or not
  they improved over time?
• A Yes, if we look at the correct data. Simply
  looking at one 5th grade class as they move to
  6th grade does not take into consideration
  whether or not they were the same students. In
  2006 we know that additional students who were
  tested out of level will be included where they
  had not been previously included in the average
  score. Students also move from school to school.
  If your intent is to truly assess progress of
  groups, it is important you look at the students
  who were actually there for both tests!

56
Commonly asked questions

• Q My child did not perform well on the
  geometry subtest of the math NECAP. What does
  that mean?
• A It simply means that they did not perform on
  those items on this one state assessment. When
  you meet with your teacher at parent conferences,
  ask if there is any additional evidence from
  classroom work to suggest your child is needing
  extra support in this area. There are many ways
  to show if a child is meeting or not meeting a
  standard. Ask your classroom teacher to help you
  find out how they are doing in other settings.

57
Commonly asked questions

• Q If we compare our scores to other school
  districts can we assess how well our students are
  performing overall?
• A Yes and no. Again, NECAP is only one
  measurement. It is important to look at a
  variety of data to see how your school is
  performing, including other state assessment
  data, local assessment data, graduation rates,
  attendance rates, school and community
  demographics, etc. Also, keep in mind the issue
  of mobility. If students are moving in and out
  of your district they are likely doing the same
  in other districts. Districts may find it
  helpful to find comparable school districts
  and/or use the VT averages rather than simply
  looking at the neighboring school district.

58
What are our next steps?

• Obviously these should be locally designed.
  Below are a few suggestions
• Continue to look at individual, team and
  school-wide data to look for trends.
• Triangulating local, state and classroom data and
  develop any recommendations towards individual or
  group level student intervention plans.
• Examine existing resources towards providing
  interventions for students who are below
  proficiency levels.
• Prioritize any target improvement areas and
  incorporate our findings into the new CSSU action
  plan and local action plans.
• Take a look at the calendar and build in time for
  teachers to work together in looking at data and
  working on identified target areas.

59
4. QA
Additional information is available at the VT
Department of Education website
www.state.vt.us/educ