Monitoring and PreScoring Activities

1
Monitoring and Pre-Scoring Activities

• Virginia Department of Education
• 2007-2008 Alternative Assessments Administrators
  Update Workshop
• August 2007

2
Monitoring

• Periodic and systematic review of evidence-based
  assessments in the development process with
  opportunities for feedback and intervention.

3
What Monitoring is Not

• Checking with the teacher
• Flipping through an evidence based assessment
• Review without feedback

4
Pre-Scoring

• A detailed review of evidence-based assessments
  against scoring rules prior to scoring with
  opportunity for feedback and intervention

5
What Pre-Scoring is Not

• A mad dash to create a collection of evidence

6
Why Monitor and Pre-Score?

• To create better Collections of Evidence (COE)
  and Coursework Compilations (CWC).

7
Why COEs and CWCs Fail

• Missing evidence to support the SOL
• Use of unacceptable evidence
• Use of evidence not aligned with the SOL
• Un-graded or incorrectly graded evidence
• Missing or incomplete SEI tags

8
Components of Monitoring

• Trained Reviewers
• Feedback Loops
• Intervention Options

9
Trained Reviewers

• Persons with Content Knowledge and V-Program
  Knowledge
• School Coordinator
• Assigned or Hired Staff
• Building Level Teams
• Central Office Level Teams

10
Feedback Loops

• Teachers preparing collections
• Principals
• V-Program Leaders
• Trainers/Technical Assistance
• Providers

11
Intervention Options

• Individualized Consultation and Support
• Peer or small group sessions
• Attendance at Local Training
• Access to state and local SOL or assessment
  resources

12
Monitoring Decisions

• Who will monitor?
• How will you train them?
• When will monitoring occur?
• Who will receive feedback?
• How will feedback be given?
• What are your intervention options?

13
A Sample Monitoring Plan

• School-based Review Teams created and trained
• Teachers turn in evidence-based assessments every
  nine-weeks
• All evidence-based assessments reviewed by team
  according to division pacing chart
• Team report shared with principal and central
  office
• Central office dispatches Rapid Response Teams

14
A Sample Monitoring Plan
School-Based Teams Review COEs every 9 weeks
according to Pacing Chart and report To Principal
and Division V-Program Leaders
Select And Train School-Based Oversight Teams
Instructional Teams dispatched Based on 9 week
Report
15
A Sample School Review Team 9-Week Monitoring
Form
16
A Sample Teacher Review Form

• VGLA 9 Week Review Sheet
• School-Based Teams
• Teacher ____________________
• Student ___________________________ Grade
  Level______
• Content Area_____________________________________
  ____
• REVIEW for ____ 1st Nine Weeks ____
  3rd Nine Weeks
• ____ 2nd Nine Weeks
  ____ 4th Nine Weeks
• Reviewed By____________________________
  Date________
• Collection Status Address the following
  questions (1) Is there evidence for all of the
  standards for the nine weeks according to the
  Pacing Chart? (2) Does the work submitted align
  with the standards satisfactorily? (2) Does the
  evidence demonstrate student mastery? (3) Has the
  student work been graded accurately? (4) Other?
• Recommendation(s) Be as specific as possible
• Follow-up review needed for implementation of
  recommendations
• ___Yes ____No
• Follow up Review Date ________________

17
Pre-Scoring Activities

• Permits a review of the completed (or nearly
  completed) collections of evidence and course
  work compilations against scoring rules with
  opportunity to make corrections

18
Rule 1- Evidence Must Show Individual Student
Achievement

• Does the evidence show any level of achievement
  according to the SOL being defended?
• Is the evidence on grade level?
• Are all the SOL addressed (stems and bullets)?

19
Rule 2- Evidence Must Be Student-Generated

• Has the evidence been copied from the blackboard,
  textbook, or computer?
• If the evidence is from a group project, is the
  achievement of the V-Program participant clearly
  identified?
• If the evidence is a worksheet, are there
  examples or directions that provide answers?

20
Rule 3 - Evidence Must Be Labeled with SEI Tags

• Does each piece of evidence have a completed SEI
  tag with
• Content area
• SOL
• Bullet (s)
• Demonstrated or Inferred checked

21
Rule 4- Evidence Must Address Accommodations,
Captions and Grading

• If accommodations have been used, are they
  documented in the evidence?
• Does each picture have a caption that clearly
  address the students level of achievement?
• Has the work been graded accurately with correct
  and incorrect answers clearly identified?

22
Media Considerations

• Do videotapes and audiotapes have transcripts?
• Are videotapes and audiotapes clearly labeled
  with SEI tags?

23
Organizational Considerations

• Is the evidence organized according to the
  scoring worksheet?
• Are all required state and local forms completed?

24
Pre-Scoring Decisions

• Who will pre-score?
• How will you train them?
• When will pre-scoring occur in relation to our
  submission date?
• Who will receive feedback?
• How will feedback be given?
• What are your intervention options?

25
Sample Pre-Scoring Plan

• PHASE 1 - Building Level Review
• Submitting Teacher Review
• School-Based Oversight Teams Review
• Building Administrator Review and sign-off

• PHASE 2 - Central Office Review
• Submitted from schools
• Reviewed by Central
• Office Teams
• Return to Schools to Correct identified errors or
  omissions

26
VGLA Collection of Evidence Coversheet
27
Monitoring Pre-Scoring Results

• Reduces or Eliminates COEs and CWCs with
• Missing Evidence
• Unacceptable Evidence (textbooks, homework)
• Inaccurate or Ungraded Evidence

28
Monitoring Pre-Scoring Results

• Eliminates Surprises
• Collections not done for students educated
  outside of the division
• Incomplete collections in the division
• Collections not driven by IEP, 504 or LEP Student
  Assessment Participation Plans

29
Questions

• Division of Student Assessment and School
  Improvement
• [phone]