Monitoring

1
Monitoring Pre-Scoring Activities

• 2007-2008
• Administrators Update Workshop
• Virginia Department of Education
• September 2007

2
What Monitoring is

• Periodic and systematic review of evidence-based
  assessments
• in the development process
• with opportunities for feedback
• and intervention.

3
What Monitoring is Not

• Checking-In with the teacher
• Flipping through a evidence based assessment
• Review without feedback

4
Pre-Scoring

• A detailed review of evidence-based assessments
• against scoring rules
• prior to scoring with
• limited opportunity for feedback and intervention.

5
What Pre-Scoring is Not

• A mad dash to create a collection of evidence.

6
Why Monitor and Pre-Score?

• To create better Collections of Evidence (COE)
• To ensure accountability for all participants

7
Why VAAP COEs Fail

• Omitted Reporting Category
• Duplicated Reporting Category
• Use of evidence not aligned with the ASOL
• Ungraded or incorrectly graded evidence

8
Why VAAP COEs Fail

• Missing or Incomplete SEI tags
• Photo without descriptive statement of student
  achievement
• Evidence with hand-over-hand instruction
• Evidence from group work in which the
  contribution of the VAAP participant is not clear

9
Components Monitoring

• Trained Reviewers
• Feedback Loops
• Intervention Activities

10
Monitoring Decisions
Trained Reviewers Persons with understanding of
the ASOL and program knowledge School Test
Coordinator Building Level Teams Central Office
Level Teams
11
Monitoring Decisions

• Feedback Loops
• Teachers preparing collections
• Principals
• VAAP Leaders
• Trainers/Technical Assistance
• Providers

12
Monitoring Decisions

• Intervention Activities
• Individualized Monitoring
• Attendance at Local Training
• Access to State local
• ASOL or assessment
• resources

13
Monitoring Decisions

• When will monitoring occur
• Monthly
• Grading Period
• Division Developed Timeline

14
A SAMPLE MONITORING PLAN

• Train reviewers
• Develop master list of VAAP participants by
  school, grade level, and content areas

15
A SAMPLE MONITORING PLAN

• Dispatch reviewers to schools to met with
  teachers and review VAAP Collections of Evidence
  based on master list of participants
• Review Collections of Evidence based on defense
  of ASOL.

16
A SAMPLE MONITORING PLAN

• Provide feedback teacher, principal and central
  office
• Design individual intervention(s) as appropriate
• Design division intervention(s) as appropriate

17
A Sample Monitoring/Feedback Plan
18
Pre-Scoring Activities

• Permits a review of the completed collection
  against Scoring Rules and opportunity to make
  corrections

19
Rule 1- Evidence Must Show Individual Student
Achievement

• Does the evidence show any level of achievement
  with the ASOL being defended?
• Are all of the ASOL addressed (stem and a
  bullet)?
• Has hand-over-hand (full physical assistance)
  been used?

20
Rule 2- Evidence Must Be Student-Generated

• Has evidence such as homework, open-book tests or
  group work been included?
• If the evidence is from a group project, is the
  work of the VAAP participant clearly identified?

21
Rule 2- Evidence Must Be Student-Generated

• Has the evidence been copied from the blackboard,
  textbook or computer?
• If the evidence is a work sheet, are there
  examples or directions that provide answers?

22
Rule 3 - Evidence Must Be Labeled with Correctly
Completed SEI Tags

• Does each piece of evidence have a completed SEI
  tag with the content area, ASOL and the bullet
  (if appropriate) listed?

23
Rule 4- Evidence Must Include a Completed VAAP
Content Area Cover Sheet

• Is there a completed VAAP Content Area Cover
  Sheet for each Content Area included in the COE?

24
Rule 5- Evidence Must Include a Signed
Affidavit

• Is there a signed affidavit that includes all
  teachers and other school staff who have made
  contributions to the COE?
• Has the building administrator (or designee)
  signed-off on the COE?

25
Rule 6- Evidence Must Address Accommodations,
Grading and Captions

• If accommodations have been used, are they
  documented on the evidence or on the VAAP Content
  Area Cover Sheet?
• Does each picture have a caption that clearly
  address the students level of achievement?

26
Rule 6- Evidence Must Address Accommodations,
Grading and Captions

• Has the work been graded accurately with right
  and wrong answers clearly identified?

27
Media Considerations

• Do videotapes and audiotapes have transcripts?
• Are videotapes and audiotapes clearly labeled
  with SEI tags?

28
Organizational Considerations

• Are all required state and local forms completed?
• As the evidence been organized so that scorers
  will not have difficulty finding evidence?

29
Pre-Scoring Phase One

• Building Level Review Tools
• Teacher Checklist for Collections of Evidence
• (Available in VAAP Implementation Manual)
• Administrators Checklist for Collections of
• Evidence
• (Available in Administrators Manual)

30
Pre-Scoring

• PHASE TWO
• Central Office Teams
• Specialists
• Directors
• Assistant principals
• Department chairpersons
• Expert teachers

31
Monitoring Pre-Scoring Results

• Reduces or Eliminates COEs with
• Missing Evidence
• Unacceptable Evidence
• Inaccurate or Ungraded Evidence

32
Monitoring Pre-Scoring Results

• Eliminates Surprises
• Collections not done for students educated
  outside of the division
• Incomplete collections in the division
• Collections not driven by IEP

33
Pre-Scoring Results

• Return COEs to teachers to address all
  deficiencies prior to local submission and
  regional scoring events.