CriterionReferenced Tests Administration

1
Criterion-Referenced TestsAdministration

• Deborah Swensen, Evaluation and Assessment,
  Director
• David Smith, Test Development, Coordinator
• Wendy Carver, Specialist
• Julie Quinn, Specialist
• Sarah Moore, Production Specialist
• Sharon Marsh, Computer Services
• Utah State Office of Education
• February 2008

2

• Welcome
• Introductions
• Post-Its

Thoughts and Ideas
3

• Legislative Update

4
UBSCT Results Update

• March 26th (Wednesday) File Arrives at USOE
• March 27th (Monday) Cleaned Electronic File to
  Districts
• March 27th April 1th (Monday, Tuesday) Data
  Clean-up

5
UBSCT Results Update

• April 2nd Paper Reports Shipped by MP
• April 8th (Tuesday) Paper Reports Arrive at
  Districts
• May 1st (Monday) State Disaggregated Report on
  Website

6
CRT Timelines

• March 21
• All Test Materials arrive at District Office
• 2 Weeks Before Testing Window
• District Pre-print File to USOE

7
CRT Timelines

• 1 Week Before Testing Window
• Answer Documents arrive at District
• Testing Window

8
CRT Timelines

• Testing Window
• CBT 48 hour raw score preliminary report
• PBT within 5 Days after Testing Window
• Answer Documents to USOE
• Answer documents not returned will count against
  participation
• All student file to USOE

9
CRT Timelines

• PBT - 4 Days after all Answer Documents arrive at
  USOE (Traditional and year round may be sent
  separately)
• CRT Raw Score Roster Report
• Electronic in move it folder
• Individual student by class
• District, school, class summary
• Teacher/Subject summary report
• Electronic in move it folder
• Error List
• Electronic in move it folder

10
CRT Timelines

• CBT end of Testing Window
• CRT Raw Score Roster Report
• Electronic in move it folder
• Individual student by class
• District, school, class summary
• Teacher/Subject summary report
• Electronic in move it folder
• Error List
• Electronic in move it folder

11
CRT Timelines Summary

• CBT raw scores 48 hour turn-around
• PBT reports 4 days after answer docs received
• CBT reports end of Testing Window

12
CRT Timelines Summary

• District submits corrected error list to USOE
  within 5 days of receipt
• Corrected raw roster sent electronically to
  district within 3 days
• 6. Combined PBT CBT Research Disk on move-it,
  within a day of receiving corrected raw roster
  error report

13
CRT Timelines

• Due Dates for Error Reports
• Submit as Early as Possible
• No Later than
• June 25nd
• Final Error Report Due for Traditional Schools
• July 6th
• Error Report Due for Year Round Schools

14
CRT Timelines

• September 15
• Written appeals due to USOE
• September 19
• U-PASS AYP Reports released to public
• November 1
• Deadline for clearinghouse corrections

15
Home and Private School Board Rule R277-604

• Private School Students MAY participate
• Payment in advance for assessment costs to USOE
• Payment in advance for administration costs (LEA)
• Participation under the supervision of LEA

16
Home and Private School Board Rule R277-604

• Home School Students SHALL be allowed to
  participate
• Must be enrolled in at least one course in public
  school
• Costs the same as public students
• Participation under the supervision of LEA

17
Home and Private School Board Rule R277-604

• Bureau of Indian Affairs (BIA) students
• Costs the same as public students
• Participation under the supervision of LEA

18
Home and Private School Board Rule

• State Costs for each test
• ELA CRT 6.00
• Math CRT 4.00
• Science CRT 4.50
• UBSCT reading 10.00
• UBSCT writing 20.00
• UBSCT math 10.00
• DWA 15.00

19
Electronic High School

• If EHS is the Students school of record they
  will administer the CRT
• If the student is making up credit, they will not
  take the CRT

20
CRT Administration Changes

• ELL students enrolled before April 15 of current
  school year and who are new to the United States
  (first year of enrollment in any U.S. school)
• and
• ELL students enrolled on or after April 15 of
  previous school year
• Must take the Science CRT (not counted in any AYP
  calculation)
• Non-Regulatory Guidance, May 2007

21
CRT Information -

• Development Team
• Whats New

22
CRT Reminders

• Remember the CRTs are an untimed test.
• Most students can complete a test section in the
  normal class period
• Allow a student to complete each section
• Make up any incomplete session (CBT and PBT)
• Do not allow a student to return to previous
  pages if they are making up an incomplete session

23
CRT Reminders

• Mark the accommodations box
• PBT top right hand corner of demographics page
• CBT mark accommodated and the accommodations
  that are being provided

24
Development

• David A. Smith
• Assessment Development Coordinator
• David.Smith_at_schools.utah.gov
• 801-538-7731

24
25
Test Administration Manuals (TAMS)

• All TAMS are now as consistent as possible in
  format and directions, given the individual
  variances of the three content areas
• TAMS are meant for test proctors/administrators
  (teachers, counselors)

25
26
Math Reference Sheets

• There are two reference sheets for the math CRTs
  this year
• One is for operational items (sections 1 and 2)
• One is for pilot items relating to the new core
  (section 3)

26
27
Math Reference Sheets

• Be sure to use the right reference sheet for the
  right sections
• The sheets are on different colored paper
• Gather the sheets used in sections 1 and 2 before
  distributing the sheet for section 3
• These instructions are clearly explained in the
  TAMs

27
28
Elementary CRT Calculator Usage

• For calculator usage per test section refer to
  the Test Overview in the TAM
• For PBT students may use their own calculator or
  one provided by the teacher
• For CBT the four function calculator will be
  available for all grades. The scientific
  calculator will be available for 6th grade

28
29
Secondary CRT Calculator Usage

• For PBT students may use their own calculator or
  one provided by the teacher
• For CBT Four function and scientific calculators
  will be available for 7th grade through Algebra 2
  with graphing calculator added for Algebra 2

29
30
Changes to Elementary ELA CRTs

• Fourth, Fifth, and Sixth Grades
• The tests are four sessions each with minimal
  teacher interaction
• After getting students started at the beginning
  of each session, it is only during the listening
  portions of the tests that teachers speak with
  the whole class
• This is a significant change, especially for
  fourth grade

30
31
Changes to Elementary ELA CRTs

• Tabs in the Test Administration Manuals
• There is only one Test Administration Manual for
  each grade. However, since this is a pilot
  year, there are multiple forms for each grade

31
32
Changes to Elementary ELA CRTs

• To accommodate the different forms, the fourth
  session of each test is located at the end of the
  Test Administration Manual with a shaded tab
  listing the form information at the edge of the
  page
• It is very important that teachers use the
  appropriate form when administering the fourth
  session

33
33
34
ELA CBT Testing

• This year passages on both the Elementary and
  Secondary ELA CRTs will be vertically aligned on
  the computer screen
• At first the student will see the passage on the
  left side of the screen
• For passages longer than the screen, only
  vertical scrolling will be required

34
35
35
January 22, 2008
CRT Trainings 2008
36
ELA CBT Testing

• Once the student reads the passage, s/he advances
  to the next screen questions for the passage
  will be positioned on the right side of the
  screen
• One question appears at a time on the screen
  until the student answers it or marks it for
  review and advances to the next question
• Passages will always be visible on the left as a
  student progresses through the questions on that
  passage

36
37
37
January 22, 2008
CRT Trainings 2008
38
2008 Elementary Test Overview
38
39
2008 Elementary Test Overview

40
2008 Elementary Test Overview

40
41
2008 Secondary Test Overview

41
42
2008 Secondary Test Overview

42
43
2008 Secondary Test Overview

43
44
Site Coordinators Manual

• Meant for the person coordinating testing in each
  school
• Three Sections
• General
• Paper-Based Testing (PBT)
• Computer-Based Testing (CBT)

44
45
Accommodations

• Wendy Carver
• Special Education Assessment Specialist
• Wendy.Carver_at_schools.utah.gov
• 801-538-7639

46
Include all students in instruction and
assessments

• Assessments without accommodations
• Assessments with accommodations
• English Language Learners
• Students with Disabilities
• Students on a 504 plan
• Utahs Alternate Assessment (UAA) for students
  with significant cognitive disabilities

47
Utahs Alternate Assessment (UAA)

• Language Arts grades 1-12
• Math grades 1-12
• Science grades 4-12
• Exempted from other U-PASS tests
• Tasks will not change this year
• Bubble sheets for UAAs new last year
• A student either takes the UAA in all subject
  areas or the CRT in all subject areas

48
Utahs Alternate Assessment (UAA)

• If a student is tested on a CRT in one subject
  and a UAA in another subject, the student will be
  non-proficient on the UAA no matter what the
  proficiency score
• Example CRT in math science UAA in Language
  Arts non-proficient on UAA
• The presumption is if a student is cognitively
  able to take the CRT in one subject, that student
  is cognitively able to take the CRT in all
  subjects

49
Utahs Alternate Assessment (UAA)

• Teacher should fill in only one task for each
  content area (language arts, math science)
• If the student has more than one math goal,
  choose only one math task for the UAA
• Accommodations have been removed from the UAA
  answer document since the UAA is individually
  accommodated

50
Accommodations

• On CRT answer documents mark accommodations in
  both places
• Non-Standard Participation - lower left corner
• Specific accommodations under Accommodations
  (ELL, IEP, 504) - upper right corner

51
Accommodations

• While teachers are still in school check the
  error reports so that the teachers can correct
  accommodation errors

52
Modifications

• Definition Modifications are changing, lowering,
  or reducing learning or assessment expectations
• May result in implications that could adversely
  affect a student throughout that individuals
  educational career
• Modified tests will NOT be counted toward
  PROFICIENCY in AYP

53
Presentation Accommodations

• Directions Read Aloud in English and Signed
• CRT - Clarification, paraphrasing and
  re-reading of directions are allowed
• Should be in a place separate from students who
  do not have this accommodation

54
Presentation Accommodations

• Questions Read Aloud in English, Signed, and
  Screen Reader
• CRT Math Science Yes
• CRT Elem. Sec. Language Arts Indicated by a n
  (will be given items that can and cannot be read
  aloud prior to CRTs)
• UBSCT Writing and Math Yes
• UBSCT Reading - NO

55
Response Accommodations

• Word Processor (No Spell Check)
• Word Processor not used on CRTs
• Grammar spell check must be disabled
• DWA - Yes
• Adult should copy to answer document
• UBSCT Writing Yes
• Adult does not need to copy to answer document

56
Response Accommodations

• Calculation Devices
• Includes abacus, arithmetic table, manipulatives
• CRT YES on all sections including
  non-calculator section
• UBSCT Math Yes

57
Response Accommodations

• Graphic Organizers Visual Organizers
• Student generated

58
Setting Accommodations

• Reduce Distractions to Student
• Change in location
• Inside classroom
• In another room in the school

59
Setting Accommodations

• Reduce Distractions to Others
• Change in location
• Students who need a reader or scribe
• Students who read or think out loud
• Students who make noises

60
Setting Accommodations

• Physical Access
• Examples
• Auditorium lacks room for assistive technology
• Large print materials may require table instead
  of desk
• A student has access to either computer-based or
  paper-and-pencil-based test administration

61
Timing/Scheduling Accommodations

• Timing/Scheduling Accommodations
• Increase the allowable length of time to complete
  a test or assignment and may also change the way
  the time is organized
• It is recommended that subtest sessions be
  completed the same day on which they are started.
  If necessary, additional completion sessions
  should be scheduled, within the LEAs determined
  testing window, and carefully proctored to ensure
  that the student does not access previously
  completed items

62
Timing/Scheduling Accommodations

• Extended Time
• Common often time and one half
• Reader or scribe often double time
• CRT it is recommended that subtests be
  completed on same day as started
• UBSCT subtests must be given on assigned day
  and completed within school day

63
Timing/Scheduling Accommodations

• Multiple Breaks
• Supervision is required for all students on
  breaks
• Schedule Change
• Best time of day
• Multiple days

64
Computer-based Testing(CBT)

• Some LEAs are saying students with disabilities
  do better on CBT than paper-based tests
• Teachers will have to read aloud to student while
  student takes computer-based test
• If student has directions read aloud
  accommodation, the directions may be paraphrased,
  clarified, etc.

65
Computer-based Testing(CBT)

• Calculator
• Elementary need to bring calculator for
  non-calculator sections of the CRT
• Secondary do not need a calculator since no
  non-calculator sections of the CRT
• Large print and Braille are not available on CBT
• No need for scribe/word processor on CBT

66
Computer-based Testing(CBT)

• If student has multiple breaks accommodation or
  needs to stop testing before the section is
  completed, make sure the student does not have
  access to item previously completed before break,
  etc.

67
Science ELL

• ELL students enrolled before April 15 of current
  school year and who are new to the United States
  (first year of enrollment in any U.S. school)
• and
• ELL students enrolled on or after April 15 of
  previous school year
• Must take the Science CRT (not counted in any AYP
  calculation)

68
Production and Distribution

• Sarah Moore
• Production Manager
• Sarah.Moore_at_schools.utah.gov
• 801-538-7862

69
Test Materials Receipt

• All materials are packaged by school and shipped
  to the district except Language Arts Grade 2
• Verify the number of boxes received
• Principals do not distribute to teachers until
  test window opens

70
District Test Materials

• Materials Receipt Acknowledgement Form
• Copies of all school packing lists
• Test Booklet Overage
• Coordinators guides

71
Verifying District Shipment

• Verify box contents using the district packing
  list in box 1 of district labeled boxes
• Do not open the shrink-wrapped packages at this
  time
• Contact USOE if the shipment appears incomplete
  or if you have any questions

72
School Test Materials

• Test Administration Manuals for CBT or PBT
• Test booklets for each subject based on ordered
  materials and mode using

73
Paper Based Test Materials

• Math, English Language Arts (ELA) and/or Science
• Math and ELA will be shipped from Pearson
• Science will be shipped from Measured Progress
• ALL Re-orders through USOE

74
Paper Based Test Materials

• Math grades 5 and up have reference sheets
  included in the booklets
• Science Physics and Chemistry Reference
  sheets are individual, included in shrink-wrapped
  packs
• Shrink-wrapped packs of 10 booklets

75
Paper Based Test Materials

• Math grade 1 None
• Math grade 2 through Algebra 2 Shipped by
  Pearson per order requests, boxed by school,
  shipped to districts for verification
• NEW THIS YEAR Re-orders through USOE, shipped
  by USOE

76
Paper Based Test Materials

• ELA grade 1 None
• ELA grade 2 Shipped from Computer Services at
  USOE after pre-print upload
• ELA grade 3 through 11 Shipped by Pearson per
  order requests, boxed by school, shipped to
  districts for verification
• NEW THIS YEAR Re-orders through USOE, shipped
  by USOE

77
Paper Based Test Materials

• Science Grades 4 Physics
• Shipped from Measured Progress per Order
  Requests, boxed by school, shipped to district
  for verification
• Reorders through USOE, shipped by USOE

78
Paper Based Test Materials

• OVERAGE
• Extra booklets included labeled for district
• Shipped to districts with single grade enrollment
  of 2000 plus automatically

79
Paper Based Test Materials

• Additional materials available through USOE web
  ordering, open April 1st.
• 48 hour turn around
• To encourage CBT participation and reduce waste.

80
Computer Based Test

• Pre-ordered through USOE
• Pre-loaded through Computer Services, USOE
• Additional students loaded on the fly
• Two TAMs should be provided for each lab
• TAMs should already be ordered through UTOS

81
Braille and Large Print

• We will include standard print booklets
• We dont provide answer documents in Braille and
  Large Print.
• Therefore, make sure answers are transferred!

82
Security

• NEW
• District Assessment Director Sign-off Guide

83
Sign-off Books for Assessment Directors
84
Pre-Prints

• Sharon Marsh
• Computer Support Specialist
• Sharon.Marsh_at_schools.utah.gov
• 801-538-7915

84
85
Testing 2007-2008 Are you ready?
85
86
Training is paying off but heres a reminder of
last years problems
86
87
Problems 2006-2007

• Not checking header sheets for teacher and
  section information
• All student files still not in on time
• Answer Documents not preprinted were not bubbled
  correctly
• UAA more than one task entered
• Tests packaged by school and not test type

87
88
Problems 2006-2007

• Both UAA and CRT test given.
• Error reports not corrected on time
• Accommodation errors
• Withdrawn students not being returned with
  regular batch to be scored causing a error on the
  missing document report
• Bar-code label for booklets not being returned on
  booklet with special code box marked

88
89
Problems 2006-2007

• Remember the test-id is printed on the bar code
  label so please remind your teachers to check for
  LA0202 for language and MA0202 for math.

89
90
Old Answer Documents 2006-2007

• Sharon Marsh

90
91
Dont Hang On To Old Answer Documents
91
January 22, 2008
CRT Trainings 2008
92
92
January 22, 2008
CRT Trainings 2008
93
93
January 22, 2008
CRT Trainings 2008
94
Trimester Testing 2007-2008

• This would be the only time you would use last
  years answer documents and you should have
  received them when you uploaded pre-print files
• This year we are going to send out the student
  reports as soon as all of the districts that have
  tested during that trimester period are finished

94
95
Trimester Testing 2007-2008

• If you have students that received an answer
  document but wont test until spring mark the
  answer document withdrawn and send it back with
  the batch to be scored

96
Do You Feel Like You Get Mixed Signals
96
97
On When to Send Files or If Im just doing CBT
why do I need to do anything?
97
98
Secure Testing Website to upload files for PP
Documents and CBT Preloads
98
99
Information on Testing Website

• Log onto the testing website
  (https//cs.schools.utah.gov/assessment)
  and click on support on the toolbar to download
• Verification Application
• How To Manual
• Assessment Pre-print File Name Specification
  File Format

99
100
Pre-Print Layout is different for PP read pg. 9
10 of the pre-print file name document
100
101
Pre-print File Layout Contd
101
102
Test File Upload

• You must send a pre-print file for every test
  that you are giving
• UAA students have to be in 3 separate files one
  for every test that they will take remember on
  science dont send in students grade 3 or below
  they will take US0112, UM0112, UL0112

102
103
Testing Answer Documents

• Pearson will send you the math 2 booklets, but
  you will send in pre-print files to create the
  bar code labels
• Language 2 booklets will be sent from USOE with
  your labels
• All math, science, and language answer documents
  will be printed from the files that you send in

103
104
Testing Answer Documents

• All UAA answer documents will be sent to you from
  pre-print files that you send to USOE even if you
  are doing everything online UAA must be done PP.

104
105
Spring 2008 Language
105
106
Spring 2008 Math
106
107
Spring 2008 Science
107
108
Spring 2008 UAA Language
108
109
109
January 22, 2008
CRT Trainings 2008
110
Spring UAA Math
110
111
Spring 2008 Science
111
112
Multiple forms for both language and math booklets
112
113
Language Arts Grade 2
114
CBT PRE-LOADS
114
115
CBT DOWNLOADS

• Use testing website to upload files
  (https//cs.schools.utah.gov/assessment)
• Files will be sent to Pearson the same day they
  arrive at USOE
• Refer to page 9 10 of the preprint file naming
  specification document the layout for CBT is
  different than PP
• Instead of PRE as part of your file name you will
  use CBT

115
116
CBT Downloads

• A paper backup is no longer necessary. UAA
  students however will need to use the new UAA
  answer document
• The 9 2 position used for CBT testing is used
  to both load students into lab classes this must
  be a numeric field
• When those files are returned to USOE for the
  official raw roster reports and raw class
  summaries it will be used as a class identifier

116
117
Managing Answer Documents

• Sharon Marsh

117
118
UALPA is batched different please call if you
have questions
119
Batching UALPA for Scoring

• Kindergarten is the only grade that does not have
  multiple grade levels or a different form. You
  cant batch different forms of the test together
  C1 cant be with C2 and you still need to batch
  by school and grade but because they have
  multiple grade levels you can send in all of your
  3-6 as long as you have a header sheet for each
  grade. If you have more than one elementary you
  can do the same for that school.

120
Batching UALPA for Scoring Contd

• All C1 grade 3-6 for your schools can be boxed
  together as long as they each have header sheets
  that have been filled out correctly all UALPA
  documents need to be cut so use a light rubber
  band to separate individual classes
• Then you would do the same for all of your other
  grades and forms
• Please see example

121
UALPA GRADE 3 6 COVER SHEET FORM C1
122
Use this header for a 3rd grade teacher
123
GRADE 3-C1
124
Use this header for a 4th grade teacher
125
GRADE 4-C1
126
UALPA Reminders

• You dont need to send in an all student file
  until the end of the year
• Please make sure to batch the different forms
  separate they wont scan if they are in the same
  batch
• You do need to use teacher number and section
• If you have UALPA teams that test assign them a
  teacher number

127
UALPA Reminders

• Please make sure that those doing the scoring
  mark the scores
• Keep the timing marks of the header sheet and the
  answer document going the same directions so that
  it is easier to cut the documents

128
WE SCAN 6 DAYS A WEEK 2 SHIFTS A DAY
128
129
Louise
Louise
Thelma
Louise
What about Bob?
129
January 22, 2008
CRT Trainings 2008
130
This is Shane.The machine is Bob.
BOB
130
January 22, 2008
CRT Trainings 2008
131
We dont mind pulling out your extra papers when
were scanning really we dont!
131
132
132
January 22, 2008
CRT Trainings 2008
133
Anything that is not part of the answer
document, STOPS the scanner
133
January 22, 2008
CRT Trainings 2008
134
The following is a list of problems that continue
to slow scanning

• Tests grouped together by school instead of by
  test
• Bar code labels put over other labels or lined
  out and used for a different student
• Photocopied answer documents substituted for
  actual test, these will not scan
• Tests mixed with other tests, e.g., language 2
  booklet in with math 2

134
135
The following is a list of problems that continue
to slow scanning

• Using only one header sheet for a teachers
  elementary classes
• Mixing 3rd and 4th grade classes together under
  one header sheet
• Sending 1st TRI answer documents in with your 2nd
  TRI or worse keeping them until SPRING. Remember
  the error reports shows your missing answer
  documents, and every spring we have new answer
  documents and test

135
136
The following is a list of problems that continue
to slow scanning

• When using blank answer documents check to see
  that all information has been bubbled in
• We still have districts that use another students
  pre-printed answer document and cross out the
  information remember every pre-printed documents
  must be returned with the special code box marked
  with the reason why the student didnt take the
  test

136
137
Why is quality screening of your various tests so
important?

• Anytime a problem is detected, scanning is halted
  until the problem is resolved, which causes
  additional delays not only for your district but
  subsequent districts that are next in line for
  their documents to be scanned and processed
• Cooperation on your part ensures that we can
  return results to you as quickly and accurately
  as possible

137
138
What happens after testing is finished?
138
139
CRT Timelines

• CBT - 48 hour preliminary raw score reports
  
• All student file to USOE when you close your
  testing session for your district

139
140
CRT Timelines

• PBT - within 5 days after Testing Window
• Answer Documents to USOE
• All student file to USOE

140
141
The following is a list of potential problems
that should be recognized

• Answer documents should be stacked with the
  student names face up with the timing tracks to
  the left. When the folded edge is cut preparatory
  to scanning, timing tracks will be cut off
  causing significant delay in scanning
• Use a light rubber band to separate individual
  teacher batches, with the header sheet on top, so
  they will stay in order when cutting the answer
  documents

141
142
The following is a list of potential problems
that should be recognized

• This year there will be different forms for the
  math 2 and language 2 booklets please make sure
  that when you package them for scoring if you
  have more than one form that you keep them
  separate. And if you would indicate on the
  outside of the box the form because they will all
  have different scan programs and this will help
  to ensure the turn around time that will be
  needed this year.

142
143
Packaging Shipping

• Please make sure that the answer documents are
  all facing the same way
• Check to see that answer documents that werent
  pre-printed have been bubbled in
• Check all header sheets and make sure that
  teacher number section number have been bubbled
  (MUST)
• Teacher number is left justified with no spaces
  in the 9 position field

143
144
Shipping Address

• Answer documents are to be delivered to your data
  technician one of two ways
• Option 1 Hand-deliver, FedEx, or UPS
  Recommended
• Utah State Office of Education
• District Computer Services
• 250 East 500 South
• Salt Lake City, UT 84111-4200

144
145
Shipping Information

• All tests must be grouped in one shipment to DCS
• All test materials need to be boxed in
  appropriately sized boxes to avoid damage
• Each box must display the following on the
  outside (short side) of the box

146
Shipping Information

• District Name
• School Number
• School Name
• Grade Level
• In small districts, multiple schools of the same
  grade level may be placed in one box. Each must
  have a control document and header sheet

147
Package Box Label Example

• DISTRICT
• TEST
• GRADE

148
Understanding Error Reports and Corrections
148
149
Edit and Scoring

• As soon as your answer documents are scanned,
  they are processed. Students that match the all
  student file and dont have any other type of
  errors will be on your raw reports. Students that
  have problems will be listed on your error
  reports
• All reports will be placed in the secure
  Move-It folder as soon as they are created

149
150
The following is a list of problems that appear
on error reports

• If you have students that have withdrawn from the
  class that is being tested mark withdrawn in the
  special code box and return with that teachers
  class
• On the math and language 2 if you have a student
  that does not test put their label on a booklet
  and mark the special code box and return with the
  teachers class this will help with your missing
  documents

150
151
Take A Close Look At Your Reports
151
January 22, 2008
CRT Trainings 2008
152
Missing Document Report

• If you dont return every document that was
  pre-printed you will see them listed by test and
  by school
• Every bar code label needs to be returned on a
  booklet and the special code box marked. This was
  a really big problem last year for lots of
  districts

152
153
Resolving Error Reports

• Any of the students that have error problems
  will not be on your raw roster or summary reports
  but will be included on the error reports
• It will be the districts responsibility to
  provide the information to make the corrections.
  In some cases it might be necessary to send
  another all student file

154
New Reports

• This year we have two new reports
• one will compare all of your pre-prints to the
  documents that are returned for scanning
• the second report will list the students that are
  missing and from which test

155
New Reports

• REMEMBER every answer document needs to be
  returned with the Special Code Box marked
• Those answer documents not returned will count
  against your participation

156
CBT Error Reports

• When your districts has completed testing the
  files will be returned by Pearson at that time we
  will match them to the all student file and
  create raw roster and raw summary reports. This
  process also will generate error reports just
  like on your PBT testing. These will be placed in
  your move-it folder.

157
Types of CBT Errors

• No match on all student file
• Absent or withdrawn with responses
• Accommodation errors
• Just like the PBT you might need to send a new
  all student file to fix your errors or you might
  need to tell us what type of accommodations were
  used

158
Types of CBT Errors

• Remind teachers that if they mark accommodated
  they must fill in all of the accommodations that
  they used
• Also if the students took any part of the test
  they were not absent
• The same applies to withdrawn
• You can tell your reports apart because they will
  say CBT on them

159
Count of Pre-Print to Scan File
160
Missing Answer Documents
161
LEA Error Report
162
LEA Test ID Error
163
Student Not on File
164
Check Scores On Summary Reports
165
Often Low Scores Can Indentify Batching Problems
166
CRT Raw Roster RPT
167
No need to stress when you know who to call for
help!
167
168
Who Do I Call?

• Sharon Marsh [phone]
• Becky Andrews [phone]
• Cindy Marshall [phone]
• Dawn West [phone]
• Shane Johnson [phone]

168
169
Getting jumpy Im almost done
169
170
Gentle Reminders for Teachers
170
171
Gentle Reminder

• Bar Code labels are printed by grade and test ID
• They need to be placed in the area that has apply
  actual student label here
• If teachers have applied their labels in any
  other area dont try and remove it

172
Non-Standard Participation

• Accommodated (ELL, IEP, 504) (1) if this is
  bubbled you must show what type of accommodations
  were used. (If this bubble is marked you must
  bubble accommodations above)
• Private or Home School (5)
• Modified (IEP Team) (2)

173
Non-Participation

• Absent-Unable to make up test (1)
• Excused-Medical Emergency (2)
• Unknown Student (3)
• Officially withdrawn from class/school (4)

174
Non-Participation

• Private or Home School (7)
• RT (8)
• UAA, Utahs Alternate Assessment (IEP team) (9)

175
What Do I need to Do?

• When teachers bubble students as accommodated
  they need to also mark the types of
  accommodations that were used
• This year you have a new error report that will
  need to be fixed when that isnt resolved
• When applying bar code labels make sure they have
  the correct subject

176
Remember if you have problems just call
176
177
Any Questions?
177
178
Equating Process

• David Smith
• Assessment Coordinator
• David.Smith_at_schools.utah.gov
• 801-538-7731

179
Equating Process

• Criterion-Referenced or standards-based tests are
  those where examinee scores are judged against an
  established criterion or standard

179
180
Equating Process

• Scores on criterion-referenced or standards-based
  tests are often reported in terms of whether the
  examinee met the standard (e.g., proficient/pass)
  or in terms of a particular criterion-based
  category (e.g., basic, advanced)

180
181
Equating Process

• There is no limit to the percentage of examinees
  scoring in a particular category. In other words,
  all can pass or all can fail.

181
182
Equating Process

• The scores that divide the performance categories
  on criterion-referenced or standards-based tests,
  referred to as cut scores, are arrived at
  through a systematic judgmental process called
  standard setting that often involves groups of
  content experts reviewing test and other data to
  make determinations about where to place the
  lines to divide the score distribution.

182
183
Equating Process

• Standard setting typically occurs early in a
  testing program and is reviewed (validated) every
  five years or so or when the program undergoes a
  change.

183
184
Equating Process

• Norm-referenced tests, on the other hand, compare
  examinee performance to that of a norm group
  (which is often intended to be a representative
  sample of the target of generalization, e.g., a
  national sample) that took the test in a
  particular year.

184
185
Equating Process

• Most norm-referenced tests are re-normed every
  seven or eight years. Student scores are usually
  reported as percentiles, or the percentage of
  students from the norming group that the student
  outscored.

185
186
Equating Process

• Linking, the umbrella term that includes test
  equating, is the mathematical process of
  connecting the scores from one test to the scale
  of another test so that valid comparisons can be
  realized.

186
187
Equating Process

• To put it simply, the equating procedure
  determines, and then attempts to equalize, the
  comparative difficulty of each test form. In
  state testing programs this involves placing the
  scores from the 2007 assessment, for example, on
  the 2006 scale.

187
188
Equating Process

• The tests each year are linked using items common
  to both test forms and then raw scores are placed
  on the scale based on the results of that
  equating procedure.

188
189
Equating Process

• In some years, the raw score necessary to reach a
  certain scale score and performance level goes
  up.
• In other years it goes down

189
190
Equating Process

• Equating is based on item characteristics and
  changing student performance, not impact
  percentages.

190
191
Equating Process

• In other words, if the students in the state
  improved dramatically from one year to another,
  the equating process will ensure that this
  improved performance is reflected in higher
  average statewide test scores.

191
192
Equating Process

• The converse is also true. The equating process
  does not limit at all the percentage of students
  that score in a particular performance category.

192
193
Ethics

• Deborah Swensen
• Assessment Director
• Deborah.Swensen_at_schools.utah.gov
• 801-538-7811

194
Computer-Based Testing (CBT)

• Julie Quinn
• Computer-Based Assessments Specialist
• Julie.Quinn_at_schools.utah.gov
• 801-538-7723

195
USOEs CBT Website

• February
• Proctoring PowerPoints
• Video Conference recordings
• December
• Proctoring and Technical PowerPoints
• Technical Training Video Conference recording
• http//schools.utah.gov/assessment/info_cbt.aspx

196
Technical WebEx Training

• All steps needed for technical set-up of CBT
• February 13th
• February 21st
• March 4th
• New March 7th
• Cancelled March 21st

• Multi-media module should be reviewed prior to
  WebExs.
• Access info is posted on the USOEs CBT website

197
Training Website

• Training Tests Educators Students
• Increase student familiarity with tools, moving
  between sections, and submitting tests.
• Increase educator familiarity with test
  administration processes
• System Verification Tests - Technical
• Large files, used to test bandwidth/hardware
• Used to verify hardware/infrastructure set-up
• Students do NOT take this test

198
Training Tests

• Encourage students to explore all tools
• Scientific/Graphing calculators
• Reference Sheets
• Help tool with animated explanations
• Option eliminator pop-up
• Go to screen between sections
• Please emphasize the difference between Save and
  Exit and Submit

199
Live Testing Website

• User ID sent to Assessment Director in February.
• This ID is DIFFERENT than the Training ID.
• Assessment Director must set up IDs for other
  users in the LEA.
• Not needed until pre-load file is submitted.
• In general, Training Site IDs had broad
  authority, but Live Site IDs have narrow
  permissions.
• All Users MUST have their own ID,
• Do not share IDs.

200
CBT Pre-Load

• The CBT pre-load file places 92 groups or
  classes with students, into the live site
  hierarchy.
• Do not manually add groups to the live site.
• ALL students MUST be accounted for.
• Submit only CBT students on the CBT pre-load.

201
TestNav URLs

• Training Tests through TestNav
• www9.etest.pearson.com/UPASSTRNG
• Live Testing through TestNav
• www9.etest.pearson.com/UPASS
• Note TestNav will remember the last URL used.
• If training tests are given, change the URL
  before students begin live testing.

202
Student Management

• ALL students must be placed into sessions
• Submitted in pre-load, or manually added
• ALL students must be stopped
• Student submits test, or proctor selects Mark
  Test Complete and enters a reason.
• ALL sessions must be stopped

203
Student Management

• Non-Standard, Non-Participation Codes
• MUST be marked BEFORE the students test is
  stopped.
• Can be identified when session is built.
• Specific accommodations must be identified if
  Accommodations drop-down is selected.

204
Student Management
205
Student Management

• Between sections, students must be Resumed.
• The test can not be accessed if a student is in
  Exited status.
• Note Anytime a student exits TestNav before
  submitting, they are placed into Exited status.

206
Submit/Exit Issues

• Between test sections
• Students Save and Exit
• At the end of the final section of the test
• Students Submit
• Students should NOT Submit their test, until
  they are finished taking ALL sections.

207
Submit/Exit Issues

• Example A student is absent on day 1 of
  testing, but take sections 2 and 3 with their
  class. At the end of section 3, students in the
  lab are instructed to Submit their tests.
  If the student missed section 1 and
  submits their test with the rest of the class, it
  must be manually fixed by calling Pearson.

208
Proctoring CBT

• Educators must actively proctor computer-based
  testing
• Proctors will not have copies of the tests to
  refer to student questions should be addressed
  using the item on their screen.
• It is recommended that 2 individuals be in each
  room during CBT administration.
• Ethics-trained educator, and another person

209
Organizing CBT

• Color-coded secure folders for each computer lab
  or classroom with
• Printed session roster
• Student authorization tickets
• Mark which sections students are absent for
• Log sheet for irregularities
• Absences, last day a student attends, etc
• 48 hour raw score reports
• Review reports and log data anomalies

210
Organizing CBT

• Coordinate with technical personnel
• Process for reporting problems during testing
• Document interruptions, error messages
• Schedule completion, make-up sessions
• Set aside time in the schools testing schedule
• Can (dont have to) move students from regular
  session rosters into session rosters just for
  finishing testing.

211
Additional Resources

• Self-guided, multi-media module for technical
  training now available through Pearsons
  website.
• Under Resources Training Resources
  Presentations
• Pearsons help-desk 800-627-7900 x 845
• Open Monday Friday, 600 AM 730 PM
• Password resetting, problems accessing websites,
  etc.

212
Thank you for participating!