Online Testing Initiatives in North Carolina

1
(No Transcript)
2
Online Testing Initiatives in North Carolina

• Presenters
• Randy Craven, Director for Information
  Technology, Technical Outreach for Public Schools
  (TOPS), NC State University
• Laura Kramer, Senior Psychometrician, Division
  of Accountability Services, NCDPI
• Scott Ragsdale, Project Manager, Online Test of
  Computer Skills, Technical Outreach for Public
  Schools (TOPS), NC State University

3

• Modern science has been a voyage into the
  unknown, with a lesson in humility waiting at
  every stop. Many passengers would rather have
  stayed home.
• Carl Sagan

4

• As North Carolina students prepare for the 21st
  century, the states accountability program is
  also looking to the future for new ways of
  assessing student achievement. This session will
  explore online testing and the use of authentic
  or performance tasks to assess student skills and
  knowledge. Specific topics include Online Test
  of Computer Skills, Online End of Course Test of
  Physics, optional online EOC testing, procedural
  changes such as pre-registering using
  NCRegistration.

5
What Initiatives?

• Online Test of Computer Skills
• End-of-Course Online Physics Field Test
• End-of-Course Tests (optional 06-07)
• Writing Online Special Study
• Online formative assessments (i.e., ClassScape)
• Other online resourced assessments
• NCEXTEND1
• NCCLAS
• Other web applications supporting test
  development/maintenance
• Online Item Writing System
• Experimental Section Review (ESR)

6
Online Assessments Success

• The Online Test of Computer Skills was successful
  in its initial administration year of 2005-06 and
  continues to be successful this year.
• NCCLAS online data collection was successful in
  2005-06 and continues to work this year.
• Online Physics Field Test was administered to
  over 3,700 students during the fall
  administration with no significant issues.
• For 2006-07, NCEXTEND1 students have been matched
  to their assessors and have had their Decision
  Tree processes completed.

7
Elements of Online Assessments

• More interactive
• Navigation features Section breaks
• Test Simulations required to be available
• Opportunity for increased test administration
  standardization
• More data can be collected
• Opportunity to expand the use of innovative items
• Virtual/performance based items
• Short and/or extended response
• Quicker data processing and reporting
• Opportunity for improvements in data accuracy
• Reduction of materials management
• Students have a positive attitude for online
  testing

8
Elements of Online Assessments (continued)

• Technology issues
• Internet aware applications (NCDesk)
• Java version
• Browser based (EOC Physics)
• Session State Cookies
• Javascript
• Flashpaper / Interactive Flash Items / Flash
  player
• Versions
• New management processes systems
• Security
• Systems design
• Systems maturation rate flexibility

9
Performance Components

• Online Test of Computer Skills
• 50 multiple choice (within an online interface)
• 50 performance (within an online interface with
  simplified applications)
• Physics Online
• Mostly multiple choice (within an online
  interface)
• Small number of performance-based items using
  collections of variables
• EOC Online
• All multiple-choice (within a online interface
  having item views that match paper-pencil tests)
• Online Writing Special Study
• Single prompt with essay response

10
Performance Scoring Models for Online Assessments

• Pattern matching Student response compared to a
  pattern of expected valid response(s) to
  determine score
• Variable set analysis Variables returned are
  analyzed against a set of valid values to
  determine score
• Writing
• Automatic Mechanics analysis (spelling grammar)
• Artificial intelligence
• Distributed human scoring

11
Preservation of State

• Students expect responses to items to retain
  state as they move from one item to another and
  if they return to an item, it will be as they had
  left it before.
• Preservation of state is not always easy since
  online assessment systems must be able to recover
  from disconnection.
• At this time for the Online Test of Computer
  Skills and the EOC Online Tests we are able to
  manage state for all items except the item the
  student is currently working on.

12
Problems/Opportunities

• Rate of technology change
• Window of opportunity for testing
• Scheduling -- EOCs are given on one day
• Having enough
• Bandwidth
• Computers
• Space
• Time

13
Managing Things

• NCRegistration
• User Accounts
• Student Registrations
• Scheduling Sessions
• Receiving Reports
• Tagger
• Student Registrations

14
Educator/Administrator Centric Applications

• Not directly accessed by students being
  administered these assessments
• NCCLAS
• Data Collection
• Special Study Data Collection
• NCEXTEND1
• Assessment Management
• Task Delivery
• Assessment Delivery
• Data Collection

15

• "There is no reason anyone would want a computer
  in their home."
• Ken Olson,
• President, chairman and founder of
  Digital Equipment Corporation (DEC), maker of
  business minicomputers, arguing against
  the PC in 1977.

16
Online Test of Computer SkillsNotable
Preliminary/Unofficial Numbers from Fall 2006

• 205 students tested in one day at one school (for
  05-06 this was 139)
• 52 schools tested 100 or more students in one day
  (for 05-06 this was 7)
• 5,068 maximum testing sessions used in a single
  day (for 05-06 this was 3,063)
• 23 days of testing in which volume of used
  sessions was 3,000 or more
• 133,980 total students tested
• 106,035 8th grade students tested
• 2,228 different test administrators with 52
  logging in 300 or more students
• 3,446 students had the Computer Reads Aloud
  option

17
Online Progress Indicators
18
Platform Shifting
19
Online Computer SkillsItems for Spring 2007

• NCRegistration SIQ Rollover available on 2/5/07
• NCRegistration session scheduling open on 2/5/07
• After first week embargo for analysis, then
  scoring continues overnight with reporting next
  day
• Window 3/12 6/8
• LIR (last in reported) Student Results -
  NCRegistration ISR and Labels output not additive
  new testing results in the same subject
  (computer skills) replace results from earlier
  window (fall).
• NCRegistration CSV Score Report current window
  activity only
• JAWS compatibility not tested/approved also
  newer versions of JAWS released since last tested

20

• Even if you're on the right track you'll get run
  over if you just sit there.
• Will Rogers

21
NCDesk Moving Forward

• Adjustments for accommodating delivery of Writing
  assessments online
• Assessment selection on login page
• Programmatically remove spell check functions
• Programmatically remove image insertion functions
• Other features deemed necessary
• Additional features for testing audio
  functionality
• Additional features for viewing correct responses
  for the Test Simulation possible

22
Computer Skills (1998 Curriculum Test Edition
2)(precursor to Online Computer Skills)Moving
Forward (Next Year) Proposals

• Should have far reduced participation for next
  year (only 11th and 12th graders not meeting the
  standard will be eligible)
• Propose to change the window to match the online
  windows
• Whole schools completed would be eligible to be
  shipped to TOPS for scoring
• First-come-first-serve scoring except for last
  month senior materials.
• Last month seniors scored and reported ASAP
• School reporting as scoring completes for entire
  school.
• Reporting on an as finished basis, with a goal of
  10 working days from receipt at TOPS

23

• It worked.
• J. Robert Oppenheimer
• Exclamation at the atomic bomb test
  according to his brother (1945).

24
EOC Online Physics Field Test

• Fall 2006
• 3,733 Students Tested
• New blocking added for incorrect Flash versions
• Spring 2007
• Registration and Scheduling now available
• No Reports (Field Test)

25
EOC Online PhysicsMoving Forward (Next Year)

• Interim standards will be set on the fall 2007
  operational test data.
• In the fall block (2007), scores will be delayed.
  
• The recommended Interim Standards will go to the
  State Board for approval in February 2008.
• In spring 2008, scoring will be reasonably
  immediate after QC checks are complete, and
  students will be held to the INTERIM standard.
• After an entire year is completed (2007-08),
  additional standard setting methodologies will be
  employed to confirm or shift the interim
  standards.
• In fall 2008, the final standards will be
  approved by the board and in place for students
  who take the EOC in 2008-09 and beyond until we
  change things again.

26
EOC Online (Optional)

• Fall 2006
• First time EOCs offered online as an option
• One school participated in Algebra I
• No problems reported
• Students enjoyed it
• Spring 2007
• Registration Scheduling now available
• Not included
• Chemistry Field Test
• Physical Science Field Test
• Biology Field Test

27
EOC Online (Optional)Next Steps

• Scoring for Online EOC Total Multiple-Choice
  Tests that have established standards set by the
  SBE
• Once data are certified, then results will be
  immediately available to LEA Test Coordinators
  upon student completion of the test via the
  NCRegistration system.
• Early testers will have a slight delay (week?)
• Add ISRs Label output to NCRegistration for EOCs

28
EOCs Online Moving Forward (Next Year)

• Making the answer choices clickable
• Improvement of the calculator
• Moveable/draggable on screen (within test
  interface)
• Add graphing capability
• Improvement of online reference content
• Moveable/draggable on screen (within test
  interface)
• Attempt to improve item view state preservation
• Position
• Zoom percentage
• Optional to continue until such time as it is no
  longer feasible to maintain paper-and-pencil
  testing.

29
NCEXTEND2 Online 2007-08

• Process under way to present all NCEXTEND2 in an
  online format
• Format similar to EOC Online
• Format for Writing would be delivered using
  NCDesk with a single editor item.

30
NCRegistrationNext Steps

• Management features to be improved
• May begin work on tighter user account
  integration across many online projects
  (including TNN)
• Improvement of speed on SIQ entry, screen (ajax
  elements) performance, etc.
• Adding query/sort order control features to
  report output (ISR Labels)
• A good percentage of sessions scheduled continue
  to be left unused we are looking into ways of
  reducing this and may seed the Maximum
  Concurrent Students Testing in the School
  Profiles with values from data collected from
  Computer Skills to limit the number of students
  that can be scheduled for a start time window.
• Report output enhancements
• Control by query report output results
• Possibly adding different output formats
  (XML/SIF, others)

31

• Computers are useless. They can only give you
  answers.
• Pablo Picasso