Arizona Skill Standards Assessment System Update July 2009

1
Arizona Skill StandardsAssessment SystemUpdate
July 2009
2
Assessment Team

• ADE ASU
• Barbara Border Maggie Mangini Robert
  Keim
• Helen Bootsma Judy Balogh Joanne
  Bauman
• Jan Brite Charles Losh Pat Loughrin
• State Supervisors
• Stephanie Hahn Micki Hurley
• Ruth Kerr Shea Padilla
• Tracy Rexroat Lynn Tuttle
• Jason Wocjik Jimmy Wocjik
• CECi VTECS Pitsco

3
Assessment System 3 Years of Progress

• Year 1 - Getting Organized
• Year 2 - 13 Pilot Assessments
• Year 3 30 Assessments Issue certificates
• - 5 Commission Meetings
• - 7 Stakeholder Meetings

4
Assessment Participation

• Secondary Spring 2009
• 10,446 Students Assessed
• 8,407 End-of-program Students Assessed
• 4,700 Students Recd Certificate upon HS
  graduation
• Over 500 Business/Industry Representatives
  Teachers Involved in assessment content
• Postsecondary 2008-2009
• 40 Programs with National Certifications/Licensure
  s
• 8 Programs with Locally Developed Assessments

5
Industry Standards Validation/Item Bank
Development

• Quality Assurance
• Standards 20 New Areas Being Developed
• Research Provided by VTECS
• Item Banks 20 New Areas Established
• 30 Under Revision
• Validity and Reliability Processes

6
Item Development Process

• Research/Draft Standards and Measurement Criteria
  based on business/industry
• Validate Standards and Measurement Criteria with
  business/industry
• Develop Items
• Assemble and Pilot Assessment
• Analyze Pilot Results and Modify Items
• Conduct End-of-Program Assessment
• Re-analyze Results Continuous Improvement

7
The Task- Items

• Write items for a criterion- referenced
  assessment
• Verb in the question stem must be measurable
• Each item must match an indicator for a standard

8
Item Analysis

• Assessment items were flagged for further
  discussion if they fell below the following
  parameters
• Difficulty index of .45 with discrimination index
  of .3
• Difficulty index of .5 with discrimination index
  of .25

9
Acceptable Assessment Item

• Difficulty Index 0.86
• Discrimination Index 0.34
• Sample Question
• What is NOT a technique for drawing a line in a
  CAD software program?
• 325 (86.44) T-square
• 32 (8.51) Drop down menu
• 10 (2.66) Keyboard command
• 9 (2.39) Toolbar icon

10
Not Acceptable Assessment Item

• Difficulty Index 0.35
• Discrimination Index 0.14
• Sample Question
• What is a basic purpose for systematically
  organizing and maintaining drawings?
• 130 (34.57) Management of multiple projects
• 147 (39.1) Accurate interpretation
• 80 (21.28) Neatness
• 19 (5.05) Required by law

11
Management Tool

• Management of Standards/Measurement Criteria
• Item Development Origination, Performance,
  Results
• Reporting and Retrieval

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Spring 2009 Percentile Summary by Program
19
Spring 2009 Percentile Summary by Program
20
Next Steps

• Create an Online Community of Practice
• for CTE Teachers
• to Share Resources

21
(No Transcript)
22
Online Assessment Delivery

• E-SESS
• Projected System
• ADE State of Arizona Owned
• Access for ASU
• System Criteria Data Warehouse Specifications

23
Multi-State Collaboration

• March 2009 Meeting Scottsdale, Arizona
• Alignment with other states
• Potential for Involvement

24
Future Activities

• Additional Modes of Assessment
• Scenarios Soft Skills
• Performance Assessment
• Association Endorsements
• Decision Analytics

25
Benefits of the System

• Meets Federal and State Requirements
• Provides Added Value to Students
• Benchmarking to Industry Standards
• Certificate for End-of-Program Completers
• Provides Added Value to Schools
• Feedback for Program Improvement
• Provides Added Value to Employers
• Skill Documentation
• Influence on Curriculum
• Graduates Better Prepared to Meet Their Needs