Sterling Practices in Designing and Scoring PerformanceBased Exams

1
Sterling Practices in Designing and Scoring
Performance-Based Exams

• Ronald C. Rodgers, Ph.D.
• Continental Testing Services, Inc./
• Employment Research Institute

Presented at the 2005 CLEAR Annual
Conference September 15-17 Phoenix,
Arizona
2
Design Principles

• Select work products that can be tested uniformly
  for all candidates (materials, etc.)
• Blind Scoring If possible, have examiners score
  work products anonymously
• Observed Scoring Checklists of observable steps
  may limit bias if blind scoring not feasible
• Apply industry standards in scoring criteria
• Special security steps may be required

3
Sample 1 Plumbing Practical Exam

• Residential Drawing
• Three-story apartment building with fixtures
• Finish plumbing specifications for all vents and
  drains in compliance with Plumbing Code
• May use isometric or diagrammatic response
• Copper Pipe Assembly
• Bring specified 5' 2' copper tubing fittings
• Measure, cut, assemble and solder pipe left- or
  right-hand rolling offset around obstruction

4
Sample Copper Pipe Specifications
5
Blind Scoring Plumbing

• Residential Drawing (Did candidate follow Code?)
• Bathroom soil stacks, drains, vents, sizing
  (50)
• Kitchen waste stack, drains, vents, sizing
  (24)
• Basement ejector, laundry, sewer connection
  (26)
• Copper Pipe Assembly (Quality of workmanship)
• Measurement, Assembly Alignment (53)
• (deduct for each ¼? difference over ¼?)
• Pressure Test at 100 psi under water (30)
• Quality of soldering reaming (17)

6
Plumbing Practical Exam Security

• Two building drawings with same fixtures
  candidates seated with alternate drawings
• Two copper projects no two candidates with same
  project placed together in shop
• Pipe dimensions change for every test
• Must show calculations for 4 pipes not given
• Repeat candidates receive different copper pipe
  project and drawing for any retesting

7
Sample 2 Court Reporters

• Professionally recorded 5-minute opening or
  closing argument at 200 wpm
• Professionally recorded 5-minute testimony in two
  contrasting voices at 225 wpm
• Two-minute practice test in same voice(s)
• Word syllabic counts controlled for each
  15-second interval throughout dictations
• CD controls playback speed intervals

8
Blind Scoring Court Reporters

• Define scoring criteria in candidate study guide
• Wrong, omitted, added or misspelled words
• Wrong or missing dates or numbers
• Omitted capital letters for specified words
• Also specify what is NOT scored as an error
  (misplaced commas periods, spacing, etc.)
• Passing score 95 accuracy in transcript
• 50 errors or less in 200 wpm transcription
• 56 errors or less in 225 wpm transcription
• National Court Reporters Association standard

9
Court Reporters Exam Security

• Exam CDs repeat in two-year rotation
• Proctors have directions to erase digital memory
  from transcription machines
• All cell phones and recording devices must remain
  outside room during testing
• Paper-less transcription machines prohibited
  (proctors would need to erase exam from three
  digital memory sites in machine)

10
Sample 3 Carpentry/Rafters Test

• Quality and Safety Requirements
• Layout and cutting 1/16" of specifications
• Layout must be performed in required measurement
  and marking sequence
• Layout/cutting must be done within time limits
• The test must be stopped and candidate scored as
  failed if a candidate removes safety glasses
  while working on project and/or fails to follow
  any other OSHA safety procedures

11
Observed Scoring Carpentry

• Cutting
• All common rafter cuts 1/16" of plans or
  specifications
• Tracing Second Rafter
• Traced common rafter marks 1/16" of plans or
  specifications
• Followed Measurement and Marking Sequence
• Finished Cutting Tracing within Time Limit

12
Contact Information

• Ronald C. Rodgers, Ph.D.
• Director of Measurement Services,
• Continental Testing Services
• President, Employment Research Institute
• 809 Ridge Road, Suite 201
• Wilmette, IL 60091
• 847-256-5240 Fax [phone]
• rrodgers_at_qmail.com