Putting it All Together and Bringing it Home

1
Putting it All Togetherand Bringing it Home
2
Workshop Flow

• The construct of MKT
• Gain familiarity with the construct of MKT
• Examine available MKT instruments in the field
• Assessment Design
• Gain familiarity with the Evidence-Centered
  Design approach
• Begin to design a framework for your own
  assessment
• Assessment Development
• Begin to create your own assessment items in line
  with your framework
• Assessment Validation
• Learn basic tools for how to refine and validate
  an assessment
• Plan next steps for using assessments

3
Assembly of Final Version of Test

• Select refined items that
• Are aligned with content and process laid out in
  the test specification (i.e., via expert panel
  review)
• Perform appropriately in think-alouds and pilot
  testing
• Perform appropriately in statistical analyses
  (results of field testing IRT analyses,
  coefficient alpha, factor analysis)
• Layout test booklet
• Sequence of items so that a few easy items appear
  in the beginning and then items of different
  levels of difficulty are distributed throughout
  the form
• Create multiple forms, if applicable
• Review and finalize test administration directions

4
Purpose of Technical Documentation

• To provide test users with the information needed
  to make sound judgments about the nature and
  quality of the test, the resulting scores, and
  the interpretations based on the test scores

5
Technical Documentation Standards

• Provide information in the technical
  documentation on the following
• Nature of the test
• Intended use
• Description of test development process
• Technical information related to scoring and
  interpretation
• Technical information related to evidence of
  validity and reliability
• Guidelines for test administration and
  interpretation

6
Some Examples of Technical Documentation

• The NAEP 1998 Technical Report This report
  provides details on the instrument development,
  sample design, data collection, and data analysis
  procedures for the 1998 National Assessment of
  Educational Progress (NAEP) in the subject areas
  of reading, writing, and civics. Nearly 448,000
  public- and nonpublic-school students across the
  United States were assessed at the national
  (grades 4, 8, and 12) and state (grades 4 and 8)
  levels.
• http//nces.ed.gov/pubsearch/pubsinfo.asp?pubid20
  01509
• TIMSS 2003 Technical Report
• http//timss.bc.edu/PDF/t03_download/T03TECHRPT.pd
  f

7
Some Examples of Technical Documentation

• Behar, L. B. Stringfield, S. (1974). A behavior
  rating scale for the preschool child.
  Developmental Psychology, 10, 601-610.
• College Board (2005). Score range. Retrieved
  October 1, 2005 from College Board SAT Website
  http//www.collegeboard.com/student/testing/SAT/sc
  ores/understanding/scorerange.html
• Midgley, C., Maehr, M. L., Hruda, L. Z.,
  Anderman, E., Anderman, L., Freeman, K. E. et al.
  (2000). The Manual for the Patterns of Adaptive
  Learning Scales. Ann Arbor University of
  Michigan.
• Pintrich, P. R., Smith, D. A. F., Garcia, T.,
  McKeachie, W. J. (1991). A manual for the use of
  the Motivated Strategies for Learning
  Questionnaire (MSLQ) (Tech. Rep. No. 91-B004).
  Ann Arbor University of Michigan School of
  Education.
• Wagner, R. K., Torgeson, J. K., Rashotte, C. A.
  (1999). Comprehensive Test of Phonological
  Processing (CTOPP). Austin, TX Pro-Ed.
• Weinstein, C. E., Schulte, A., Palmer, D.
  (1987). LASSI Learning and Study Strategies
  Inventory. Clearwater Florida H H.

8
Important Caution

• Our emphasis is on evaluating programs, not
  individual teachers
• We are not discussing assessments that will have
  consequences for individual teacher evaluation or
  employment decisions

9
Resources for Assessment Development

• Personnel
• Money
• Time
• It will take as little or as much of these as you
  can put into it.

10
Types of Personnel
TEST OPERATIONS EXPERTS
ASSESSMENT DESIGN SPECIALISTS
Define item Template
(Define Test Specs)
Assemble Test
CONTENT EXPERTS
Define item Specs
Domain Analysis
RESEARCH PARTICIPANTS
Document Technical Info
CONTENT EXPERTS
Domain Modeling (Design Pattern)
PSYCHOMETRICIAN
11
Selected Major Time Considerations
Define item Template
DEVELOPING AND REFINING ITEMS
(Define Test Specs)
Assemble Test
Define item Specs
EXPERT PANEL REVIEWS THINK-ALOUDS PILOT
TESTING FIELD TESTING DUPLICATION, SCORING, DATA
ANALYSIS
Domain Analysis
DOMAIN MODELING
Document Technical Info
Domain Modeling (Design Pattern)
12
Financial Considerations
LABOR
OUTSIDE EXPERTISE
Define item Template
(Define Test Specs)
OUTSIDE EXPERTISE
Assemble Test
Define item Specs
Domain Analysis
PARTICIPANT STIPENDS
MATERIALS, DUPLICATION, SCANNING
Document Technical Info
Domain Modeling (Design Pattern)
SCORING
DATA ANALYSIS
13
Activity 6Bringing this to Your Work

• Reflect on what we have discussed in this
  workshop
• MKT
• Assessment Design
• Assessment Development
• Assessment Validation
• Suggested questions are on the activity sheet
• Following, please share

14
Sharing

• What are the important points you will bring home
  from this workshop to your own assessment
  development processes?
• What questions did you have at the beginning of
  the workshop? Are they answered?
• What will you do differently in your assessment
  processes?
• Were there any ah-ha moments for you in this
  workshop?
• What challenges do you expect to face?
• What questions do you still have?
• What are your next steps?

15
Feedback
Nikki Shechtman nicole.shechtman_at_sri.com
Teresa Lara-Meloy teresa.lara-meloy_at_sri.com
SRI International