Remaining True to Your Construct in ComputerBased Delivery of PerformanceBased Assessments

1
Remaining True to Your Construct in
Computer-Based Delivery of Performance-Based
Assessments

• Cheryl Alcaya
• Minnesota Department of Education
• Linda Zimmerman
• Pearson Educational Measurement

2
it is safe to say that computer-based testing
will increasingly become the testing medium of
choice in the future. In many respects, the
future is now most large-scale testing programs
are either computerized or moving toward
implementation. (Sireci Zenisky, 2006)
3
Reactions to online testing in Minnesota
1. Denial
4

• Collaboration How can we make computer-based
  testing work in K-12, large-scale assessment?
• Concern Maybe we can do it, but should we do it?
  
• Is sound assessment practice or technology
  driving development?
• Can computer delivery increase the potential for
  construct-irrelevant variance in test scores?

5
Our question for today
Can computer delivery increase the potential for
construct-irrelevant variance in test scores?
is examined through MDEs experience with
web-delivered, performance-based, assessments.
6
Test of Emerging Academic English Listening
Speaking(TEAELS )

• Why computer-delivered?
• Increase authenticity
• Use innovative item types
• Incorporate principles of Universal Design
• Reduce anxiety by allowing students to work at
  their own pace

7
Step 1 Build a prototype using innovative item
types
Listening Items

• Multiple choice
• Click a picture based on single-word prompt
• Click a picture based on descriptive paragraph
• Listen for verb tense in sentence and click past,
  present, or future button

8
Listening Items

• Multiple selection
• Click multiple pictures based on oral
  instructions
• Select and classify
• Drag-and-drop pictures and labels to create a
  poster

9
Listening Items

• Completion
• Watch science experiment (video) and fill in data
  table
• Listen to lecture and fill in blanks in
  paraphrased text

10
Speaking Items

• Elicited imitation
• Repeat sentences verbatim (machine scored)
• Short answer
• Or questions, e.g., Who flies a plane, a pilot
  or a bird? (machine scored)
• Mini-dialogues
• Question-asking task based on incomplete pictures

11
Speaking Items

• Extended response
• Oral presentation of poster completed in drag and
  drop task
• Retell a story seen in video
• Express and support an opinion
• English for specific purposes (Science)
• State and support a hypothesis
• Describe and compare data

12
Step 2 Lessons learned from limited pilot test
of prototype

• Effective, authentic, integrated listening
  speaking tasks can be designed and administered
  by computer, but they may be multidimensional.
• Innovative technologies available when running
  test locally are not practical in Internet
  environment.

13
Step 2 Lessons learned from limited pilot test
of prototype

• Many innovative item types many instructions to
  test takers.
• Technology drove some decisions, particularly in
  the case of elicited imitation and or question
  item types scored by voice recognition.

14
Step 3 Build and field test TEAELS
Listening Items

• Picture selection (MC) Single sentence audio
  prompt, picture options
• Passage comprehension (MC) Story or lecture,
  picture (K-5) or text (6-12) options
• Drag and drop

Speaking Items

• Elicited imitation
• Extended responses to picture prompts

15
Advantages and disadvantages of innovative item
types

• Construct representation
• ability of a test to fully represent all the
  knowledge, skills, and abilities inherent in the
  construct measured.
• Construct-irrelevant variance
• other attributes unintentionally measured by a
  test that affect test scores.
• Sireci and Zenisky, 2006

16
Two questions

• Does visual information in listening
  comprehension test items alter the construct?
• The EI item type is an indirect measure of
  speaking ability in second language. Is the
  trade-off between an indirect measure and the
  possibility to score speech by machine an
  acceptable alteration of the construct of
  speaking ability?

17
Visual information in listening items

• Traditionally, stimulus was read aloud or
  delivered on audio tape. All input was aural,
  responses were text-based.
• As technology is increasingly used for
  assessment, listening comprehension tests make
  use of multimedia. Input is both aural and
  visual. Multiple modes are available for
  responses.

18
Visual information in listening items

• The oral linguistic information (what we are
  testing) and the visual information interact in
  ways which we do not fully understand. Yet, in
  order to ensure construct validity, test
  developers need to take into account the
  interaction between the linguistic information
  and the visual information. (Buck, 2005)

19

• Does the choice of drawings vs. photographs alter
  the listening task?

20
Draw a triangle and color it in.
21
Get out your compass and paper, and draw a
circle.
22

• Are inferences based on listening comprehension?
  I.e., are students interpreting the linguistic
  information, the graphic information, or the
  interaction of the two?

23
Inference based on linguistic information
Next, get a calculator from the supply closet.
For the rest of this period, I want you to work
on the math problems.

• Which task are students most likely to do this
  period?
• Draw a timeline
• Solve equations
• Read an essay
• Lift weights

24
Next, get a calculator from the supply closet.
For the rest of this period, I want you to work
on the math problems.
25

• Can graphics cause students to focus attention
  inappropriately?

26
Put your books away, and take out a piece of
paper.
27
Mathematics and Science

• language factors may confound test results in
  content areas such as math and science where
  language should not play a role (Abedi,
  Courtney, Leon, 2003).

28
Mathematics for ELL

• Graphic support for language
• Sheltered English
• Common settings
• Optional audio

29
Mathematics for ELL

• Advantages to Computer-delivered Test
• Focus on one question at a time
• Color photographs and art
• Audio
• No separate answer sheet
• Visual indicator of answer selected

30
Mathematics for ELL

• Disadvantages to Computer-delivered Test
• Need to scroll to see complete item
• Need to navigate to see other items associated
  with the set
• Cant write on test booklet
• Lack of experience using computer

31
Mathematics for ELL
Sample Items
32
Science

• Given the nature of science, MDE will use a
  web-based format to administer the MCA-IIs in
  science. This format will allow students to view
  simulations of experiments and phenomena, and
  manipulate visual representations of relevant
  materials.
• All students, including students in special
  education who are capable of testing and students
  designated with Limited English Proficiency
  (LEP), must take the MCA-IIs in science.
• The assessment user interface and delivery will
  be designed to run on current hardware and
  software configurations used in Minnesota school
  districts.

33
Science

• Scenario-based
• Item Types
• Multiple-choice
• Figural response
• Hot spot (single and multiple)
• Drag and drop (click and click)
• Graphing (bar, broken line)
• Constructed response

34
Science
35
Science
36
Science
37
Science
38
Science

• Reduce variance due to lack of familiarity
• Sample Scenarios
• http//education.state.mn.us/mde/Accountability_Pr
  ograms/Assessment_and_Testing/Assessments/MCA_II/M
  CA_II_Item_Samplers/index.html
• Tutorial

39
Things to think about

• Item Continuum

40
Things to think about

• Does computer delivery increase the potential for
  construct-irrelevant variance in test scores?

41
Things to think about

• What compromises do we have to make?
• What have we learned?
• Where do we go from here?