Effect of Composing Responses with a Computer on Writing Test Performance Michael Russell Technology

1
Effect of Composing Responses with a Computer on
Writing Test PerformanceMichael
RussellTechnology and Assessment Study
CollaborativeBoston College
2
Prior Research

• Past research indicates .4-1.0 effect size for
  students accustomed to writing with computer
• Limitations
• Single district (2 in Worcester, 1 in Wellesley)
• Group n ranged from 40s to 80s
• Measure of keyboarding speed only reliable
  indicator of prior use
• Insufficient number of IEP and ELL students to
  conduct sub-group analyses

3
Study Design

• Three formats
• Paper-based handwritten response
• Computer-based basic text editor
• Computer-based Microsoft Word with Spell-checker
  and grammar-checker turned on
• All responses presented in computer text with
  identical formatting
• Double-scored

4
Basic WP Interface
5
Robust WP Interface
6
Sample

• 966 8th grade students
• 6 schools, 3 states
• Approximately 12 ELL, 13 IEP
• 19 minority
• Students randomly assigned to groups within
  schools

7
Data Collected

• Writing test
• 1 hour, single prompt from Maine
• Topic Dev. (1-6) And English Conventions (1-4)
• Double-scored
• Computer Fluidity
• Computer Literacy - VA NC
• Computer Use Survey

8
Preliminary Findings

• Inter-rater Reliability
• Top Dev r.82, 55 exact, 99 within 1
• Eng Conv r.67, 59 exact, 99.8 within 1

9
Preliminary Findings
Pooled SD 2.1 1.5 3.3
10
Preliminary Findings
11
Keyboarding Speed Groups
12
Keyboarding Speed
13
Keyboarding Speed
14
Keyboarding Speed
15
Fluidity Groups
16
Fluidity
17
Fluidity
18
Fluidity
19
Fluidity
20
Literacy Groups
21
Literacy
22
Writing Use Groups
23
Writing Use
24
Gender
25
IEP Status
26
Race/Ethnicity
27
Race/Ethnicity and Gender
28
Multilingual Status
29
Summary

• Robust WP positive statistically significant and
  practical effect across all score categories
• Basic WP has small/mod effect for topic
  development, but not Eng. Conventions
• Effects larger in High Keyboarding, Fluidity,
  Literacy, and Writing Use groups, mixed in
  moderate level groups, and not present in low
  groups
• Interaction between race/ethnicity and modal
  effect, possible interaction for multilingual
  status

30
Next Steps

• Sub-group analyses for Gender, IEP, and ELL
• Consider effect of Word environment versus access
  to additional wp tools in word (e.g.,
  spell-checker and grammar-checker)
• Acknowledge under-estimate of writing achievement
  of computer savvy students by paper-based writing
  tests