A Network Approach to Improving Writing Assessment in Canada

1
A Network Approach to Improving Writing
Assessment in Canada

• David H Slomp
• University of Alberta

2
Background

• Slomp, D. (2005). Teaching and Assessing
  Language Skills Defining the knowledge that
  matters. English Teaching Practice and
  Critique 4(3), 141 155.
• CLLRNet Post-Graduate Scholarship Project
• Analysis of Albertas Grade 3, 6, 9, 12
  Provincial Writing Assessments and its
  Provincial ELA Programs of Studies
• Doctoral Research
• Holding Tests Accountable A multi-method
  investigation into Students and Teachers
  experiences with high-stakes writing assessment.

3
Writers on Writing

• The way she discovers . . .truth with words . . .
  is lovely to watch. The essay moves out of the
  language of the everyday world and into that of
  literature it has rhythm, rhetorical repetition,
  and symmetry in it objects become images, which
  become metaphors. It has a powerful climactic
  order and a dramatic sense of closure. It is in
  short, an achievement of the writers craft which
  I admire very much. (Rebecca Faery, in Coles and
  Volpat, 1985, p. 335)

4
Writers on Writing

• There are in fact many kinds of excellence in
  writing that are not represented in this piece
  there is no fine writing, no flights of rhetoric,
  no impassioned argument to praise what is here
  is not to cast doubt on those other virtues, but
  rather to propose a criterion appropriate in
  judging all forms of writing What does the
  writing achieve for the writer? (James Britton,
  in Coles and Volpat, 1985, p. 79)

5
Writers on Writing

• This essay is good then because it is an attempt
  to find a logic and a structure adequate to the
  rhetorical situation that he confronted. He
  wanted to argue something, and he accepted the
  responsibility for arguing it in such a way that
  the needs of his audience would be met. (John
  Gage, in Coles and Volpat, 1985, p. 103)

6
Writers on Writing

• This essay is not artistic writing, nor does it
  convey a strong emotional message or a strikingly
  new perspective on its topic. Yet I think it is
  good writing, even very good writing, given its
  circumstances. For the essay is examination
  writing, test writing performed on demand to
  demonstrate competence in written argumentation.
  The college underclassman who wrote it did so in
  two hours on an assigned topic not announced
  ahead of time, with no compositional aids other
  than a dictionary, knowing his performance would
  be judged as a one-time test of his writing
  ability. In this test setting, the writer has
  produced a 600-word discussion cast in seven
  well-developed paragraphs simply begun and just
  as simple ended, mechanically almost flawless,
  and expressed in sentences of sensible content
  and mature form. (John Mellon, in Coles and
  Volpat, 1985, p. 130)

7
Contextual Factors Influencing Definitions of
Good Writing

• The writing context
• Classroom context
• Assignment design
• Selected Genre
• Individual perspectives/values
• Research/Teaching/Literature Background
• Personal Preferences

8
Freewrite

• Drawing on your experiences and/or research
  perspective, answer the following question
• What are the qualities, skills, and/or processes
  that either lead to or exemplify good writing?
• Table Discussion
• create a list on flip chart paper

9
Accountability in Canadian Education

• Accountability in Canadian Education has been
  pursued through test-based accountability systems
  in all but one Canadian Province (Volante, 2006
  Jaafar, 2006).
• Literacy and Mathematics assessments are core
  elements of these systems (Volante, 2006).

10
Accountability in Canadian Education

• Two main goals of accountability in Canadian
  Education
• To hold students individually accountable for
  their learning.
• To improve systems of education (McEwen, 1995
  Jaafar, 2006)

11
The Ethics of Accountability in Canadian
Education

• Large scale assessments used in Canada often
  lack strong reliability and validity data and are
  often developed without consideration of accepted
  test development standards to assure psychometric
  soundness (Miles Lee, 2002). Very few provinces
  currently provide any documentation to the public
  regarding the reliability and validity of their
  assessments, or report having data on these
  issues that is only available internally. While
  lacking this data, such assessments are still
  used to make decisions about individual students
  and the effectiveness of schools and teachers
  with regard to the achievement of children
  (Crundwell, 2005, p 2).

12
Validity Keys to Ethics in Assessment

• Validity relates the quality of inferences one
  can draw from test results. It hinges on the
  question
• To what degree does the test measure what it
  purports to measure?

13
Validity Keys to Ethics in Assessment

• Construct The skill set or knowledge domain
  that the test is designed to tap.
• A writing assessment, for example needs to
  capture both the product qualities and the
  process attributes that define good writing, both
  as a skill and a product.

14
The Ethics of Testing

• One implication of the. . . formulation is that
  both meanings and values, as well as both test
  interpretation and test use, are intertwined in
  the validation process. Thus, validity and
  values are one imperative, not two, and test
  validation implicates both the science and the
  ethics of assessment. (Messick, 1989, p. 26)

15
The Ethics of Testng

• Using test scores that work in practice
  without some understanding of what they mean is
  like using a drug that works without knowing its
  properties and reactions (Messick, 1989, p. 8).

16
Table Activity

• At each table you will find 1 of 4 provincial
  grade 3 or 4 writing tests.
• As a group, deconstruct the test at your table,
  answering the following question
• What skills, processes, and/or product features
  is this test measuring/valuing?

17
Table Activity 1 Example

• Critical literacy skills
• Ability to generate original, personal response
  to original text.
• Ability to generate original, critical response
  to previously studied text
• Time
• Ability to write 2 essays in 3 hours

18
Challenges to Construct Validity

• Construct Under-representation
• Occurs when key elements of the construct are not
  measured by the test.
• Construct irrelevant-variance
• Occurs when skills or knowledge outside the
  construct influence test scores.

19
Table Activity 3

• Compare your first list of skills/knowledge/produc
  t attributes which define good writing with the
  list of skills/knowledge/product attributes
  measured or valued by the exam.
• If they exist, list instances of potential
  construct under-representation and construct
  irrelevant variance.

20
Does Validity Matter? The Consequences of
Assessment

• This is an important accountability function.
  Not only do the examinations provide quality
  control at the individual student level, they
  also control curriculum content. As one
  commentator noted, . . . we dont need a program
  to help people implement new curriculum, all we
  need is an examination (Anderson, 2006, p 5).

21
Does Validity Matter? The Consequences of
Assessment

• What is assessed becomes what is valued, which
  becomes what is taught. (McEwen, 1995, p 42).

22
Table Activity 4

• What lessons does your test teach students about
  writing?
• Skills?
• Process?
• Product Attributes?
• How are these lessons similar to or different
  from the lessons you would want them to learn?

23
Does Validity Matter?

• When writing assessments are done in a single
  one-hour sitting, we teach that writing is not
  revision but jottings. When we ask students to
  write in silence we teach that you dont need
  an editor and that your intent matters more than
  your effect. And when we use rubrics that score
  only for obedience to rules of syntax and logic
  we teach that writing is meant to be bland and
  perfunctorydone for someone elses benefit
  (Wiggins, 1994, p 130).

24
Does Validity Matter

• Truncated thinking appears as usual a classroom
  process in Illinois and Texas for a variety of
  reasons. First, teachers imitate the state
  assessment prompts to prepare their students for
  the assessment. Second, the prompt is such that
  no evidence is available to the writers. Third,
  the criteria for judging the papers do not call
  for evidence (only support). Fourth, support is
  interpreted to include statements that reiterate
  or expand upon claims. Fifth, benchmark papers
  at the highest levels of approval incorporate
  little, if any, actual evidence. Sixth, students
  study benchmark papers as models, models that
  exemplify vacuous thinking. (Hillocks, 2002,
  201)

25
Student Writing in Alberta

• The time-restricted, impromptu nature of
  Albertas English 30-1 diploma exam primarily
  measures students' ability to create polished
  first-draft writing while the English 30-1
  writing curriculum encourages students to engage
  in a more reflective writing process. The
  different skill sets promoted by the diploma exam
  and by the curriculum make it difficult to
  ascertain what writing skills students have
  developed prior to entering university,
  especially since 50 of a students English 30-1
  mark is determined by the result of the diploma
  exam (Kwong See, Johnston, Slomp, Schneider,
  2006).

26
Student Writing in Alberta

• James One thing that really helped get me
  started on the assignment was the first thing
  that was kind of mandatory. It helped give us a
  kick in the pants. We had to do research on
  stuff from our childhood. And that got ideas
  flowing and just the nostalgia got my mind
  thinking. That was fun.As I was doing that I was
  writing down the ideas so I could use them later.
  After I had all those listed down I just started
  starting was the hardest partonce I thought of
  a line (like we had to have an anchor line which
  is the main idea), I started with that and then I
  started throwing in the ideas that I had listed,
  and as I expressed them I filled up the page and
  then the ideas kind of flowed and the entire
  thing just kind of came out once I was looking at
  the idea I had written down.
• David OK, did the ideas come while you were
  writing or before you did the writing?
• James While I did the writing. When I first
  start I have no idea what I am talking about.
• David So you develop the idea while you do the
  writing. Once youve got that first draft, or
  that first go-through done, do you go back
  through it at all or is it pretty much finished?
• James I pretty much go through it to make sure I
  didnt do any spelling or grammar errors. But
  usually when I am writing I dont like to change
  my ideas because I am in a completely different
  mind set than when I was writing it, because my
  mind is completely different about five minutes
  after I completed writing it. So I am thinking I
  just will go through it, I dont want to edit it
  too much because then it usually ends up sounding
  like my ideas werent flowing as well , so I will
  just make sure it is grammatically correct.
• David Would you say that that process is similar
  or different from the process you use for essays?
• James I use the same process.

27
Student Writing in Alberta
28
Challenges of Consequential Validity Research

• It is difficult to effectively demonstrate
  causation.
• While the writing process utilized by students
  involved in my research suggests the grade 3, 6,
  9, 12 assessments are influencing student
  writing, this evidence in itself is not
  irrefutable.

29
Network Approaches Considerations

• Accountability is driven by political rather than
  scientific concerns (Madaus, 1992 Downing, 1996
  Smith Fey, 2000 and Miles and Lee, 2002).
• Construct validity research in literacy
  assessment is difficult to conduct because the
  constructs being tested are very complex little
  consensus regarding the constructs currently
  exists (Gordon et al, 1996).
• Consequential validity research is
  under-developed and lacking in co-ordination
  (Green, 1998).
• Large-scale, multi-method case studies (defined
  at the state or provincial level) have been
  advocated as being the best method for exploring
  the consequential validity of testing programs
  (Moss, 1998, Lane, Park Stone, 1998)

30
Network Approaches Steps forward

• Collaborative approaches to defining core
  understandings of the qualities, skills, and
  processes that constitute good writing.
• Collaborative critical analysis of standardized
  writing assessments in Canada with a view to
  construct integrity.

31
Network Approaches Steps Forward

• Develop a systematic approach to investigating
  the consequences of standardized literacy
  assessments in Canada
• National research comparing teaching and learning
  of writing skills in each province. Research
  should explicitly focus on differences across
  provinces in terms of both provincial curricula
  and standardized assessment designs.

32
Network Approaches Steps Forward

• Work with psychometricians and with provincial
  Ministries of Education to develop assessments
  which support pedagogy and which encourage
  students to develop effective writing skills.