Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

1
Assessing Quantitative Reasoning in Student
Writing A QuIRKy Experience

• Nathan D. Grawe
• Carleton College
• Serc.carleton.edu/quirk
• With support from the US Department of
  Educations Fund for the Improvement of
  Post-Secondary Education, the National Science
  Foundation, and the WM Keck Foundation.

2
What is QR?
3
What is QR?

• The habit of mind to consider the power and
  limitations of quantitative evidence in the
  evaluation, construction, and communication of
  arguments in public, professional, and personal
  life.

4
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set

5
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• "The test of numeracy, as of any literacy, is
  whether a person naturally uses appropriate
  skills in many different contexts"
• -National Council on Education and the
  Disciplines (2001)

6
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves argument

7
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves argument
• Deploying numbers skillfully is as important to
  communication as deploying verbs.
• -Max Frankel, The New York Times Magazine

8
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves argument
• Numbers are the principal language of public
  argument.
• -BBC Program More or Less

9
What is QR?

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves argument
• 4) QR is a habit of mind
• QR is not a discipline but a way of
  thinking. -Lynn Steen
• Achieving Quantitative Literacy

10
What do the numbers show?
11
What do the numbers show?
12
How representative is that?

13
Compared to what?
14
How was the variable operationalized?

15
Who is in the measurement sample?
Registered Voters
Eligible Voters
2006 Midterm Elections
16
Is the outcome statistically significant?
17
Whats the effect size?
18
Whats the effect size?

• Consider two statements
• A) With a plt.001 we can conclude that whites in
  MN, WI, and IA were more likely to vote for Obama
  than whites in LA, MS, and AL
• B) The fraction of whites who voted for Obama in
  MN, WI, and IA was at least 35 percentage points
  higher than that in LA, MS, and AL (lt15 vs.
  gt50) forget the p value!

19
Whats the research design (correllational or
experimental)?

• Fact Those who work with computers earn 15-20
  more than others.
• Thus, computer training may, at least in the
  short run, be a profitable investment for public
  and private job training programs.

20
Whats the research design (correllational or
experimental)?

• Other interesting returns
• Calculator 12.8
• Telephone 11.4
• Pencil/Pen 11.2
• Work while sitting 10.1

21
Controlling for what?

22
Controlling for what?

• About 50 of the differential has to do with
  different career choices.
• 25 percent involves greater time women spend on
  care-taking.
• The other 25 percent is due to bias and
  prejudice.

23
What is QR?

• sophisticated reasoning with elementary
  mathematics more than elementary reasoning with
  sophisticated mathematics.
• -Lynn Steen
• Achieving Quantitative Literacy

24
Implications for Assessment

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves communication
• 4) QR is a habit of mind

25
Implications for Assessment

• QR is largely absent from our current systems
  of assessment and accountability.
• -NCED (2001)

26
Implications for Assessment

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves communication
• 4) QR is a habit of mind

27
Implications for Assessment

• Four facets of QR
• 1) QR requires a basic skill set
• 2) QR demands application in context
• 3) QR involves communication
• 4) QR is a habit of mind

28
Implications for Assessment

• Application in context
• Standardized conditions are decontextualized by
  design"
• -Grant Wiggins
• 'Get Real!' Assessing
  for Quantitative Literacy

29
Implications for Assessment

• Communication
• While traditional assessment tools effectively
  measure comprehension, the ability to read
  others QR exposition does not guarantee the
  ability to engage in the creation of QR arguments.

30
Implications for Assessment

• Habit of mind
• As in book literacy, evidence of students
  ability to play the messy game of the QR
  discipline depends on seeing whether they can
  handle tasks without specific cues, prompts, or
  simplifying scaffolds from the teacher-coach or
  test designer.
• -Grant Wiggins
• 'Get Real!'
  Assessing for Quantitative
  Literacy

31
Implications for Assessment

• "QR requires creativity in assessment, since
  neither course grades nor test scores provide a
  reliable surrogate."
• -Lynn Steen
• Achieving Quantitative Literacy

32
Implications for Assessment

• "The interdisciplinary and contextual nature of
  QR cries out for a cross cutting approach."
• -Lynn Steen
• Achieving Quantitative Literacy
• We want to regularly assess student work with
  numbers and numerical ideas in the field.
• -Grant Wiggins
• 'Get Real!' Assessing
• for Quantitative Literacy

33
The QuIRK Rubric

• QuIRKs idea
• Measure QR in the natural context of papers
  written in courses across the curriculumpapers
  written for authentic purposes.

34
The QuIRK Rubric

• Sophomore Writing Portfolio
• 3-5 papers plus reflective essay written in 2 of
  the 4 college divisions
• -observation
• -analysis
• -interpretation
• -documented sources
• -thesis-driven argument

35
The QuIRK Rubric

• Writing requirement assessment
• 30 faculty members, 3 days, 450 portfolios
• QR assessment
• 6-8 faculty members, 3 days, random sample of
  portfolios, 1 paper each drawn from analysis,
  interpretation, or observation categories

36
The QuIRK Rubric

• 1. A little book-keeping

37
The QuIRK Rubric

• 2. Did the student take the paper in a
  QR-relevant direction?

38
The QuIRK Rubric

• Central Use
• Use of numbers to address a central question,
  issue, or theme
• Peripheral Use
• Use of numbers to provide useful detail, enrich
  descriptions, present background, or establish
  frames of reference

39
The QuIRK Rubric

• The importance of the periphery
• Even for works that are not inherently
  quantitative, one or two numeric facts can help
  convey the importance or context of your topic.
• -Jane Miller
• The Chicago Guide to
• Writing About Numbers

40
The QuIRK Rubric

• 3. If relevant, to what extent did the student
  actually use QR?

41
The QuIRK Rubric

• 4. If present, rate the holistic quality of the
  QR.

42
The QuIRK Rubric

• 5. Spotlight on several recurring problems

43
The QuIRK Rubric

• 6. Does assignment explicitly call for QR?

44
The QuIRK Rubric

• Inter-Rater Agreement Statistics
• QR Relevance 75
• QR Extent 82
• QR Quality 67
• Problem Characteristics 68

45
Examples from Student Work

• Example of QR-irrelevant paper
• The Maiden who Needs No Savingan analysis of
  Keats treatment of helplessness and power in La
  Belle Dame Sans Merci

46
Examples from Student Work

• Example of ineffective peripheral QR paper
• Title Untitled
• Thesis A synthesis of 3 alternative views on
  poverty provides a more complete picture than any
  1 perspective alone

47
Examples from Student Writing

192 billion
60 billion
1/65th of GNP
1/230th of GNP
48
Examples from Student Work

• Example of effective peripheral QR paper
• Title Les Banlieues dIslam
• Thesis

49
Examples from Student Work

• Example of ineffective centrally QR paper
• Title Untitled
• Thesis Based on data from a time diary and
  other sources, the American quality of life is
  better than that in less-developed societies.

50
Examples from Student Work

• Example of ineffective centrally QR paper
• Day Care and Development
• Topic Literature review of effects of day care
  participation on child development

51
Examples from Student Work

• Even though quantitative evidence is referenced
  extensively, student doesnt appreciate the real
  power of the actual numbers
• Ex The amount of time that children spent in
  day-care was positively correlated with reports
  of how many friends they had.

52
Examples from Student Work

• Example of effective centrally QR paper
• Title Modeling the Implicit Learning of
  Language Acquisition
• Thesis Even if people arent aware of the
  systems undergirding language, they implicitly
  learn the system through experience.

53
Examples from Student Work

• In phase two, the mean number of correct string
  classifications the subjects gave was 17.3 out of
  24 all but two participants correctly identified
  more than 12 strings, and the mode number of
  correct responses was 18.
• Discussion of statistical test.
• It was expected that the participants would be
  able to correctly classify more than 12 of the 24
  strings in phase two, so the result is not
  especially surprising. Still, the data provide a
  striking example of implicit learning.

54
Infusing QR Across the Curriculum

• numeracy is not something mastered in a
  single course.Thus quantitative material needs
  to permeate the curriculum, so that students
  have opportunities to practice their skills and
  see how useful they can be in understanding a
  wide range of problems.
• -Derek Bok (2006)
• authentic and enduring learningcan rarely
  succeed one course at a time.
• -Lee Shulman (1997)

55
Infusing QR Across the Curriculum

• If QR remains the responsibility solely of
  mathematics departmentsespecially if it is caged
  into a single course such as Math for Liberal
  Artsstudents will continue to see QR as
  something that happens only in the mathematics
  classroom.
• -Lynn Steen
• Achieving Quantitative Literacy

56
Infusing QR Across the Curriculum
Where are centrally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
57
Infusing QR Across the Curriculum
Where are centrally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
58
Infusing QR Across the Curriculum
Where are centrally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
59
Infusing QR Across the Curriculum
Where are centrally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
60
Infusing QR Across the Curriculum
Where are centrally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
61
Infusing QR Across the Curriculum
Where are peripherally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
62
Infusing QR Across the Curriculum
Where are peripherally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
63
Infusing QR Across the Curriculum
Where are peripherally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
64
Infusing QR Across the Curriculum
Where are peripherally relevant papers written?
(Population All papers submitted to writing
portfolio)
Legend Green Math Natural Sciences Blue
Social Sciences Red Other Dark Centrally
Relevant Medium Peripherally Relevant Light
Irrelevant
65
Infusing QR Across the Curriculum

• 30 of gen ed papers are centrally QR relevant
• Of these, nearly 50 are outside natural sciences
• Another 25 are peripherally so
• Of these, 90 are outside natural sciences and
  more than half are in Arts, Lit, Humanities

66
Infusing QR Across the Curriculum

• By QR relevance, what fraction of papers use QR
  little/none, some, and extensively?

67
Infusing QR Across the Curriculum
Weasely examplesA) The downtown areas of
large cities are often home to copious
franchises. B) Recall Daycare Paper
68
Our Curricular Response
69
Our Curricular Response
70
Our Curricular Response
71
Our Curricular Response
72
Our Curricular Response
Disciplines Currently Represented

• American Studies
• Biology (2)
• Chemistry
• Economics (5)
• Environmental Science
• English
• Fine Arts
• Geography (2)
• Geoscience
• Mathematics (12)
• History (2)
• Physics
• Political Science (8)
• Psychology (4)
• Sociology (4)

73
Our Curricular Response
Coming Soon!

• Anthropology
• Biology
• Classics
• Economics (2)
• English
• French (2)
• History (3)
• Political Science (4)
• Psychology
• Religion
• Sociology (2)

74
Our Curricular Response

• Example 1 An Ill-Structured Physics Lab
• Example 2 Intro to Lintina/o Studies Paper
• Example 3 History Paper on Indias Colonial
  Census
• Example 4 Exploring and Architectural Remodel

75
Our Curricular Response

• Example 5 Environmental History Paper
• Example 6 Development Economics Paper
• Example 7 Native-American Literature Analysis

76
Our Curricular Response

• Suggestions for Assignment Creation
• Role
• Audience
• Format
• Task
• Task as
• Ill-Structured
• Problem