Quantitative Literacy: Assessment

1
Quantitative Literacy Assessment
PracticeVirtual OR Meeting, 2009

• Flora McMartin, Broad-based Knowledge, Inc.
• Corrine Taylor, Wellesley College National
  Numeracy Network

2
Presenters

• Flora McMartin
• President, Broad-based Knowledge, Inc.
• Corri Taylor
• Director, Quantitative Reasoning Program,
  Wellesley College
• President, National Numeracy Network

3
Overview of the Presentation

• What is quantitative literacy (QL)?
• Some background on assessment
• Examples of QL assessment in practice

4
QL is defined as

• the ability to understand and use quantitative
  measures and inferences that allow one to
  function as a responsible citizen, productive
  worker, and discerning consumer. (Bernie
  Madison)
• the ability to identify, understand and use
  quantitative arguments in everyday contexts.
  Quantitative literacy describes a habit of mind
  rather than a set of topics of a list of skills.
  (Deborah Hughes Hallett)

5
MAAs QL Competencies

• Reading and understanding quantitative info in
  graphs, tables, etc.
• Interpreting quantitative info and drawing
  appropriate inferences
• Solving problems using logic, math, statistics
• Estimating answers and checking for
  reasonableness
• Communicating quantitative info verbally,
  graphically, numerically
• Recognizing the limitations of mathematical or
  statistical models

6
QL in the Social Sciences

• Nicely captured in the NCED book Mathematics and
  Democracy The Case for Quantitative Literacy by
  Lynn Steen
• http//www.maa.org/ql/mathanddemocracy.html
• Statistics for data analysis in all social
  sciences
• Basic QR skills to be an informed citizen, voter,
  etc.

7
Assessing QL The Ws and the H

• What to assess?
• Students attitudes, behaviors skills
• Course materials, modules
• Program effectiveness
• Why assess?
• For student placement
• For enhancement of course materials
• For planning and accreditation
• To test innovation

8
Assessing QL The Ws and the H

• When to assess?
• Upon college entry and exit and later
• Before, during, and after particular courses
• How to assess?
• Using various kinds of tests
• Using portfolios and other tools
• Ensuring the assessment tools match the purpose
  and the level of QL being assessed
• Evaluating with clear rubrics

9
Some practical questions when considering
assessment

• Is the assessment feasible (timing, logistics,
  resources)
• What is the impact of the type of institution?
• What is the skill level of students?
• What course level(s) will be involved? (lower
  division, upper division)
• What type of administrative support is available?
• collecting/storing/analyzing data
• What is the scope of the assessment? (Course to
  Consortium)
• Is IRB review necessary?

10
Assessing Attitudes Behaviors

• Student Attitude Assessment developed at
  Dartmouth by J. Korey Four scales utility,
  personal growth, ability, interest
• Office of Institutional Research surveys and
  analyses of students course-taking behavior
• Continuous feedback from quantitative faculty

11
Assessing Quantitative Skills Four Examples in
Practice

• QR Placement Test for incoming students
  (Wellesley College)
• Test of two QR learning objectives for students
  who completed gen. ed. requirements (James
  Madison University)
• Rubric for QR in rhetoric in sophomore writing
  portfolios (Carleton College)
• Rubric for evaluating QL skills gained over
  college career in electronic portfolios
  (AACUs QL VALUE project)

12
Wellesleys QR Placement

• Assesses incoming students QR skills weakest
  performers on test need to take basic QR course
  before enrolling in quantitative courses
• 18 open-ended questions
• Booklet with info available at http//serc.carleto
  n.edu/files/nnn/teaching/wellesley_qr_booklet.pdf

13
JMUs QR Test

• Measures two specific learning objectives How
  well students who completed gen ed
• Use graphical, symbolic, and numerical methods to
  analyze, organize, and interpret natural
  phenomena
• Discriminate between association and causation
  how to establish causation
• Test is 26 computerized, multiple-choice
  questions
• http//serc.carleton.edu/nnn/numeracyprojects/exa
  mples/32007.html

14
Carletons QR in rhetoric

• Examine papers in sophomore writing portfolios
  for evidence of QR
• Rubric http//serc.carleton.edu/files/quirk/quirk
  _rubric.v5.doc
• For each paper, examines
• Potential relevance of QR (none, peripheral,
  central)
• Extent of QR (scale 1 to 3)
• Overall quality of implementation,
  interpretation, and communication (scale 1-4)
• Problems (ambiguous words rather than numbers
  fail to describe data collection methods fails
  to provide comparisons)

15
AACUs QL rubric

• For institutional use in evaluating and
  discussing student learning over all the college
  years
• Rubric (scale 1-4) applied to components of
  students e-portfolios
• Six QL skill areas for evaluation
  interpretation, representation, calculation,
  application/analysis, assumptions, communication
• http//www.aacu.org/value/rubrics/pdf/Quantitativ
  eLiteracy.pdf

16
Infusing QL throughout the Social Science
Curriculum

• NSF-CCLI funded partnership between ICPSR and
  Social Science Data Analysis Network (SSDAN)
• Purpose Transform undergrad instruction by
  improving the teaching of QR and the way students
  understand research in the social sciences
• Activities
• Study the impact of SSCAN and ICPSR teaching
  modules with regards to QL
• Develop new teaching resources (American
  Community Survey)
• Provide new instructional materials linked to
  ICPSR
• Build national social science community of users
  of these resources

17
Assessment Focus of Infusing QL

• Assessment tool/process development and testing
• 8 Faculty from diverse institutions developing
  and testing tools
• Mainly lower division introductory classes
• Tools map to QL student learning objectives
• Survey of faculty use/adoption of QL activities

18
Building on the AACU QL Rubric

• Communication
• Find/Identify/ Generate Data
• Research design
• Confidence
• Content learning outcomes

• Interpretation
• Representation
• Calculation
• Analysis
• Method selection
• Estimation/ Reasonableness Checks

Items in blue direct map to AACU Rubric Items
in green indirect map
19
Using Rubrics to Plan for Teaching and Assessing
20
Sample Assessment Plan
Adapted from Jill Boumas Assessment Plan
21
A Short Digression

• A short digression to Blooms taxonomy a
  thinking skills framework
• Why is this important?
• Practical course level, student level,
  expectations
• Methodological qualitative/quantitative

22
Assessment Tools
Tool must collect data appropriate to level
(Bloom) and Learning Outcome (Rubric)

• Lower-order

• Higher-order

• Quiz
• Multiple choice
• Definition
• List
• Problems
• Examples
• Simulations
• Performance

• Report/in-depth paper
• Chart/outline
• Debate, panel
• Investigation
• Evaluation
• Media product
• Art work

23
Sample Rubric for a Paper
Adapted from Jill Boumas grading rubric
24
Final Thoughts

• Wrap-up
• Questions from you