Collegiate Learning Assessment CLA

1
Collegiate Learning Assessment (CLA)

• Stephen KleinCouncil for Aid to Education (CAE)
• October 10, 2007
• University of California, Riverside

2
Overview

• Principles driving the CLA
• CLAs distinguishing features
• CLAs measures
• Reporting results

3
Principles Driving the CLA

• Cannot assess everything that is important
• Can measure some important abilities
• Need benchmarks to assess progress
• Use results to support improvement and curricular
  reforms
• Use psychometrically sound measures

4
1. One measure is not sufficient

• Colleges have different missions
• Students have different majors
• Multiple important outcomes require multiple
  measures

5
2. Can measure some important abilities

• Competencies listed in college mission statements
  are applicable to all students
• Writing clear, organized, etc.
• Critical thinking
• Analytic reasoning
• Problem solving
• Integrated rather than separate abilities
• Cut across academic majors

6
To produce students with 21st century skills, we
need to measure and value those skills

• Content knowledge tests are not the solution
  students will change jobs often, the content
  changes, and little agreement on what content to
  measure
• General education skills are important but they
  are not sufficient in a global market

7
3. Need benchmarks to assess progress

• How much did our students improve from the time
  they entered college?
• Is that degree of improvement comparable to that
  obtained by equally able students at other
  colleges?

8
4. Use results to

• Augment local assessment efforts
• Improve learning and instruction
• Identify effective practices
• Influence what is taught and learned, and how
  students are evaluated
• Do NOT use results to rank, rate, or punish
  schools

9
5. Use psychometrically sound tests

• Valid assess important skills (and not
  something else)
• Reliable consistent (not chance)
• Fair standardized, calibrated, and unbiased
  measures
• Cost effective in terms of testing time and
  expense

10
Some of CLAs Distinguishing Features

• Rely on open-ended tests that are
• realistic work samples
• engaging
• applicable across academic majors
• College (not student) is the unit of analysis
• Sample students within schools
• No arbitrary standards, such as proficient
• Paperless administration, scoring, reporting
• Control for differences in input between schools
• Report results in terms of value-added

11
CLA Is Not Like NCLB

• Constructed rather than selected response tests
• Use assessment for improvement (low stakes)
  rather than accountability (high stakes)
• Same tests used nationwide rather than a
  patchwork of measures and standards
• Matrix sampling of measures (like NAEP) rather
  than require that all students take all measures
• Test sample of students at each school
• Schools and students volunteer to participate
• Results released to the institution, not published

12
CLAs Measures

• Analytic writing (essay) prompts
• Make-an-argument (45 minutes)
• Break-an-argument (30 minutes)
• Performance tasks (90 minutes)
• Document based
• Contextualized questions
• Split screen/dialogue box format
• Analytic and holistic scoring
• Background questionnaire

13
Make-An-Argument Prompt

• In our time, specialists of all kinds are
  highly overrated. We need more generalists
  people who can provide broad perspectives.
• Directions 45 Minutes, agree or disagree and
  explain the reasons for your position. Student
  selects one of two prompts to answer.

14
Break-An-Argument Prompt

• Students are asked to discuss how well reasoned
  they find an argument to be (rather than simply
  agreeing or disagreeing with it).
• A respected professional journal with a
  readership that includes school principals
  published the results of a two-year study on
  childhood obesity. This study sampled 50
  children, ages 5-11, from Jefferson School. A
  fast food restaurant opened near the school just
  before the study began. After two years, students
  who remained in the sample were more likely to be
  overweightrelative to the national average.
  Based on this study, the principal of Jones
  Elementary School decided to address her schools
  obesity problem by opposing the opening of any
  fast food restaurants near her school.

15
Performance Tasks

• 90-minute real life problems
• General directions and context
• Need to combine information from different
  types of sources
• A few open-ended questions, no one right
  answermust explain rationale
• Split screen
• Right Document Library
• Left Question and answer block

16
Swiftaire 235

• You advise the president of DynaTech
• DynaTech makes airplane navigation and
  communication equipment
• DynaTechs sales manager suggests buying a
  Swiftaire 235 to visit customers and demo its
  products
• Recent accident wing came off in flight
• Students tasks
• Review document library
• Write memo discussing pros and cons
• of DynaTech getting a Swiftaire 235
• Justify your recommendations

17
Document Library

• Newspaper article about recent accident
• Transcript of interview about accidents
• DynaTech email exchanges regarding reasons to buy
  a Swiftaire 235
• Trade magazine article that compares Swiftaire
  235s performance and safety characteristics to
  similar planes
• Manufacturer specifications and required pilot
  training for Swiftaire 180 and 235
• Charts about airplane accidents and sales

18
Scoring Rubric

• Writing skills clear, organized, persuasive
• Analysis, problem solving, reasoning skills
• Integrates information from different sources
• Recognize flaws and issuesnot swayed by
  emotional arguments, faulty logic, irrelevant
  information, etc.
• Anticipates consequences and implications of
  alternative solutions and strategies
• See pros and cons of competing explanations,
  points of view, and arguments
• Weighs evidence based on its credibility

19
Two Definitions of Value-Added

• Do seniors score higher than freshmen with the
  same SAT scores?
• Does a colleges seniors score higher than what
  would be expected given (1) their average SAT
  score and (2) the typical relationship between
  average SAT and CLA scores at other colleges?

20
Fig. 1 Relationship Between Mean ACT Scores and
Mean Total CLA Scores for Freshmen
31
Your Institution (Freshmen) Others (Freshmen)
27
CLAScore
23
Regression Intercept 8.02 Slope 0.66 R-square 0.80
19
15
15
19
23
27
31
ACT Score
21
Fig. 2 Relationship Between Mean ACT Scores and
Mean Total CLA Scores for Seniors
31
Your Institution (Seniors) Others (Seniors)
27
CLAScore
23
Regression Intercept 11.96 Slope 0.62 R-square 0.7
5
19
15
15
19
23
27
31
ACT Score
22
Fig. 3 Relationship Between Mean ACT Scores and
Mean Total CLA Scores for Freshmen and Seniors
31
27
CLAScore
23
19
15
15
19
23
27
31
ACT Score
23
Frequently Asked Questions

• Does maturation affect scores?
• Does student motivation affect scores?
• Does academic major interact with the prompts
  context (e.g., science versus business)?
• Why not use graduate or professional school
  admissions tests, the NSSE, or portfolios?

24
Steve_at_gansk.comwww.cae.org