Transfer Student Presentation

1
The VSA Responding to a New Era of Accountability
Accountability Today Tomorrow AACU, AASCU,
APLU Williamsburg, VA October 6-8, 2009
2
Voluntary System of Accountability (VSA)

• Initiative by public 4-year universities to
  supply basic, comparable information on the
  undergraduate student experience to important
  constituencies through a common web report the
  College Portrait.

Sponsored by APLU AASCU. Funding from Lumina.
3
VSA Goals

• Provide a useful tool for students during the
  college search process
• Assemble information that is transparent,
  comparable, and understandable
• Demonstrate accountability and stewardship to
  public
• Measure educational outcomes to identify and
  enhance effective educational practices

4
Multiple Audiences

• Prospective students and families
• High school, community college counselors
• Policymakers and elected officials
• Campus faculty and staff
• Accrediting associations, institution and state
  boards
• Taxpayers and public-at-large

5
VSA Participation

• 329 institutions from 49 states and Puerto Rico
  (10/05/09)
• 63 of 520 APLU AASCU institutions
• 250 College Portraits posted on
  www.collegeportraits.org

6
VSA Context

• Increasing disinvestment in higher education
• Perceived lack of useful and transparent data
  prevents institutions from demonstrating
  accountability and contribution to public good
  (Spellings Commission)
• Policy-makers and employers demand evidence of
  educational outcomes particularly broad
  transferable skills

7
A Community Effort

• Concept outlined in APLU white papers, refined
  through feedback from members
• Strengthened by AASCU endorsement
• Structure developed by task forces - presidents,
  provosts, student affairs, IR
• 82 members from 70 institutions
• Input from associations and larger HE community

8
Result The College Portrait
Free web report that communicates user-friendly,
consistent information about college campuses.

• Student Family Information
• Student Experiences Perceptions
• Student Learning Outcomes

9
www.collegeportraits.org
No rankings, no spin ... just the facts!
10
Student Family Information

• Institution description, highlights
• Student characteristics
• Cost of attendance, net cost calculator
• Success progress rate
• Degree programs, class sizes
• Post graduation plans

11
Introduction to Campus Community
Menu links take users to each section and data
element
Campus Defined Buttons
12
Success Progress Rate

• Developed as alternative to IPEDS graduation rate
  
• Focuses on student success in higher education
  system

• Tracks student progress across 2-year and 4-year
  institutions using NSC data
• New cohort query available to all Student Tracker
  users

13
(No Transcript)
14
VSA College Cost Calculator (new version late
fall 2009)
Estimate of costs tuition/fees, housing choice,
books, etc
Estimate of awards Grants, student loans, work
study
NET PRICE
15
Student Experiences Perceptions

• Selected senior responses from one of 4 surveys
  NSSE, CSS, CSEQ, UCUES
• Snapshot of student experiences on campus within
  6 common areas

• group learning
• interaction with faculty
• institutional commitment

• active learning
• diversity experiences
• student satisfaction

16
6 Common Constructs
Selected responses from Seniors
17
Student Learning Outcomes

• Pilot project to measure learning gains in broad
  cognitive skills at institution level
• Skills critical thinking, analytic reasoning,
  problem solving, written communication
• Institutional assessment as one part of
  comprehensive assessment plan
• Combined with other types of assessments
  courses, programs, general education
• Combined with other outcomes measures
  professional licensure exams

18
Student Learning Outcomes
Learning gains in critical thinking, analytic
reasoning, written communication measured at
institution level
Reported in a common format using CAAP, CLA or
MAPP
19
FIPSE Test Validity Study

• Findings

20
Background

• For VSA reporting we needed to measure
  value-added for the core learning outcomes of
  critical thinking and written communications.
• Because many provosts strongly objected to using
  a single test to measure these outcomes at all
  universities, VSA taskforces identified three
  tests that measured the outcomes. The VSA
  Presidential oversight taskforce decide to permit
  participating universities to choose from the
  three the test that best suited their campus
• The three companies that produce the tests agreed
  to adopt the same value-added computation and
  reporting technique and format so that reporting
  would be standardized across universities while
  controlling for entering student quality.

21
The FIPSE study

• What remained was to ensure that the three tests
  actually produced consistent scores.
• The only way to be sure of this was to have a
  large number of students take each of the three
  tests and observe the results.
• Funding from the Fund for the Improvement of
  Post-Secondary Education made the study possible.

22
Study Parameters

• 13 tests administered to 1,100 students at 13
  universities
• 13 tests components of CLA, CAAP, MAPP
• 4 tests of critical thinking
• 2 tests of reading
• 2 tests of mathematics
• 4 tests of writing
• 1 test of science

23
Participating Universities

• University of Michigan, Ann Arbor
• University of Minnesota, Twin Cities
• University of Texas, El Paso
• University of Vermont
• University of Wisconsin, Stout

• Alabama AM
• Arizona State
• Boise State
• California State-Northridge
• Florida State
• MIT
• Trinity College
• University of Colorado-Denver

24
Sample Parameters
25
Construct Validity

• Construct Validity is very high when the school
  is the unit of analysis
• All tests rank schools similarly, regardless of
  the construct or the response format
• Mean correlation .92 for 9 multiple choice
  tests
• Mean correlation .84 for 4 constructed response
  measures
• Mean correlation .85 for multiple choice test
  and constructed response measure of the same
  construct

26
Senior Freshman Score Differences

• Score gains between class levels indexed to
  common measure of effect size to control for
• score distribution differences across tests
• differences in average ACT/SAT scores
• Larger effect sizes indicate greater differences
  in freshman versus senior scores
• Seniors had higher mean scores than freshmen on
  all tests except the CAAP mathematics exam

27
Effect Sizes

• Effect sizes not systematically related to
  constructs tested, response format or test
  publisher
• Effect sizes across the 12 tests range from .25
  to .5 standard deviation (CAAP math excluded)
• Effect sizes
• ACT .33 (excluding math test)
• CLA .31
• ETS .34

28
Reliability

• Reliability is score consistency
• if there is a 100 chance a student will receive
  the same score on different forms of a test, the
  reliability score is 1.0
• if there is a 0 chance, the reliability score is
  0.0
• When the school was unit of analysis, score
  reliability varied from .75 to .87 across the 13
  tests
• Conclusion Score reliability is not a major
  concern

29
Overall Conclusions

• Validity and Reliability are high across tests
  and should not affect choice of measure
• Factors affecting adoption
• Acceptance by students, faculty, administrators
  or other policy makers
• Trade-offs in cost, easy of administration, etc
• Utility of the test for other purposes such as
  supporting campus activities and services or
  providing guidance on improving learning

30
The Premise on Which VSA Core Outcomes Evaluation
was Premised is Sound

• While the TVS study did not measure Value-Added
  using the VSA technique, it did find that the
  effect size was approximately the same regardless
  of the exam utilized.
• The premise that portions of CLA, CAAP and MAPP
  can be used interchangeably to produce roughly
  consistent value-added learning outcomes
  measurements for universities with similar
  students is consistent with the VSA finding.
• Thus VSA will continue to permit universities to
  choose to use either CLA, CAAP or MAPP.

31
Final Report

• Two Reports are available on the VSA website
• The complete (and technically complex) TVS full
  report as well as
• An interpretative (and less technical) report
  Christine Keller and I authored especially for
  VSA participating schools can be found at
• http//www.voluntarysystem.org/index.cfm