Pilot Study of the CAAP

1

• Pilot Study of the CAAP
• Critical Thinking Test
• April 27, 2005

Lanette Raymond Research Associate, Suffolk
County Community College
2
CAAP Test Description

• 32 item multiple choice test
• Designed for use with college students
• Administered within a single class meeting,
• Appeared relatively credible in an in-class
  administration protocol
• Provided documentation of reliability and
  validity across community college populations

3
CAAP Subscore Customization

• CAAP Critical Thinking test contents closely
  match the SUNY CT learning objectives
• Variance in the way the results are reported
• ACT developed a customized report for these
  sub-scores showing normative comparisons against
  ACT national community college data.
• ACT provided the student data files to SCCC, for
  further analysis of this data

4
Administration and Sample

• Fall, 2004
• Administered in-class to 154 SCCC students in 7
  general education courses
• Predominately white (77)
• Traditional age (60 20 years old or younger, 25
  between 21 and 25 years old)
• 50 male, 50 female
• Mostly sophomore status (46)
• Fulltime enrollment (85)

5
Student Motivation

• No motivational tactics were employed
• CAAP-CT instrument included an item that
  addressed students' self-reported motivation
  levels
• One-third of students (n 52) did not respond to
  the motivation item
• 5 students indicated that they gave no effort
  (n 1) or gave little effort (n 4) to the
  assessment test.

6
Student Motivation

• Lower motivation results in less optimal
  performance
• Less motivated students scores are less reliable
  and less valid
• Reporting sample is based on data from the 97
  students who reported moderate to best effort
• The reliability coefficient (calculated with the
  data from the original 154 tests) for objective 1
  (26 items) is within acceptable range (alpha
  .80)
• Due to the small number of items (6 items)
  contributing to objective 2, its reliability
  coefficient is much lower (alpha .49).

7
(No Transcript)
8
(No Transcript)
9
Results

• Confirmatory factor analysis substantiates the
  utility of the CAAP-CT test as a measure of 2
  separate but related sets of critical thinking
  skills based on the 2 GEAR learning objectives

10
2-factor model of Critical Thinking based on the
GEAR Objectives
11
Results

• All of the items loaded well onto their
  respective factors, with item 1 being only
  slightly below 1.96 (at 1.81).
• The model shows an excellent fit to the data
  (?2(463) 466, p .46, CFI .94), providing
  additional context validity to the assessment.

12
Data for item 20 should be listed under
objective 2
13
Standards

• Does not meet standard 59 or less
• Approaches standard 60 - 69
• Meets standard 70 - 79
• Exceeds standard 80 or more

14
Standards