Pilot Study of the CAAP Critical Thinking Test April 27, 2005 Lanette Raymond Research Associate, Suffolk County Community College

CAAP Test Description z32 item multiple choice test zDesigned for use with college students zAdministered within a single class meeting, zAppeared relatively credible in an in-class administration protocol zProvided documentation of reliability and validity across community college populations

CAAP Subscore Customization zCAAP Critical Thinking test contents closely match the SUNY CT learning objectives zVariance in the way the results are reported zACT developed a customized report for these sub- scores showing normative comparisons against ACT national community college data. zACT provided the student data files to SCCC, for further analysis of this data

Administration and Sample zFall, 2004 zAdministered in-class to 154 SCCC students in 7 general education courses zPredominately white (77%) zTraditional age (60% 20 years old or younger, 25% between 21 and 25 years old) z50% male, 50% female zMostly sophomore status (46%) zFulltime enrollment (85%)

Student Motivation zNo motivational tactics were employed zCAAP-CT instrument included an item that addressed students' self-reported motivation levels zOne-third of students (n = 52) did not respond to the motivation item z5 students indicated that they “gave no effort” (n = 1) or “gave little effort” (n = 4) to the assessment test.

Student Motivation zLower motivation results in less optimal performance zLess motivated students’ scores are less reliable and less valid zReporting sample is based on data from the 97 students who reported moderate to best effort zThe reliability coefficient (calculated with the data from the original 154 tests) for objective 1 (26 items) is within acceptable range (alpha =.80) zDue to the small number of items (6 items) contributing to objective 2, its reliability coefficient is much lower (alpha =.49).

Results zConfirmatory factor analysis substantiates the utility of the CAAP-CT test as a measure of 2 separate but related sets of critical thinking skills based on the 2 GEAR learning objectives

2-factor model of Critical Thinking based on the GEAR Objectives

Results zAll of the items loaded well onto their respective factors, with item 1 being only slightly below 1.96 (at 1.81). zThe model shows an excellent fit to the data (χ2(463) = 466, p =.46, CFI =.94), providing additional context validity to the assessment.

** Data for item 20 should be listed under objective 2

Standards Does not meet standard 59% or less Approaches standard 60% - 69% Meets standard 70% - 79% Exceeds standard 80% or more

Standards