1. Assessing the Development of Critical Thinking Skills in Undergraduate Psychology Students Phil Dunwoody & Mark McKellop Partially funded by grants from the Teagle Foundation, the Council of Independent Colleges/Collegiate Learning Assessment Consortium, and the James J. Lakso Center for the Scholarship of Teaching & Learning at Juniata College.

2. What is Critical Thinking?

3. American Philosophical Association’s “Delphi Report” (Facione, 1990)  CT is “purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based.”

4. What is Critical Thinking? CT is multidimensional construct – it is both a disposition & a learned skill (Halpern, 1998, 2003) CT has general & discipline-specific aspects with limited generalizability (Lehman & Nisbett, 1990; Toplak & Stanovich, 2002; Williams et al., 2004)

5. Psychological Critical Thinking (PCT) American Psychological AssociationAssociation “Guidelines for the Undergraduate Psychology Major” (APA, 2007) ◦Goal 3: Critical Thinking Skills in Psychology.  Students will respect and use critical and creative thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes. Documenting & Assessing our Teaching ◦Departmental Assessment PlanPlan ◦Mapping our POEPOE

6. Psychological Critical Thinking PCT is “the ability to judge the plausibility of specific assertions, to weigh evidence, to assess the logical soundness of inferences, to construct counter-arguments and alternative hypotheses.” (Nickerson et al, 1985) CT skills in psychology include: ◦Argument analysis and evaluation ◦Methodological reasoning ◦Statistical reasoning ◦Causal reasoning ◦Focusing and clarifying questions (Bensley & Murtaugh, 2012)

7. Psychological Critical Thinking Inventory (PCTI) Our assessment/measure of PCT will ◦be relatively brief ◦be (relatively) easily scored ◦evaluate “applied” critical thinking skills (can one “think like a behavioral scientist?”) PCTI focuses on following aspects of PCT: PCTI  Appropriate evaluation of expert behavioral research (i.e., peer-reviewed journal articles)  Appropriate evaluation of non-expert behavioral claims (i.e., media reports)  Identifying reasoning faults in faux-novice designed behavioral research

8. Validating the PCTI California Critical Thinking Skills TestTest Watson-Glaser Critical Thinking AppraisalAppraisal Lawson’s Psychological Critical Thinking Exam Exam Collegiate Learning AssessmentAssessment Psychology Area Concentration Achievement Test (PACAT)PACAT National Survey of Student Engagement (NSSE)NSSE

9. Participants Participants ◦Intro Psych: 96 (Spring, 2011) ◦Senior Psych Capstone: 39 (Spring & Fall, 2011 and Spring, 2012) ◦Pre and Post Research Methods: 29 (Fall, 2011 and Spring, 2012)  Pre and Post with CAT currently

10. Descriptive Stats PCTI Popular Press ◦Possible range: 0-10 points ◦Observed range=2-8, Mean=5.9, SD=1.7 Critique Faux-Novice Study ◦Possible range: 0-7 points ◦Observed range= 0-6, Mean=2.9, SD=1.3 Peer-review Articles ◦Possible range: 0-7 points ◦Observed range= 0-6, Mean=3.28, SD=1.3 Total PCTI ◦Possible range: 0-24 ◦Observed range= 6-20.5, Mean=12.1, SD=2.7

11. Hypothesis 1 Graduating psychology seniors will score higher on PCTI than freshman in Introductory Psychology. Popular Press ◦Intro:M=5.19, SD=1.48 ◦Capstone: M=6.55, SD=1.99 ◦Significant: t(91)=-3.72, p<0.001, d=0.78 Critique Faux-Novice Study ◦Intro: M=2.48, SD=1.19 ◦Capstone: M=3.88, SD=1.26 ◦Significant: t(91)=-5.27, p<0.001, d=1.14 Peer-review Articles ◦Intro: M=3.03, SD=1.37 ◦Capstone:M=3.34, SD=1.45 ◦Significant: t(91)=-1.02, p=0.31 Total PCTI ◦Intro:M=10.70, SD=2.34 ◦Capstone: M=13.77, SD=2.76 ◦Significant: t(91)=-5.64, p<0.001, d=1.20

12. Hypothesis 2 Students in Research Methods should score higher at the end of the semester than at the beginning of the semester (alternate form). Popular Press ◦Pre:M=6.52, SD=1.46 ◦Post: M=7.30, SD=1.32 ◦Significant: t(26)=-2.09, p=0.05, d=0.56 Critique Faux-Novice Study ◦Pre: M=3.26, SD=1.10 ◦Post: M=3.81, SD=1.27 ◦Significant: t(26)=-1.86, p=0.07, d=0.46 Peer-review Articles ◦Pre: M=3.67, SD=1.30 ◦Post:M=3.93, SD=1.75 ◦Significant: t(26)=-0.55, p=0.59 Total PCTI ◦Pre:M=13.44, SD=2.68 ◦Post: M=15.04, SD=2.49 ◦Significant: t(26)=-2.39, p=0.02, d=0.62

14. PCTI Convergent Validity

15. PCTI with measures of generic (abstract) critical thinking

16. PCTI with Lawson’s Psychological Critical Thinking Exam

17. PCTI with measures of educational achievement

18. PCTI with measures of Psychology Knowledge (PACAT)

19. Conclusion Our PCTI has good face validity and shows clear differences not only between Freshman and Seniors, but also within one course. The PCTI does not correlate much with more traditional abstract measures of CT. The PCTI clearly correlates with the psychology knowledge-based PACAT. The PCTI also clearly correlates with general measures of academic achievement such as GPA, SAT V, and SAT M. Intelligence tests also correlate with these same measures. So…

20. Is critical thinking just another term for intelligence? "[Intelligence is] the aggregate, or global capacity to act purposefully, think rationally, and deal effectively with the environment. It is global because it characterizes the individual's behavior as a whole; it is an aggregate because it is composed of elements or abilities which, though not entirely independent, are qualitatively differentiable.” ◦ David Wechsler, 1938

21. References American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Washington, DC: Author. Retrieved from www.apa.org/ed/resources.html Bensley, D. A., & Murtagh, M. P. (2012). Guidelines for a scientific approach to critical thinking assessment. Teaching of Psychology, 39, 5-16. Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: The California Academic Press. Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449–455. Halpern, D. F. (2003). Thought and knowledge: An introduction to critical thinking. Mahwah, NJ: Erlbaum. Lehman, D. R., & Nisbett, R. E. (1990). A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology, 26, 952-960. Nickerson, R. S., Perkins, D. N., & Smith, E. E. (1985). The teaching of thinking. Hillsdale, NJ: Lawrence Erlbaum. Toplak, M. E., & Stanovich, K. E. (2002). The domain specificity and generality of disjunctive reasoning: Searching for a generalizable critical thinking skill. Journal of Educational Psychology, 94, 197-209. Williams, R. L., Oliver, R, & Stockdale, S. (2004). Psychological versus generic critical thinking as predictors and outcome measures in a large undergraduate human development course. The Journal of General Education, 53, 37-58.