1. Critical Thinking and Information Literacy: Assessing Student Performance Work in Progress Critical Thinking and Information Literacy: Assessing Student Performance The InfoSkills Research Team Dr. Senay Purzer, Mr. Michael Fosmire, Ms. Amy Van Epps, and Ms. Ruth Wertz Project Collaborators Ms. Megan Sapp Nelson, Dr. Brian Dillman, and Mr. Austin Saragih Presented at the 120 th ASEE Annual Conference & Expo – Atlanta, GA June 24 th, 2013 1InfoSkills, 2013

2. 2 OverviewOverview Project Background – Information Literacy and Critical Thinking Research Methods – RQs and Research Design Project Results – Correlation of CELT and CAT Project Conclusions – Implications & Further Study InfoSkills, 2013

3. skillsrecognize the need search foraccessevaluate use A set of skills that enables the ability to recognize the need for information, and the ability to search for, access, evaluate, and use information to fulfill a specific purpose. 1 InfoSkills, 20133 DefinitionsDefinitions Project Background Information Literacy cognitive process conceptualizingapplyinganalyzing synthesizingevaluating The cognitive process of “conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.” 2 Critical Thinking 1.American Library Association. (2000). Information literacy competency standards for higher education. Retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf 2.The Foundation for Critical Thinking. (2013). Defining critical thinking. Retrieved from http://www.criticalthinking.org/pages/defining-critical-thinking/410.http://www.criticalthinking.org/pages/defining-critical-thinking/410 3.Albitz, R. S. (2007). The what and who of information literacy and critical thinking. portal: Libraries and the Academy, 7(1), 97-109. recognize need access search for/gather evaluate use/apply synthesize analyze conceptualize SET OF SKILLS SET OF SKILLS 3 COGNITIVE PROCESSES COGNITIVE PROCESSES 3

4.  Impetus for our research: Important 21st century skills ABET criterion 3i (lifelong learning) Engineering decision-making  Primary focus of our research: How do we assess information literacy and critical thinking? 4 Impetus for Our Research InfoSkills, 2013 Project Background

5.  InfoSEAD framework for item generation 1 Seeking – Gathering and access Evaluation – Assess quality, credibility, etc. Application – Using information to satisfy a need Documentation – Citation and in-text referencing  Two Scenarios  18 total items 16 multiple choice; 2 select all that apply 5 About the CELT(v2.1) InfoSkills, 2013 Project Background 1.Wertz, R. E. H., Purzer, S., Fosmire, M. J., & Cardella, M. E. (in press). Assessing information literacy skills demonstrated in an engineering design task. Journal of Engineering Education.

6.  Research question: How well does the CELT (v2.1) measure critical thinking?  Research method: Pearson’s correlational analysis between CELT instrument and the Critical Thinking Assessment Test (CAT) developed by Tennessee Technology University. 1 InfoSkills, 20136 Correlational Study Research Methods 1.Center for Assessment and Improvement of Learning at Tennessee Technological University. (2010). CAT technical information, from http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf

7.  N = 44 first-year engineering students  CELT reliability measures KR-20 = 0.67 1 N = 188 first-year students (Fall 2012) Based on 16 MC questions  CAT reliability measures 15 constructed response items; scoring reliability 0.82 2 Cronbach’s alpha 0.70 2 InfoSkills, 20137 Population and Setting Research Methods 1.Wertz, R. E. H., Saragih, A., Fosmire, M. J., & Purzer, S. (2013). An Evaluation of the Critical Engineering Literacy Test (CELT) Instrument through Item Analysis and Comparison to the Critical Assessment Test (CAT). Paper presented at the 2013 Illinois/Indiana - ASEE Section Conference, Angola, IN. 2.Center for Assessment and Improvement of Learning at Tennessee Technological University. (2010). CAT technical information, from http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf http://www.tntech.edu/files/cat/reports/CAT_Technical_Information_V7.pdf

8.  Overall Correlational Analysis: CELT total score was positively related the CAT total score (r = 0.47, p <0.01)  Itemized Correlational Analysis: Items 5, 11, 16, 17 had positive associations with the CAT total score InfoSkills, 20138 CELT Results Project Results

9. InfoSkills, 20139 CELT Results Project Results most relevant Which of the following comparisons of omega3/omega6 ratios is most relevant in determining whether GE fish is equivalent to its non-GE counterpart? Item 11 (r = 0.32, p <0.05) Item 11 (r = 0.32, p <0.05) validate What would help the review panel validate the data presented? Item 16 (r = 0.37, p <0.05) Item 16 (r = 0.37, p <0.05) authoritative Where would you likely find authoritative information on a typical omega-3 levels of salmon? Item 17 (r = 0.35, p <0.05) Item 17 (r = 0.35, p <0.05) Evaluate [ Evaluate ] which of the citations is incorrect or incomplete? Item 5 (r = 0.51, p <0.01) Item 5 (r = 0.51, p <0.01)

10.  14 of 18 CELT items positively correlated to some portion of CAT  Correlation between the CELT and CAT total scores is moderately strong (r = 0.47, p <0.01)  CELT items 5, 11, 16, and 17 provide insight into a possible subset of skills where information literacy skills and critical thinking overlap. InfoSkills, 201310 General Conclusions Project Conclusions

11.  What are the common elements in CELT items 5, 11, 16, and 17 that made them correlate to CAT? Expand sample Item-to-item analysis Verbal protocol InfoSkills, 201311 Future Study Project Conclusions

12. 12 Dr. Senay Purzer – senay@purdue.edu Mr. Michael Fosmire – fosmire@purdue.edu Ms. Amy Van Epps – vanepa@purdue.edu Ms. Ruth Wertz – rwertz@purdue.edu InfoSkills, 2013 THANK YOU! Contact Information