Assessing Critical Thinking Skills Dr. Barry Stein - Professor of Psychology, Director of Planning, Coordinator of TTU Critical Thinking Initiative Dr. Ada Haynes - Professor of Sociology Jenny Unterstein - Doctoral Student Tennessee Technological University

Assessing Critical Thinking for Continuous Improvement Why? Why? How? How?  Process & Product

What is Critical Thinking Classic Emphasis Classic Emphasis

What is Critical Thinking Expanded Contemporary Emphasis Expanded Contemporary Emphasis

Why Critical Thinking is Important for Good Citizens? Informed and Rational Participation Informed and Rational Participation Free from Ignorance, Confusion, and Unjustified Claims Free from Ignorance, Confusion, and Unjustified Claims Ability to Compete Globally Ability to Compete Globally Goals 2000: Educate America Act Goals 2000: Educate America Act

Critical Thinking in an Information Age Information becomes obsolete quickly Information becomes obsolete quickly Gauge Accuracy Gauge Accuracy Sort Relevant from Irrelevant Sort Relevant from Irrelevant Sort Quality from “Junk Science” Sort Quality from “Junk Science”

Researchers Find Bread Causes Crime 99.5% of people who committed crimes consumed bread 24 hours before the crime 99.5% of people who committed crimes consumed bread 24 hours before the crime In areas where bread consumption is low there are few crimes reported In areas where bread consumption is low there are few crimes reported

Recent TTU Employer Survey

Why Assess Critical Thinking? Need to measure success for accountability Need to measure success for accountability Assessment can focus faculty attention on the need to improve critical thinking Assessment can focus faculty attention on the need to improve critical thinking

The Test Development Process Maximize face validity for faculty Maximize face validity for faculty Maximize faculty involvement in scoring Maximize faculty involvement in scoring Develop an assessment tool that encourages continuous improvement

Recommended Steps 1. Identify faculty members interested in critical thinking 2. Find a suitable existing assessment tool 3. Evaluate the strengths and weaknesses by testing on a representative sample strengths and weaknesses strengths and weaknesses 4. Determine if a new assessment is needed for your institution

ETS “Tasks in Critical Thinking” Positives Essay Test Essay Test Faculty Involvement in scoring Faculty Involvement in scoring A good start A good start Negatives Criterion Validity Reliability Face Validity Some tasks not intrinsically motivating Not enough opportunity to observe learning and problem solving skills Pulled from the market

Faculty Identify CT skills for Assessment Focus on what graduates should be able to do in the real world that is not discipline specific Focus on what graduates should be able to do in the real world that is not discipline specific Skills that are relevant to being a Skills that are relevant to being a  good citizen  good consumer of information information  productive member of society

Larger Tires Improve Safety?

Major Areas of Assessment Identified by TTU Faculty Evaluate claims, hypotheses, conclusions Evaluate claims, hypotheses, conclusions Apply new information to solve real-world problems Apply new information to solve real-world problems Communicate ideas effectively Communicate ideas effectively Detailed list

Important Skills Identified at TTU 1.Separate factual information from inferences that might be used to interpret those facts. 2.Identify inappropriate conclusions. 3.Understand the limitations of correlational data. 4.Identify evidence that might support or contradict a hypothesis. 5.Identify new information that is needed to draw conclusions. 6.Separate relevant from irrelevant information when solving a problem. 7.Learn and understand complex relationships in an unfamiliar domain. 8.Interpret numerical relationships in graphs and separate those relationships from inferences. 9.Use mathematical skills in the context of solving a larger real world problem. 10.Analyze and integrate information from separate sources to solve a complex problem. 11.Recognize how new information might change the solution to a problem. 12.Communicate critical analyses and problem solutions effectively.

Develop Assessment Questions Small groups or individuals to design questions/problems Small groups or individuals to design questions/problems  Minimize familiarity bias  Maxmize correspondence to real-world tasks Larger faculty groups evaluate face validity Larger faculty groups evaluate face validity

Develop Scoring Criteria Clear & Reliable Clear & Reliable More points reflect better skills More points reflect better skills Anticipate possible answers Anticipate possible answers Provide guidelines for scoring creative answers Provide guidelines for scoring creative answers Refine, refine, refine Refine, refine, refine

Refine Test Improve Face Validity – expand faculty input Improve Face Validity – expand faculty input Establish Criterion Validity – correlate with other tests and performance measures Establish Criterion Validity – correlate with other tests and performance measures Establish Construct Validity – relate underlying skills to contemporary theory Establish Construct Validity – relate underlying skills to contemporary theory Maximize Inter-rater Reliability Maximize Inter-rater Reliability

Maximize Reliability of Scoring Scoring workshops Scoring workshops Multiple Graders for each question Multiple Graders for each question Modify scoring criteria using faculty input Modify scoring criteria using faculty input Modify or eliminate questions that cannot be reliably scored Modify or eliminate questions that cannot be reliably scored

TTU’s CAT © (Critical thinking Assessment Test) Tested on over 600 students in 3 years Tested on over 600 students in 3 years Inter-rater reliability = 0.78 to 0.98 Inter-rater reliability = 0.78 to 0.98 Test-retest reliability >.60 Test-retest reliability >.60 Internal consistency  =.64 Internal consistency  =.64 Correlation with ACT (.659) and CCTST (.645) Correlation with ACT (.659) and CCTST (.645) Sensitive to class standing (freshman vs senior) Sensitive to class standing (freshman vs senior)freshman vs seniorfreshman vs senior Sensitive to effects of a class in critical thinking Sensitive to effects of a class in critical thinkingclass in critical thinkingclass in critical thinking Seven underlying factors (71.8% variance) Seven underlying factors (71.8% variance)

CAT materials

CAT features One hour exam One hour exam Mostly short answer essay Mostly short answer essay Faculty scored in workshops Faculty scored in workshops  10 faculty score exams in 1 day Detailed scoring guide Detailed scoring guide

Status and Future Involve other institutions to improve face validity Involve other institutions to improve face validity Evaluate and correct cultural bias Evaluate and correct cultural bias Develop national norms Develop national norms

Thank You TTU Critical Thinking Initiative