Robust Process to Establish Validity and Reliability Developing Materials to Assess Process Skills in an Active Learning Classroom Suzanne M. Ruder 1, Renee S. Cole 2 and Juliette Lantz 3 1 Virginia Commonwealth University, 2 University of Iowa, 3 Drew University Targeted Process Skills and Definitions Rubric Design Process - an Iterative Approach ELIPSS Strategy for Cohesive Development Student Interaction RubricsELIPSS Project Overview ELIPSS Project Team Primary Collaboration Team Chris Bauer, Chemistry Teresa Bixby, Chemistry Patrick Brown, Biology Caryl Fish, Chemistry/Environ. Jill Guerra, Math Stanley Lo, Biology Chris Mayfield, Computer Science Eunice Yang, Engineering Graduate Students Jennifer Schmidt-McCormack Gilbert Reynders Student Product Rubrics Workshops Student Interaction Rubrics Student Product Rubrics Implementation Guide Do rubric descriptors look appropriate? Face Validity Does rubric measure what it is intended to measure? Construct Validity Can different people use rubric to get consistent results? Reliability Can rubric be used in multiple disciplines, environments, and student populations? Generalizability Can the rubric be used effectively by instructors and students? Utility Problem Solving Critical Thinking Information Processing Written Communication Assessment Metacognition IUSE Collaborative grant: # # # Interested in using the rubrics go to: Look for future website at ELIPSS.com Process skills are frequently cited as critical components of a successful education. Assessing process skill development and providing feedback to students and instructors is a key component for enhancing these skills in STEM programs. This project involves the development of resources to provide feedback to students, and to inform instructors as to the effectiveness of their instructional strategies in supporting process skill development. This feedback may ultimately result in increasing adoption of evidence-based pedagogical strategies that foster process skill development. Summary Refinement of process skill definitions aided in development of rubrics that are applicable to a broad STEM audience Primary Collaboration Team assembled for continuing rubric testing for reliability, validity and utility Materials Developed: Student interaction rubrics (4 + 2 drafts) Student product rubrics (1 + 2 drafts) Workshop piloted – “Enhancing Learning by Assessing Process Skills: Group Interaction Rubrics” Student Product Rubrics Student Interaction Rubrics IP PS IP PS CT OC M TW VALIDITY RELIABILITY UTILITY Appropriate Tasks CT Generate “anchor” examples Construct preliminary rubric SkillDefinition A Assessment (Self & Peer) Gathering information and reflecting on experiences to improve subsequent learning and performance. OC WC Communication Oral - Exchanging information and understanding through speaking, listening and non-verbal behaviors. Written - Conveying information and understanding to an intended audience through written materials (paper, electronic, etc.) CTCritical Thinking Analyzing, evaluating or synthesizing relevant information to form an argument or reach a conclusion supported with evidence. IP Information Processing Evaluating, interpreting, manipulating or transforming information MManagement Planning, organizing, directing and coordinating one’s own and others’ efforts to accomplish a goal. MCMetacognition Thinking/reflecting about one’s thinking and how one learns, and being aware of one’s knowledge. PSProblem Solving Identifying, planning and executing a strategy that goes beyond routine action to find a solution to a situation or question. TWTeamwork Interacting with others and building on each other’s individual strengths and skills, working towards a common goal. Tested with STEM faculty in: Biology Chemistry Computer Science Engineering Environmental Science Mathematics Designed to use on student written work (projects, quizzes, exams, lab reports) to assess process skills. Designed to use during group work in the classroom, to observe process skills in action. Data currently being gathered at 11 institutions. Enhancing Learning by Improving Process Skills in STEM