C16 – Tools for Assessing Learning in Engineering Teri Reed-Rhoads Purdue University School of Engineering Education Inventions and Impact 2: Building Excellence in Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education 13-15 August 2008 Washington, DC

Engineering Education Grant Writing Workshop ASEE Section Meeting April 3, 2008 Teri Reed-Rhoads Purdue University trhoads@purdue.edu

Assessment Formative – checking along the way Summative – final analysis of achieving research goals Consider an outside evaluator – unbiased, measurable feedback Consider an external advisory board Give details, actual tools, timelines, etc.

Assessment Course Curriculum Program Project

Types of Assessment Cognitive Domain Content knowledge (Concepts) Process knowledge, e.g., teamwork (Skills, Abilities) Student interest, perceptions, and attitudes (Affective) Rate change (Achievement)

PD’s Response – Goals on Cognitive Changes Increase understanding of concepts Ability to solve statics problems Ability to construct free-body diagrams Ability to describe verbally the effect of external forces on a solid object Increase processing skills Ability to solve out-of-context problems Ability to visualize 3-D problems Ability to communicate technical problems Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

Concept Inventories https://engineering.purdue.edu/SCI/workshop/tools.html Engineering Related 20 instruments Science and Mathematics Related 8 instruments Physics and Astronomy Specific 12 instruments

Engineering Related CI

Science and Mathematics CI

Physics and Astronomy CI* *Rebecca S. Lindell, Elizabeth Peak and Thomas M. Foster, “Are They All Created Equal? A Comparison of Different Concept Inventory Development Methodologies”, 2006 Physics Education Research Conference Proceedings, American Institute of Physics.

Goals on Process Knowledge Teamwork Team Effectiveness (Imbrie) Team Developer (McGourty) Communications 3-D Visualization Etc. Leadership

PD’s Response – Goals on Affective Changes Improve students’ attitude about Profession Curriculum Department Improve students’ confidence Improve students’ intellectual development Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

Affective Measures Pittsburgh Freshman Engineering Survey (Besterfield-Sacre) Attitudes Toward Statistics (Wise) Computer Science Attitude Survey (Moskal) Structured Interviews

Consider a Graphic

PD’s Response – Goals on Achievement Rate Changes Improve Recruitment rates Retention or persistence rates Graduation rates Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

Know the Motivation or Background

URMs 30% of population, few in labor force Women 50% of population, 10% in labor force # of BS Engineering Degrees % of Freshman Engineering Class 1966 1977 2002 1975 2001 Under Represented Minorities 1,915 (4.8%) 7,971 (11.6%) 17.4 15.8 Women 146 (0.4%) 1,961 (4.9%) 14,102 (20.5%) 19.9 18.3 Chubin and Babco (2003), Walking the Talk in Retention to Graduation

Engineering Disciplines Retention Rates Engineering Discipline Proportion Women ProportionURM Total URM Non-URM Chemical 32.4 11.5 35.0 48.1 Civil 20.1 13.2 37.9 44.2 Computer 8.8 47.6 58.7 Electrical 11.8 14.3 36.3 47.2 Industrial 29.4 17.7 51.3 Mechanical 12.0 11.0 38.0 48.7 Total 19.0 11.7 38.9 49.5 Chubin and Babco (2003), Walking the Talk in Retention to Graduation National Science Foundation. Women, Minorities, and Person with Disabilities in Science and Engineering: 2000. Arlington, VA, 2000 (NSF 00-327)

Authentic Teaching Alliance Assessment Cycle

Group Questions What are the known instruments/methods in each of the three areas? (Cognitive, Affective, and Rate Change) Where have these been used successfully/unsuccessfully? Where are the gaps in each area?

Process One note taker One reporter Everyone discusses We will report out with 15 minutes left

There is no failure. Only feedback. -Anonymous Happy grant writing! trhoads@purdue.edu https://engineering.purdue.edu/SCI/Workshop

Affective National academy of engineering website with book of surveys

Rate Change Longitudinal measurements, courses, rate changes of minorities, retention, enrollment, GPA for performance measures Use Associate Dean’s offices as resources for stats, changes, etc.

Wish Critical Thinking – one that is free Definition of engineering changes for students as they go through the curriculum Digital Systems within computer engineering Teaming attitude by individual Measuring how well the student can deal with an out of box situation. Clearinghouse that explains what already exists, how long is it, how it was used in the past, cliff notes version

Wish ABET could sponsor (or someone) or post tools that are being used More CI https://engineering.purdue.edu/SCI/workshop/tools.html Tools for writing and communication Life-long learning tools (continuous learning) Procedural knowledge on engineering design FREE Critical Thinking

Wish Meta-cognition Societal context of knowledge, global competencies, service learning, research skills, discourse and collaborative dialogue, oral communication, problem solving, leadership (some instruments from Business schools), ability to integrate knowledge, longitudinal data, qualitative data analysis Evaluation of case studies

Cognitive Math placement test English Placement test Concept Inventories https://engineering.purdue.edu/SCI/workshop/tools.html FE, GRE – standardized tests Critical thinking Criticalthinking.org – has tools, and rubric developed by Gary Brown at Washington State, Watson Glazier (also not free), U of South Carolina EFFECTS (web-based) http://www.ce.sc.edu/effects

California Critical Thinking Collegiate Learning Assessment CAAP – from ACT SALG – Student Assessment of Learning Gains IDEA – teaching assessments MSLQ – Motivated Strategies for Learning Questionnaire NSF User Friendly Guide to Assessment

Rogers and Sando, Rose-Hulman Learning Styles Instruments (Kolb, Felder) Career Instruments (Holland) FMRI, Caps measuring brain activity Experience sampling method – student wears a bracelet and when it goes off the student writes down what they are doing (engagement measures) Calibrated Peer review Teamwork effectiveness (Imbrie, Smith, Layton) National Center for Case Studies (Buffalo)