Caso 1 A Sampler of Various Assessment Projects Aero CRCD “Smart Materials Project” Rita Caso August 22, 2003 Extract from

Caso 2 CONTENTS INTRODUCTION INSTITUTIONAL & POLICY RESEARCH COMPREHENSIVE IMPACT ASSESSMENT NARROW IMPACT ASSESSMENT

Caso 3 NARROW IMPACT ASSESSMENT Grant-Project Evaluation Institutional Policy Change Assessment Faculty Inquiries into the success of their pedagogical strategies AERO CRCD “Smart Materials” Project Evaluation Goal: To introduce and integrate interdisciplinary instruction and learning activities pertaining to smart materials and intelligent systems into the undergraduate engineering curriculum

Caso 4 AERO CRCD Assessment & Evaluation Plan: Year 1 Outcome Measurement For Freshmen = F For Seniors = S

Caso 5 Assessment Methods: AERO CRCD “Smart Materials” Project Pre-to-Post Change :Self-Comparison Paired t-test Pre & Post “treatment-to-control” Comparison Discriminant analysis LSD tests, two sample t-test Pre-to-Post Change treatment-to-”control” Comparison 2-sample t-test Discriminant analysis of change scores Benchmark comparisons Baseline and Benchmark identification Assessing at different levels of expertise Response Content Analysis at different levels of expertise Measurement Tool Refinement Concept Learning Assessment – Instrument Development

Caso 6 Impact Assessment of AERO CRCD Fall 2001 “Smart Materials” Curriculum on Freshman –Student Perceptions –Design Knowledge 1 first-Freshman Engineering Course class experienced the “treatment” 2 other first-Freshman Engineering Course classes served as comparison groups AERO Senior Design Course provided benchmark comparisons in Sept & November 1 second-Freshman Engineering Course class provided Freshman end-of-year benchmark for comparison Assessing AERO CRCD Impact: Student Perceptions Design Knowledge

Caso 7 1 class of the first Freshman Engineering Course, Fall 2001, CRCD “Smart Materials” curriculum was introduced Classroom Demonstration of Smart Material Class “Stiquito” projects Visitor Lectures AERO CRCD Freshman Implementation

Caso 8 First Freshman Engineering Course Project: Walking Robot Robot (Stiquito) specifications: –Must be actuated by SMAs –Goal is maximum distance in 3 minutes –Only contact can come from ground –Must be an autonomous system Assigned to 24 four-person student teams in ENGR 111 Maximum distance traveled was 48cm.

Caso 9 Assessing AERO CRCD Impact: Improvement in First Semester Freshman To measure the change in perceptions of knowledge & learning, the 1 CRCD and 2 Non-CRCD classes took a pre and post survey of perceptions Only subscales common to both Pre and Post surveys were examined: Self Appraisal Team Attitudes Outside Help Pre-Test (Sept 2001) Post-Test (Nov 2001) Freshman Engineering Perceptions Test(EPT) Assessment Tools Knowledge of Engineering Design Engineering Design Process Teamwork Communication. Pre-Test (Sept 2001) TIDEE Team-Based Design Knowledge Assessment Test and Scoring Rubric Post-Test (Nov 2001) TIDEE Team-Based Design Reflective Essay and Scoring Rubric

Caso 10 AERO CRCD Impact:Student Perceptions Engineering Perception Pre-Test SUBSCALES*Self AppraisalTeam AttitudesOutside Help Comparison Group mean Treatment Group mean On Pre-Test, Comparison and Treatment Groups were found to be similar. (discrim. anal. n.s ) * Scale 1=most positive & 5=most negative

Caso 11 SUBSCALESSelf AppraisalTeam AttitudesOutside Help Comparison Group mean 2.15 * Treatment Group mean 1.97 (α = 0.013) t-test 2.63 (α = 0.017) t-test 4.13 Engineering Perception Post Test On Post-Test Comparison Group and Treatment Group were found to be significantly different. (discrim. anal. s ) Treatment Group was more positive in perceptions as compared to Comparison Group. * Scale 1=most positive & 5=most negative AERO CRCD Impact:Student Perceptions Engineering Perception Post Test

Caso 12 The Treatment Group’s changes in perceptions over the semester differed significantly from that of the Comparison Group (Discrim. Anal, p= 0.001, Effect size-12%’) The Comparison Group became LESS positive between the Pre testing at the beginning of the semester, and the Post testing at the end. (paired T test) Treatment Group appeared to have succeeded in sustaining positive perceptions. (paired T test) SUBSCALESSelf AppraisalTeam AttitudesOutside Help Comparison Group mean diff * (α = 0.000) pr-t (α = 0.000) pr-t (α = 0.032) pr-t Treatment Group mean diff (α = 0.001) pr-t * Scale 1=most positive & 5=most negative AERO CRCD Impact:Student Perceptions Freshman Perception Test : Pre to Post Comparison

Caso 13 Design Assessment Instruments 0—5.5 ScaleDesign Process Team Work Communication Team Design Knowledge Pre- Test 2 (Sept. 2001) Mean Scores Std. Dev Reflective Essay on Team Design Experience-Based Knowledge 3 (Dec. 2001) Mean Scores Std. Dev samp t-test signifP<.0005 P< Members of one ENGR 111 class which utilized AERO CRCD Project curriculum, Fall Adapted TIDEE Project Mid Program Assessment Instrument #1, Design Knowledge 88), Sept Adapted TIDEE Project Mid Program Assessment Instrument #3, Reflective Essay (n=87), Dec Scores given on a scale of 0 – 5.5, with 0=no knowledge & 5.5=exceptional knowledge AERO CRCD Impact: Design Knowledge Improvement in First Semester Freshman

Caso 14 AERO CRCD Impact: Design Knowledge Improvement in First Semester Freshman AERO CRCD’s Freshman 1 Class Design Projects contributed to growth in knowledge about Engineering Team-Design

Caso 15 AERO CRCD Impact: Design Knowledge Freshmen vs. Seniors (Baselines -- Sept 2001 Samples) AERO CRCD Students 0—5.5 ScaleDesign Process Team Work Communication Freshmen 2 (n=88) Mean Scores Std. Dev Seniors 3 (n=23) Mean Scores Std. Dev sample t-test P<.01nsP< Adapted TIDEE Project Mid Program Assessment Instrument #1, Design Knowledge 2 Members of one ENGR 111 class which utilized AERO CRCD Project curriculum 3 Members of AERO Senior Design course 4 Scores given on a scale of 0 – 5.5, with 0=no knowledge & 5.5=exceptional knowledge

Caso 16 Benchmark Identification--Comparative Results: AERO Senior Design Students

Caso 17 Senior AERO Design Responses: Content Analysis & Comparison with Rubric Knowledge of Design Process (Pre-test) F 01

Caso 18 Freshman Baseline-to-Benchmark Design Knowledge Comparison: Beginning vs. End of Yr. *Different students in Pre and Post test groups TAMU-TIDEE Instrument V.1 used for Pre. TAMU-TIDEE Instrument V.2 used for Post V.1 TAMU-TIDEE Rubric used for Pre & Post.

Caso 19 Freshman Engineering Design Responses: Content Analysis & Comparison with Rubric– Knowledge of Design Process (Post-test) Sp 02

Caso 20 Work to Improve Measurement Increasing sensitivity Increasing the range of knowledge, skill, or attitude measured Adjusting for context specificity Improving internal consistency Improving depth