W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland

W W W. C E S. C L E M S O N. E D U / G E / Workshop Objective Describe steps in the planning and assessment of engineering education research Give examples of each step Small-group discussion to implement the step using a common example

W W W. C E S. C L E M S O N. E D U / G E / Workshop Agenda 4:00-4:05Introduction 4:05-4:20Defining the Purpose of the Evaluation (What questions are we asking?) 4:20-4:40Clarify Project Objectives (What do we expect?) 4:40-5:00Create a Model of Change (How will our efforts lead to the objectives?) 5:00-5:15Select Criteria and Indicators (What data do we need to measure our progress?) 5:15-5:25Identify Data Sources (Where will we get that data?) 5:25-5:45Design Evaluation Research (How will we analyze that data?) 5:45-5:50Monitor and Evaluate (What actually happened?) Use and Report Results (Who needs to know?) 5:50-6:00Wrap-up and evaluations

W W W. C E S. C L E M S O N. E D U / G E / Step 1: Purpose of the Evaluation Formative Summative Efficacy Effectiveness Cost Examples Is there a benefit to doing this? Can we improve it? Does that benefit lead to better retention / grades? Do the benefits justify the program cost? Can we achieve the same benefits on a larger scale?

W W W. C E S. C L E M S O N. E D U / G E / Step 1 Action Steps Decide purpose(s) Primary questions? Order? Primary audience(s)? How will the results be used?

W W W. C E S. C L E M S O N. E D U / G E / Step 1: Purpose of the Evaluation Do we have a learning community? Is it leading to better course retention / grades? Can we achieve the same benefits cost- effectively on a larger scale?

W W W. C E S. C L E M S O N. E D U / G E / Step 2: Project Objectives Impact Outcome Process Example Improve graduation rate by some % Increase course passing rate by some % Student or faculty opinions / behaviors change Reserve any facilities needed

W W W. C E S. C L E M S O N. E D U / G E / Step 2 Action Steps Write down project objectives Impact Outcome Process

W W W. C E S. C L E M S O N. E D U / G E / Step 2: Project Objectives Increase graduation rate from 50% to 75% Increase first-time Calculus passing rate from 50% to 70% Students have positive opinion of group work Secure dormitory and classroom space

W W W. C E S. C L E M S O N. E D U / G E / Step 3: Create a Model of Change Identify assumptions you can assess Choose relationships to test based on Resources Where you anticipate problems Where you have control / can make improvements Link what you do to what you expect to happen Example Common residence / classes  in between, we need a sound theory of why this will happen  more will graduate

W W W. C E S. C L E M S O N. E D U / G E / Step 3 Action Steps Model of change as specific / complete as needed Review model assumptions Use criteria to prioritize Resources Relevance Control

W W W. C E S. C L E M S O N. E D U / G E / Step 3: Create a Model of Change Common residence / classes  affiliation Affiliation  support Support  performance Performance  future performance Future performance  more will graduate

W W W. C E S. C L E M S O N. E D U / G E / Step 4: Criteria and Indicators Validity Reliability Sensitivity Ease of interpretation Usefulness Example: define Graduation rate Course pass rate Affiliation Life-long learning Enrollment

W W W. C E S. C L E M S O N. E D U / G E / Step 4 Action Steps Define a set of indicators and criteria Impact Outcome Process

W W W. C E S. C L E M S O N. E D U / G E / Step 4: Criteria and Indicators Graduation rate # graduated / # in original cohort Calculus pass rate # A, B, C / # in original cohort D’s are no good – must be retaken Affiliation # in study group at end of sophomore year / # students Number of students enrolled in program

W W W. C E S. C L E M S O N. E D U / G E / Step 5: Data Sources Exams Surveys Observations Student records SAT scores Frequency Resources Guidance Change Example Institutional Research office (annual) Course records (each semester) Registrar Admissions Assessment office Process - monitor until achieved

W W W. C E S. C L E M S O N. E D U / G E / Step 5 Action Steps Define data sources Define frequency of measurement

W W W. C E S. C L E M S O N. E D U / G E / Step 5: Data Sources Grad / retention rate – Inst. Res. – Annual Calculus performance – Math dept. – includes course grades and common exam results –end of semester Dorm space allocated – Housing – monitor until achieved Survey of program participants Observer evaluations of class interaction SAT scores—Admissions

W W W. C E S. C L E M S O N. E D U / G E / Step 6: Design Evaluation Research Selection Mortality Placebo is not an issue Qualitative Interviews Focus groups Systematic observation Quantitative Non-experimental -Posttest only -Pretest-Posttest Quasi-experimental -Time Series -Nonequivalent control Experimental -Pretest-Posttest Control -Multiple Intervention

W W W. C E S. C L E M S O N. E D U / G E / Step 6 Action Steps Design evaluation research studies for key questions

W W W. C E S. C L E M S O N. E D U / G E / Step 6: Example Designs Example-Interviews Student expectations— invitation process modified Resource utilization— approaches to motivate attendance Example-Efficacy Non-equivalent control group Test for selection bias using baseline measures Matched pairs / groups Calculus grades and overall GPR

W W W. C E S. C L E M S O N. E D U / G E / Step 7: Monitor and Evaluate Establish project information system Budget for evaluation Evaluation meetings Review and revise evaluation plan Carry out studies

W W W. C E S. C L E M S O N. E D U / G E / Step 8: Use and Report Results Report results – to everyone Use results to make improvements

W W W. C E S. C L E M S O N. E D U / G E / Conclusions Planning and assessment is essential Start small if resources are limited Develop a plan before starting NSF and other agencies support well- designed educational research CCLI—EMD / A&I, ASA, and other programs Seek appropriate partners from education, psychology, sociology, statistics, etc.