Engineering Education Conference - Spring 2009 Increasing Assessment Effectiveness in a Time of Decreasing Budgets Increasing Assessment Effectiveness in a Time of Decreasing Budgets Ken Stanton

Engineering Education PhD in Engineering Education at Virginia Tech  First cohort began Fall 2008  16 total students  Coursework in engineering, education, and engineering education  Teaching and research assistantships available  Graduate Recruitment Days every March and October– travel grants available! 

Engineering Education Introduction – The Problem  ABET EC2000 standards are a lot of work for engineering departments  Purposes: assessment and accountability  Problems: some faculty less familiar with assessment, requiring more and more of programs  Solutions: faculty development, assessment experts  Economic downturn affects education budgets  Problem: difficult to implement the solutions above  Enrollment, and therefore class size, increases  Time and resources limited  Any solutions: must be efficient, use what we have, be simple and easy to implement

Engineering Education Introduction – Proposed Solution  Address the need to increase assessment effectiveness from three angles 1.Utilize lean assessment techniques 2.Implement a process that maximizes intrinsic motivation 3.Take advantage of efficiency of assessment methodology Increased Assessment Effectiveness Lean Assessment Techniques Increasing Intrinsic Motivation Assessment Method Efficiency

Engineering Education Innovative Solution – Lean Assessment  Goal of assessment Teaching toward learning objectives Assignment created for student Student completes assignment Grade and feedback given to student Typical process for an assignment

Engineering Education Innovative Solution – Lean Assessment Teaching toward learning objectives Assessment created for student Student completes assessment Feedback given to student, may be graded Collect and analyze evidence of learning Use to improve teaching and assessments Typical assessment process

Engineering Education Innovative Solution – Lean Assessment  Definition  simple, efficient, and effective assessment techniques  based in the principle of lean manufacturing  productive and simple methods  still attain useful data  Methods  Some existing  Angelo and Cross (1993)  Some modified  Some novel

Engineering Education Innovative Solution – Lean Assessment Technique:Time Requirement:Purpose:Description:Notes: Rubrics (for various activities) MED; LOW if shared development Categorize important assessment and grading criteria; set up structure to summarize performance; provide clear expectations to students Important criteria are categorized and assigned point values in a table, then performance levels (e.g. novice, expert) described Many resources online for pre- defined or custom rubrics; coordinating with other faculty can decrease time investment and increase consistency of course/material One-minute problem summary LOWEvaluate student understanding of assigned problems and underlying concepts At the end of a problem, the student takes one minute to explain what major concepts were used and what was learned Good to prompt students with leading question to answer in problem summary  Use to assess teaching and learning

Engineering Education Innovative Solution – Lean Assessment Technique:Time Requirement:Purpose:Description:Notes: Individual project reports (in team project) MEDAssess and improve individual report- writing abilities When team reports are required, have each student submit individual report at least one week preceding Reviewing 3-5 times as many reports can be time consuming; a rubric should be used and specific aspects of reports focused on Oral examsLOW and can reduce time Assess oral abilities and allow for deeper probing of key topics; mimic business meeting Students present and discuss project instead of written exam/project, one- on-one with instructor; oral questions provide richer evidence of knowledge and learning; grade is given before leaving meeting A form of authentic assessment, abilities are tested in a more realistic manner that closely reflects real-world; time can be less than administering and grading exams/ projects

Engineering Education Innovative Solution – Implementation  These assessment methods alone do not solve the problem  Consider motivation theory in implementation  Self-Determination Theory (SDT) by Deci & Ryan (2000) has 3 needs 1.Competence – confidence, self-efficacy 2.Autonomy – choice that aligns with identity / values 3.Relatedness – connection to others identify with  Satisfaction of these leads to internalization of motivation – move from extrinsic to intrinsic

Engineering Education Innovative Solution – Implementation  Competence  Lacking due to inexperience with assessment  Lean assessment techniques – simple, effective  Faculty helping each other  Autonomy  Challenged by requirements being handed down  Helped somewhat by flexibility of ABET criteria  Use assessment to address faculty’s teaching concerns  Relatedness  Can take opportunity to connect with students through feedback

Engineering Education Innovative Solution – Implementation 1.Review program educational objectives (PEOs), program learning outcomes, and course outcomes established for the program and course(s) of interest. 2.Interview faculty for input on assessment and what they would like to improve in their teaching and students’ performance. Review current assignments and assessments being used in their course(s) of interest. 3.Identify lean assessment techniques that A.are good for measuring the outcomes from #1 B.meet goals that faculty name as important to them C.can complement/replace existing assignments and activities in the course(s).

Engineering Education Innovative Solution – Implementation 4.Briefly illustrate how the lean assessment techniques match the faculty member’s goals and can be used alongside or in place of existing course materials. 5.Assist with familiarization and use of the assessments, if necessary, and encourage sharing the expectations and motives with students.  Likely to be effective seeing as studies found faculty generally hold positive views of assessment

Engineering Education Innovative Solution – Assessment Methodology  Many times we develop courses by deciding material then assessments  Less efficient and less productive  Wiggins & McTighe (2005) coined “Backward Design”  Work from learning objectives to ways to measure them (assessments) to how to teach it  Ensures efforts are goal-oriented and assessments can be used for accreditation  Using this process can increase efficiency of teaching and assessment efforts

Engineering Education Introduction – Proposed Solution  Address the need to increase assessment effectiveness from three angles 1.Utilize lean assessment techniques 2.Implement a process that maximizes intrinsic motivation** 3.Take advantage of efficiency of assessment methodology Increased Assessment Effectiveness Lean Assessment Techniques Increasing Intrinsic Motivation Assessment Method Efficiency

Engineering Education Future work  Dissertation study  From a motivational standpoint, how do engineering faculty members engage in assessment of student learning?  Also look at ways to integrate assessment into the culture of teaching and learning

Engineering Education THANK YOU QUESTIONS?