Heidi Diefes-Dux Adam Carberry Matthew Siniawski Sara Atwood FOEE September 27, 2016 Data-Driven Instruction using Standards-Based Grading (SBG) Heidi Diefes-Dux Adam Carberry Matthew Siniawski Sara Atwood

My Experience with Data-Driven Instruction First-Year Engineering (N=1500+) Model-Eliciting Activities & Design Challenges Standards-Based Grading (SBG) for 4 years Student and grader data used to improve: Problem design Assessment strategies (e.g. learning objective clarification, rubric design, and feedback) Student instruction Instructor and TA training

Formative Assessment Systematic way for teachers and students to gather evidence of learning and use data to improve teaching and learning Purpose is to close the gap between expected and actual learning

assess students’ achievement of these? ABET Program Outcomes – broad statements of significant and essential learning that should be demonstrated upon graduation Course Goals – broad statements of what students should be able to do by the end of the course Course Learning Objectives – specific criterion-referenced observable behaviors (tied to individual units of instruction) But do we explicitly assess students’ achievement of these?

Standards-based Grading (SBG) Assessment of student’s mastery of essential standards or learning objectives for a course Provides clear, meaningful, and personalized feedback Directly connects students’ development with specific objectives (or standards) Promotes learning and encourages continuous improvement Provides fairness and transparency in the grading process Provides evidence for course and program assessment Sadler, 2005; Tomlinson & McTighe, 2006; Carpinelli et al., 2008; Scriffiny, 2008

Standards-Based Analytic Rubric LO02.00 Create and evaluate a x-y plot suitable for technical presentation No Evidence Underachieved Partially Achieved Fully Achieved More than 4 errors from Fully Achieved list 3-4 errors from Fully Achieved List 1-2 errors from Fully Achieved list Plot command used Independent & dependent variable on correct axis Plot formatted correctly (markers, lines, legend) Hold or equivalent is used Appropriate title Appropriate axis labels (with units)

Problem Use MATLAB to create a single plot with the populations of Rabbits versus Time and Wolves versus Time.

Rubric Item Applied to Problem Fully Achieved Plot command used Independent & dependent variable on correct axis Plot formatted correctly (markers, lines, legend) Hold or equivalent is used Appropriate title Appropriate axis labels (with units) ✓ ✓ ✓ ✓ ✓ X

Rubric Item Applied to Problem Partially Achieved 1-2 errors from Fully Achieved ✓

Overall Assessment of Learning Objective LO02.00 Create and evaluate a x-y plot suitable for technical presentation Assessment % No Evidence Under-achieved Partially Achieved Fully Achieved Problem Set 01 2% 3% 23% 71% Problem Set 02 9% 5% 27% 58% Exam 1 18% 51% 26%

Identification of Variability Across Sections LO GPA Learning Objective 15.5 15.7 Assignment PS07 PS08 Section A 3.38 3.77 2.68 Section B 3.47 3.73 3.80 Section C 3.79 3.92 3.03 … Section G 2.21 3.14 0.00 Section H 3.57 3.71 1.48 Section I 3.84 3.65 3.51 All Sections: 3.32 3.52 2.71 Leads to Improved Problems, Rubrics, and Training

Me: F TA: B Me: C/D TA: A Me: A TA: C/D Me: C/D TA: C/D LO 13.03 Appropriately use the best-fit linear model to make predictions F = No Evidence C/D = Under B = Partially A = Fully Achieved Anything less than requirements for underachieved Any 1 elements missing or incorrect from the list for fully achieved   Not Used for Grading Problem 2 Predicts aspirin yield for wintergreen values within range of original data set Acknowledges wintergreen values outside range of data cannot be used for predicting aspirin yield Answer key solution 𝒚=𝟎.𝟐𝟗𝒙+𝟎.𝟏𝟑 Student 3 Solution: y=0.3166667x+0.9473684 (Note: Their graph was y = .2939x + .1339) For 9 g: 2.5333g (Their equation yields 3.797) For 12 g: 3.4833 g (Their equation yields 4.747) For 16 g: 4.75 g (Their equation yields 6.014) For 30 g: 9.18333g (Their equation yields 10.447) Me: F TA: B For 9 g: 2.78g, assuming the linear function holds outside the data range For 12 g: 3.66 g For 16 g: 4.84 g For 30 g: 8.95 g, assuming the linear function holds outside the data range Student 4 Solution: y=0.2939x+0.1339 For 9 g: 2.7774 This is not an accurate prediction because 9g is outside of the given data set used for the prediction.) For 12 g: 3.6594 For 16 g: 4.8354 For 30 g: 8.9514 (This is not an accurate prediction because 30g is outside of the given data set used for the prediction.) Me: A TA: C/D Student 1 Solution: 0.2939x+0.1339 For 9 g: 2.779 g of aspirin For 12 g: 3.6607 g of aspirin For 16 g: 4.8363 g of aspirin For 30 g: 8.9509 g of aspirin Student 2 Solution: y=0.2939x+0.1339 For 9 g: Not in model range, could not b computed For 12 g: 0.472 g For 16 g: 4.836 g For 30 g: 8.951 g Me: C/D TA: A Me: C/D TA: C/D

What can we learn from using SBG? We can identify the most difficult learning objectives and react in real time or improve for next time

We can share best practices Identify leading sections Identify lagging sections

Next Steps Best practices and barriers for SBG Grader training strategies Enable easier post-processing of LO data Better post-assignment class discussions Quicker identification of grader and instructional challenges and successes Student dashboard for easier LO performance tracking

http://sbghub.lmu.build/ Making Grades Meaningful - Standards-based Grading for Engineering Project Courses, Grant No. DUE-1503794