Tweaking the pilot A Case Study from DVMT 100 at Frostburg State University Dr. Megan E. Bradley

DVMT credits, does not count toward graduation or GPA*3 credits, does not count toward graduation or GPA* Must take if need MATH 102 (College Algebra) or MATH 106 (Algebra with Calculus – Business majors)Must take if need MATH 102 (College Algebra) or MATH 106 (Algebra with Calculus – Business majors) About 450 students per yearAbout 450 students per year 1078% increase since inception in % increase since inception in 1985 No budget increaseNo budget increase

Course Issues Failure rate with gender gap in DVMT 100:Failure rate with gender gap in DVMT 100: 41% failure rate overall41% failure rate overall 44% rate for males; 35% rate for females44% rate for males; 35% rate for females Failure rate in next math course:Failure rate in next math course: COURSEDVMT DWF RATE NON-DVMT DWF RATE DIFFERENCE MATH 10256%39%16% MATH 10643%33%11%

Course Issues Staffing issuesStaffing issues Relied solely on undergraduate students to teachRelied solely on undergraduate students to teach Course DriftCourse Drift Delivery: ½ sections all face-to-face (f2f); other ½ all computer labDelivery: ½ sections all face-to-face (f2f); other ½ all computer lab Different textbook, syllabus, point systemDifferent textbook, syllabus, point system No system for checking reliability of gradingNo system for checking reliability of grading

What we did Emporium Model done rightEmporium Model done right 1 large lecture1 large lecture Rest of time in labRest of time in lab Hired new staff member to serve as lead instructorHired new staff member to serve as lead instructor Undergraduates became ULAs, shifting role to lab assistantUndergraduates became ULAs, shifting role to lab assistant Added material to help students in next math classAdded material to help students in next math class

Pilot – spring ‘11 Traditional lectureTraditional lecture all face-to-face (f2f) classesall face-to-face (f2f) classes no online workno online work taught by trained undergraduatestaught by trained undergraduates point system for course gradepoint system for course grade 1 final exam but could have earned other points with previous assignments to make final exam not have much weight1 final exam but could have earned other points with previous assignments to make final exam not have much weight RedesignRedesign Lecture 1x/week by instructor & lab 4x/week with trained Undergraduate Learning Assistants (ULAs) using ASAlgebra by PlatoLecture 1x/week by instructor & lab 4x/week with trained Undergraduate Learning Assistants (ULAs) using ASAlgebra by Plato 3 modules & corresponding exams3 modules & corresponding exams Mastery learning – retake exams until passedMastery learning – retake exams until passed Pass course by passing all 3 modules with 80% or higherPass course by passing all 3 modules with 80% or higher Extra credit for attending and doing online homework & evaluatesExtra credit for attending and doing online homework & evaluates Traditional lectureTraditional lecture all face-to-face (f2f) classesall face-to-face (f2f) classes no online workno online work taught by trained undergraduatestaught by trained undergraduates point system for course gradepoint system for course grade 1 final exam but could have earned other points with previous assignments to make final exam not have much weight1 final exam but could have earned other points with previous assignments to make final exam not have much weight RedesignRedesign Lecture 1x/week by instructor & lab 4x/week with trained Undergraduate Learning Assistants (ULAs) using ASAlgebra by PlatoLecture 1x/week by instructor & lab 4x/week with trained Undergraduate Learning Assistants (ULAs) using ASAlgebra by Plato 3 modules & corresponding exams3 modules & corresponding exams Mastery learning – retake exams until passedMastery learning – retake exams until passed Pass course by passing all 3 modules with 80% or higherPass course by passing all 3 modules with 80% or higher Extra credit for attending and doing online homework & evaluatesExtra credit for attending and doing online homework & evaluates

assessment Pass/Fail ratesPass/Fail rates Scores on “core questions”Scores on “core questions” Questions that show up on the redesign module exams & the final exams for the traditional sectionsQuestions that show up on the redesign module exams & the final exams for the traditional sections Focus groupsFocus groups

Pilot results Pass/fail Pass/fail Historical failure rate: : 41% Historical failure rate: : 41% Redesign failure rate: 47.2% which was significantly worse than… Redesign failure rate: 47.2% which was significantly worse than… Traditional failure rate: 22.6% Traditional failure rate: 22.6% Males failed more than females Males failed more than females

Pilot results Core questionsCore questions Difficult to use final grades due to different grading systemsDifficult to use final grades due to different grading systems Considering all core questions, a one-way ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = , p =.000, eta2 =.327.Considering all core questions, a one-way ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = , p =.000, eta2 =.327. Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%). Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%). Core questionsCore questions Difficult to use final grades due to different grading systemsDifficult to use final grades due to different grading systems Considering all core questions, a one-way ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = , p =.000, eta2 =.327.Considering all core questions, a one-way ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = , p =.000, eta2 =.327. Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%). Redesign students (X = 87.98%) performed significantly better than traditional students (X = 63.14%).

Pilot results Core questionsCore questions Below is a breakdown of core questions per module,Below is a breakdown of core questions per module, Students from the redesign section scored significantly higher than traditional sections for all three modules:Students from the redesign section scored significantly higher than traditional sections for all three modules: M1: Redesign (X = 86.20%) > traditional (X=83.66%)M1: Redesign (X = 86.20%) > traditional (X=83.66%) M2: Redesign (X = 84.90%) > traditional (X=74.07%)M2: Redesign (X = 84.90%) > traditional (X=74.07%) M3: Redesign (X = 90.85%) > traditional (X=59.05%)M3: Redesign (X = 90.85%) > traditional (X=59.05%)

Pilot results Regression indicated which of course activities significantly related to student grade on core questions. Regression indicated which of course activities significantly related to student grade on core questions. Attendance: correlated but weak Attendance: correlated but weak Online homework: correlated but weak Online homework: correlated but weak Online evaluates : strongly correlated Online evaluates : strongly correlated Homework & evaluates: needed 80% to pass and move on Homework & evaluates: needed 80% to pass and move on Evaluates: Often only had 4 questions so needed to get perfect score. Evaluates: Often only had 4 questions so needed to get perfect score.

Additional results We examined students’ time on task and when they were using software.We examined students’ time on task and when they were using software. Reviewed focus group suggestions.Reviewed focus group suggestions. Compared student performance on certain items in traditional sections.Compared student performance on certain items in traditional sections. Created hypotheses and tested them out as best as we could.Created hypotheses and tested them out as best as we could. Reassessed the teamReassessed the team

Issues & tweaks 1.Students compared DVMT 100 sections. 2.Redesign students did not effectively use their lab time wisely. 3.Redesign students did not have enough deadlines – 1x/module, night before exam. 4.Students fell behind next module while retaking previous module exam. 1.Fall 2011 – full implementation. 2.Changed lab to 2x/wk and used technology to block other sites. 3.Created several deadlines with last deadline before test review day. 4.Added retake week after Mod1. 1.Fall 2011 – full implementation. 2.Changed lab to 2x/wk and used technology to block other sites. 3.Created several deadlines with last deadline before test review day. 4.Added retake week after Mod1.

Issues & tweaks 1.The grading system in the redesign confused students. 2.Redesign students found and exploited a loophole about retaking modules next semester. 3.Lab assistants were scattered across different labs. 4.No pedagogy to address gender gap. 1.Revised to be based on weights that required and rewarded important course aspects. 2.Modified retaking of modules. 3.Assigned lab assistants. 4.Created Train Your Brain Program 1.Revised to be based on weights that required and rewarded important course aspects. 2.Modified retaking of modules. 3.Assigned lab assistants. 4.Created Train Your Brain Program

Issues & tweaks 1.Failure rate on first version of module exam was very poor: Mod1 = 27% passed Mod1 = 27% passed Mod2 = 20% passed Mod2 = 20% passed Mod3 = 17% passed Mod3 = 17% passed 1.Implemented PreModule Exam Earn 85% or higher – no need to take Module exam Reward studying & doing well 1.Implemented PreModule Exam Earn 85% or higher – no need to take Module exam Reward studying & doing well

Fall 2011 pass/fail rate 20.3% failure rate overall 19.7% rate for males; 21.3% rate for females Gender analyses NOT statistically significant. Fall 2011 pass/fail rate 20.3% failure rate overall 19.7% rate for males; 21.3% rate for females Gender analyses NOT statistically significant. Full implementation results Remember this?Remember this? Failure rate with gender gap:Failure rate with gender gap: 41% failure rate overall41% failure rate overall 44% rate for males; 35% rate for females44% rate for males; 35% rate for females Pilot redesign failure rate: 47.2%Pilot redesign failure rate: 47.2%

Why Stop There?

assessment Score on pre-test in next math classScore on pre-test in next math class Non-DVMTNon-DVMT Redesign DVMTRedesign DVMT Traditional DVMTTraditional DVMT

Pre-Test in Next Math Course

Why Stop There?

assessment Final grades in next math classFinal grades in next math class Who passed with a C or higher?Who passed with a C or higher?

Impact of changes Deadlines = large % students completed deadlinesDeadlines = large % students completed deadlines Weights & lab changes = better attendance and time on taskWeights & lab changes = better attendance and time on task Train Your Brain = no gender gap, better performance overallTrain Your Brain = no gender gap, better performance overall

Impact of changes 1.Failure rate on first version of module exam was very poor: Mod1 = 27% passed Mod1 = 27% passed Mod2 = 20% passed Mod2 = 20% passed Mod3 = 17% passed Mod3 = 17% passed 1.PreModule Exam results Module 1 Premod: 36% passed Version 1: 72% passed Module 2 Premod: 16.2% passed Version 1: 60% passed Module 3 Premod: 18.3% passed Version 1: 53% passed 1.PreModule Exam results Module 1 Premod: 36% passed Version 1: 72% passed Module 2 Premod: 16.2% passed Version 1: 60% passed Module 3 Premod: 18.3% passed Version 1: 53% passed

Overall recommendations Look, look, look.Look, look, look. Add structure.Add structure. Improve based on evidence (from pilot, from other redesigns, from published research)Improve based on evidence (from pilot, from other redesigns, from published research) Add psychologyAdd psychology Provide incentivesProvide incentives Spacing effectSpacing effect Practice effectPractice effect Mastery learningMastery learning

Your reward