 Measuring Your Success in Course Redesign

The Dirty Words in Education 1. Success 2. Efficacy 3. Assessment 4. Evaluations

Our Topics for Today 1. How will you measure success? 2. Will you use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)?  Mastering platform in Sciences similar to MyLab platform in other disciplines 3. MyLab & Mastering assignments will contribute what percentage to a students final course grade? = assessment 4. What do you do with the information evaluated?

Today’s Plan My course structure Implementation of a few simple techniques Quick ways to measure success How to use MyLab / Mastering to maximize student learning How do other data support MyLab / Mastering use? How much is too much (or too little!) for a grade item? How to use MyLab / Mastering for Course Assessment

My Course General Biology (year-long sequence) Typical # of students: (taught in two sections) This year 408 Typical # of different majors: 55 from all RIT Colleges ~ 50/50 science/non-science Freshmen -> grad students Current grade calculation (2013 – 2014): 65-70% exams; 15% Mastering Bio HW; 10% Mastering Bio quizzes; 5-10% in class participation

Before you measure success…

Are Students Prepared to Succeed? Preparation (lack of!) comes in many forms Reading / writing comprehension English as a Second Language (or 3 rd, 4 th, ….) Learning disabilities Fundamental knowledge in sciences Study skills Dedication to the course Dedication to the degree / college Etc……

Preparing Students for Success Homework & Quiz Assign immediately on Day 1 Provide timely feedback on the assignment “Suggestions” from instructor do hold water! Require regular assignments In Class Response Activities Every day! (“mock quiz” type) Immediate or timely feedback critical

Preparing Students for Success Be clear about the goals of pre-course activities E.g. pretests, diagnostic tests, etc. Benefit to students Not enrolled in a class in which they cannot succeed Decreases DWF rates Increase retention – intangibles Benefit to you / course Less time on “catch up”, More time on application Less stresses associated with DWF, office hours, etc.

 So you have “prepared” students … Now what?

Measuring Success – How? Of what? Retention (D/W/F) Common exams? Shared assessments across sections? Final exam scores Final course grades MyLab / Mastering Alternative learning initiatives

Measuring Success – How? Of what? What works for … You Your students Your discipline Your assessment office? Your supervisor?

Measuring Success – How? Of what? Let’s focus on MyLab / Mastering as a success tool … We can add in the other things later …

MyLab / Mastering = X % : Not much – trial only; assignments optional 3 attempts per question; unlimited time : Not much – trial only; assignments optional 3 attempts per question; unlimited time : 5% of grade; ~20% never did any; nearly all students who completed the assignments had a 100% avg 3 attempts per question; unlimited time : 15% of grade; all complete at least most; avg scores 88-93% 2 attempts per question; unlimited time

MyLab / Mastering = X % 2012: 20% of grade HW and 5% on quizzes; all students complete nearly all (> 97% completion rate) HW = 2 attempts per question; unlimited time Quiz = 1 attempt per question; 60 min time limit 2012 Example Mitosis/Meiosis HW: 96 questions; avg time spent = 44 minutes; avg score = 85.9% Homework due before lecture Result? No time in class on process – instead, theory!

MyLab / Mastering = X % 2013 – 2014: 15% of grade HW and 10% on quizzes; all students complete nearly all (> 98% completion rate) HW = 2 attempts per question; unlimited time; weekly Quiz = 1 attempt per question; 60 min time limit; weekly Food for thought … is this % too much? Will students cheat, copy, etc.?

 Everything has challenges …

What is your biggest challenge? Time is the untamed obstacle for everyone.. Take 5 minutes Sit down! (or better yet – while you are shampooing!) What information do you really want your students to retain? WHY? What do you want your students to leave your class knowing? WHY?

Challenges to Assessment Things that I have considered and toyed with … How much do I want the students to do outside of class on MyLab/Mastering? Limited to “Time Estimated” = 1 hour Less if possible! I want their attention!

Challenges to Assessment Things that I have considered and toyed with … What is the value of a long assignment if the students don’t do it, copy it from another student, just use all of the hints, etc.? For students who are failing exams, but have 100%+ on the homeworks, I look at their time spent in MyLab / Mastering Discuss issue with them in person and make note in the Institute Reporting

Challenges to Assessment Things that I have considered and toyed with … Are the students just memorizing answers, or are they applying the information and retaining it? Have stopped assigning reading / test bank questions Implement more low stakes quizzes Forced repeated recall of information in class

 You have data … Now what?

Student Data Assessment Why assess? My general interest Are the students learning what I think they are learning? How can I improve this course? University call to assess Gen Ed courses (Accreditation) Assessment needs? My interest Exam and homework scores before and after MyLab / Mastering implementation University Guidelines: 1 – 4 scale of comprehension

Student Data Assessment What was assessed? Kept it simple to start! Exam averages Add as I have interest / time! Participation in MyLab / Mastering Scores on MyLab / Mastering Time spent on MyLab / Mastering

Student Data Assessment Exam grades (presented herein) All stored in Excel TM since 2007 Mastering (not presented) Export grades & time spent from Mastering to Excel TM Processing, Compilation, and Statistical Analyses Excel TM

Exam Score Trends Err = St Err Avg n = 244

Talk to me data! Higher exam averages every year – but leveling out Significant differences only in plants – but trends are obvious Too many students from too many majors to see huge jumps in scores despite all efforts Tightening of exam scores Standard deviations and ranges decreasing Increasing amount of “average time spent” on MyLab / Mastering assignments

Assessment Challenges Why do exam grades change? Things I have considered What changes from year to year other than MyLab / Mastering? Students? Accessibility of tutors? Me? Order of topics covered? Lab instructors?

Assessment Challenges Why do exam grades change? Things that I want to consider Students Time spent on studying (other than MyLab / Mastering)? Major? College entrance exams? AP Bio? DWF rates by major and biology “experience” Me Lectures improved? Lecture structure change with online assignments?

“Napkin Notes” General observations: Increased time spent on assignments after 1 st exam of term Increased # questions in office hours on assignments Increased time spent “off topic” in lecture – not a bad thing! Changes implemented to date: Lecture time is more interactive Participation activities with the students (worksheets, group work, think-pair-share, flipped lectures, etc.) Modifications to assignments and quizzes to “challenge” the students in preparation for class and exams

Advice for Instructors Looking to Assess Success Don’t assess while you are teaching the course! Assessment can be done any time! Store the data, save it for later Even if you have time … subconsciously not good for your teaching! Don’t try to assess everything at once! Pick one exam in your course, and compare the average ± standard deviation (standard error) between years Even simpler? Compare exam averages between class sections. TA DA! You just did an assessment!

Advice for Instructors Looking to Assess Success Don’t expect MyLab / Mastering to make your students in to geniuses! These are tools! And how the students implement the tools (accept the tools!) will be highly variable. You may not see immediate changes in exam grades Accept the “intangible” as a result! You can not grade student enthusiasm (which means you can’t assess it – sorry!) Student enthusiasm is contagious – not just to other students – but also to you!

Advice for Instructors Looking to Assess Success What can you do with your data? FLAUNT IT! White papers Peer-reviewed papers Teaching portfolio Promotion dossier Assessment office

 Final thoughts …

What else is assessable? I got my class back! Not spending class time on giving quizzes, answering questions about homework format, etc. What do we do then? Talk about Biology! AND … most of my assessment is built in to MyLab / Mastering and my course management system! Don’t reinvent the wheel! Download grades, assign your assessment strategy,.. TA DA!

What else is assessable? HUGE time savings in my classes I have “recovered” 35 minutes per week in class (of 150 min) with no quizzes and homework things in class Prepared students means < 45 min lecture! > 105 min biology exploration! Time saving is so huge … how huge is it? Enough that I am completely “flipping” one section DATA! DATA! DATA! DATA!

But.. The Most Important Thing to Me?

What the Students Have to Say! "Mastering Bio helps a lot! The more you have us do, the more I learn!” "I wish that our exams were as interactive as Mastering! I love learning that way!" "(Favorite part about Mastering) aspects of the assignments helped me to better grasp the materials through the videos. I would not have done as well on those parts of the test without Mastering.” "More Mastering - 'nuf said!"

Let’s Recap 1. How will YOU measure success? 2. Will YOU use historical data (common exams, etc.) to support the efficacy of your assessments (MyLab & Mastering)? 3. MyLab & Mastering assignments will contribute what percentage to YOUR students final course grade? = assessment 4. What do YOU do with the information evaluated?

The Dirty Words in Education 1. Success 2. Efficacy 3. Assessment 4. Evaluations

 THOUGHTS!?