1. 1 Use of Mastery Software in the Undergraduate OM Course Peter J. Billington Colorado State University – Pueblo

2. 2 Undergraduate OM Course Junior Level Required of all Business students Blend of quantitative and qualitative Tools and techniques, e.g. Project management, waiting lines MRP, inventory, SPC, learning curves Statistics as a pre-requisite

3. 3 Mastery Software Quant Systems’ Adventures in Operations Management Modules for a number of topics Demo mode – shows how it works Practice mode – student can practice Certify mode

4. 4 Certify Mode Student has ~ 10 problems with some subsections, varies each module Read question, calculate answer, submit answer. If correct, moves to next question. If incorrect, indicates a “strike” and gives correct answer Cannot correct the submitted answer

5. 5 Certify continued 3 “strikes” and you’re out. Must start over. If answer all questions with fewer than 3 strikes, student certifies. Gets a certification number and prints out certification sheet. Submits this to professor.

6. 6 AiOM in the OM course 10 modules assigned in semester 10% of course grade “All of nothing” - Must certify in all modules to receive the 10%, otherwise zero % Started with required submission before corresponding exam If student missed first, they gave up

7. 7 AiOM in the course - continued Changed to: submit any time before end of semester, but strongly urged to submit before exam. Some students certified before exam. Some did not. Positive student comments on course evaluation. Ah ha – “Maybe I should find out if they are affecting test scores!”

8. 8 Collecting Data After each exam, noted if student had completed corresponding AiOM. 138 students 636 exams Normalized each exam average to 100

9. 9 Results “Yes” means student did certify AiOM module before exam. “No” means student did not. Significance tests have not yet been performed on the data.

10. 10 Results NoYesTotal % of exams 60.5%39.5%100% Average exam grade 96.9105.0 +8.1 100.0

11. 11 Analysis Were the exam results and % completed before exam related to Grade in pre-requisite statistics course? GPA of student? Other factors?

12. 12 Related to Stats Grade Grade in stats course Average Exam Score % Exams Yes A112.152.6% B97.832.7% C97.232.4% D92.520.0% F94.227.8% All100.039.5%

13. 13 Related to GPA Student GPAAverage Exam Score % Exams Yes 3.5-4.0113.870.8% 3.0-3.49105.136.7% 2.5-2.9997.541.0% 2.0-2.4990.719.1% All100.039.5%

14. 14 Average Exam Grade Grade in stats course NoYesDifferenceTotal A111.1113.1+2.0112.1 B94.7104.3+9.697.8 C95.7100.3+4.697.2 D90.799.7+9.092.5 F94.693.1-1.594.2 All96.92105.0+8.1100.0

15. 15 Average Exam Grade Student GPA NoYesDifferenceTotal 3.5-4.0113.6114.1+0.5113.8 3.0-3.49103.3108.1+4.8105.1 2.5-2.9995.2100.2+5.097.5 2.0-2.4988.997.9+9.090.7 All96.92105.0+8.1100.0

16. 16 Conclusions Certifying appears to improve grades. Makes little difference to top students. Makes a large difference for other students. GPA better predictor than stats grade.

17. 17 Next Add this semester’s 60 students and 300 exams. Significance tests. Break out by difficulty of test material? Is difference in MRP test greater than for learning curve test? Other factors to consider? For a student that did not complete all on time, did it make a difference? End the experiment! Require for all!