1. Revolutionizing the Problem Solving Feedback Process ASEE Global Colloquium 2009 Marcia Horton Margaret Waples Claire Masson

2. Agenda Registrations Live Demo Coffee Break Report on Engineering Studies in India Efficacy Research (optional)

3. Mastering expanding

4. Live Demo www.masteringengineering.com

6. 10% 9% 19% 13% 23% 26%

7. Questions / Answers on Live Demo

8. COFFEE BREAK Return to hear about: India and Engineering Studies Efficacy Research (Optional)

9. India Research conducted in urban settings New Delhi Mumbia Goa Hyderabad

10. Background Half its population (~1 billion people) is under the age of 25 ~10% of Indians aged 18-25 are enrolled in university (40% of 25+ are illiterate) India produces about 400,000 engineers annually By 2020, global spending on engineering services is estimated to be $1 trillion. (India’s share expected to be $125-150 billion)

11. POSITIVES –India has largest pool of engineering talent –Outsourcing has already forged strong client relationships –English speaking

12. NEGATIVES –Missing skill sets: “soft skills” Finishing schools –Small percentage of engineering graduates pursue master’s or doctoral degrees. ~1000 Ph.D.’s annually –Weak physical and engineering infrastructures

13. Education Three months met with 265 educators, 200 students and 38 vendors Engineering colleges remain primarily teaching institutions and are not vibrant research institutions –Small percentage of engineering majors pursue masters or doctorates in India –about 1000 Ph.D.’s in engineering annually Demand for Engineering education out strips supply of qualified instructors –Variable infrastructure at universities and high schools

14. Identify the mistake and type in the correct answer Soft Skills: MyWritingLab

15. Soft Skills: MyReadingLab

16. Efficacy

17. OPTIONS Historical Study using MC: increase scores for all levels of students Experimental Study on problem solving transfer skills

18. MC @ University at Buffalo, NY An observational study Score Trends

19. F04: No MC

20. F05: No MC

21. F07: No MC

23. F06: No MC “As for Fall of '06, we had what we call the "October Surprise" an ice storm which hit the night of exam 1. The exam was delayed 1 week and many of our students (and professors) were out of their homes for about 2 weeks and living in the campus center, hotels, and other accommodations. There was no power and limited access to internet, phone, etc... This may have had a global effect on the scores, particularly for exam 1, and perhaps a lasting effect on the semester.”

24. F08: MasteringChemistry

25. Data Speaks: The final exam score distribution for a student belonging to the bottom 25% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

26. Data Speaks: The final exam score distribution for a student belonging to the middle 50% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

27. Data Speaks: The final exam score distribution for a student belonging to the top 25% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

28. Probability that a student in the bottom 25% of the class would obtain a final exam score ≥ 50 F04 Baseline F05 Baseline F06 Ice Storm F07 Baseline F08 MC 42%26%~ 0%17%81% Assuming: 1.that there is no teacher effect in F08, 2.that the F08 final exam is comparable in difficulty and coverage, 3.no drastic changes to the F08 course structure beside MC, which at present seems to be quite reasonable, A Powerful Result: Students who were less skillful or were at risk of failing the course were positively affected by MC.

29. MP @ Clemson Experimental Study: problem solving transfer skills

30. Study Design Class Tutorials EOCs Identical Tutorials (but hints/feedback/learning goals removed) Identical EOCs N~260 n~130 Preparatory Items Test Items compare Tutorial Group Non-Tutorial Group

31. Chap 4: In 6/8 items the Tutorial Group outperforms the Non-Tutorial Group

32. Results (over 4 assignments) A tutorial group student makes 10% fewer errors on avg. (13/16 items) 34% fewer solution requests were made by tutorial students Preparatory phase: A tutorial group student makes 8% fewer errors on avg. (5/8 items) 7% fewer solution requests were made by tutorial students Test phase:

33. BACK UP IF NO INTERNET These slides would be in front of India and Efficacy, but they are moved here to keep them out of the way for now.

34. Student Benefits

35. A Tutorial Problem

36. Tutoring Problem in Engineering Some problems have randomized variables

37. Student enters “.711”

38. Student A enters “.94”

39. Students open optional Hints

40. Declarative Hint Student Selects Hints for Individualized Learning Path Many Hints are Socratic; emulating the Expert / Novice experience

41. Hints also provide students with Individualized Feedback

42. Rigid Body Free Body Diagrams Students select and label vectors

43. Rigid Body Free Body Diagrams Students draws labels and receives coaching

44. Rigid Body Free Body Diagrams The vector information tab helps students accurately draw angles.

45. Instructor Benefits

46. Automatic Grading Shades of pink to red show struggling students

47. Gradebook Click to see one Student’s marks, detailed work and time stamp

48. Diagnostics @ Student Level One Student’s score, time stamp. Click on problem name to see her work

49. Diagnostics @ Student Level See Time Stamp Review wrong answer attempts with coaching advice

50. “Law of Conservation of Energy” Most difficult Most time-consuming Lowest average score What did my students misunderstand? Diagnostics @ Assignment Level

51. Diagnostics @ Class Level Compare My Class to System Avg: time and difficulty Click on Title to see more info

52. Diagnostics @ Class Level 50% of the class made a common error and received coaching tip; another 50% made a trig error and received different advice.

53. Thank You! www.masteringengineering.com