Revolutionizing the Problem Solving Feedback Process ASEE Global Colloquium 2009 Marcia Horton Margaret Waples Claire Masson

Agenda Registrations Live Demo Coffee Break Report on Engineering Studies in India Efficacy Research (optional)

Mastering expanding

Live Demo

10% 9% 19% 13% 23% 26%

Questions / Answers on Live Demo

COFFEE BREAK Return to hear about: India and Engineering Studies Efficacy Research (Optional)

India Research conducted in urban settings New Delhi Mumbia Goa Hyderabad

Background Half its population (~1 billion people) is under the age of 25 ~10% of Indians aged are enrolled in university (40% of 25+ are illiterate) India produces about 400,000 engineers annually By 2020, global spending on engineering services is estimated to be $1 trillion. (India’s share expected to be $ billion)

POSITIVES –India has largest pool of engineering talent –Outsourcing has already forged strong client relationships –English speaking

NEGATIVES –Missing skill sets: “soft skills” Finishing schools –Small percentage of engineering graduates pursue master’s or doctoral degrees. ~1000 Ph.D.’s annually –Weak physical and engineering infrastructures

Education Three months met with 265 educators, 200 students and 38 vendors Engineering colleges remain primarily teaching institutions and are not vibrant research institutions –Small percentage of engineering majors pursue masters or doctorates in India –about 1000 Ph.D.’s in engineering annually Demand for Engineering education out strips supply of qualified instructors –Variable infrastructure at universities and high schools

Identify the mistake and type in the correct answer Soft Skills: MyWritingLab

Soft Skills: MyReadingLab

Efficacy

OPTIONS Historical Study using MC: increase scores for all levels of students Experimental Study on problem solving transfer skills

University at Buffalo, NY An observational study Score Trends

F04: No MC

F05: No MC

F07: No MC

F06: No MC “As for Fall of '06, we had what we call the "October Surprise" an ice storm which hit the night of exam 1. The exam was delayed 1 week and many of our students (and professors) were out of their homes for about 2 weeks and living in the campus center, hotels, and other accommodations. There was no power and limited access to internet, phone, etc... This may have had a global effect on the scores, particularly for exam 1, and perhaps a lasting effect on the semester.”

F08: MasteringChemistry

Data Speaks: The final exam score distribution for a student belonging to the bottom 25% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

Data Speaks: The final exam score distribution for a student belonging to the middle 50% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

Data Speaks: The final exam score distribution for a student belonging to the top 25% of the class F04, 05, 07: Baseline F06: Ice storm F08: MC

Probability that a student in the bottom 25% of the class would obtain a final exam score ≥ 50 F04 Baseline F05 Baseline F06 Ice Storm F07 Baseline F08 MC 42%26%~ 0%17%81% Assuming: 1.that there is no teacher effect in F08, 2.that the F08 final exam is comparable in difficulty and coverage, 3.no drastic changes to the F08 course structure beside MC, which at present seems to be quite reasonable, A Powerful Result: Students who were less skillful or were at risk of failing the course were positively affected by MC.

Clemson Experimental Study: problem solving transfer skills

Study Design Class Tutorials EOCs Identical Tutorials (but hints/feedback/learning goals removed) Identical EOCs N~260 n~130 Preparatory Items Test Items compare Tutorial Group Non-Tutorial Group

Chap 4: In 6/8 items the Tutorial Group outperforms the Non-Tutorial Group

Results (over 4 assignments) A tutorial group student makes 10% fewer errors on avg. (13/16 items) 34% fewer solution requests were made by tutorial students Preparatory phase: A tutorial group student makes 8% fewer errors on avg. (5/8 items) 7% fewer solution requests were made by tutorial students Test phase:

BACK UP IF NO INTERNET These slides would be in front of India and Efficacy, but they are moved here to keep them out of the way for now.

Student Benefits

A Tutorial Problem

Tutoring Problem in Engineering Some problems have randomized variables

Student enters “.711”

Student A enters “.94”

Students open optional Hints

Declarative Hint Student Selects Hints for Individualized Learning Path Many Hints are Socratic; emulating the Expert / Novice experience

Hints also provide students with Individualized Feedback

Rigid Body Free Body Diagrams Students select and label vectors

Rigid Body Free Body Diagrams Students draws labels and receives coaching

Rigid Body Free Body Diagrams The vector information tab helps students accurately draw angles.

Instructor Benefits

Automatic Grading Shades of pink to red show struggling students

Gradebook Click to see one Student’s marks, detailed work and time stamp

Student Level One Student’s score, time stamp. Click on problem name to see her work

Student Level See Time Stamp Review wrong answer attempts with coaching advice

“Law of Conservation of Energy” Most difficult Most time-consuming Lowest average score What did my students misunderstand? Assignment Level

Class Level Compare My Class to System Avg: time and difficulty Click on Title to see more info

Class Level 50% of the class made a common error and received coaching tip; another 50% made a trig error and received different advice.

Thank You!