1. July 8, 2008In vivo experimentation: 1 Step by Step In Vivo Experimentation Lecture 3 for the IV track of the 2011 PSLC Summer School Philip Pavlik Jr. Carnegie Mellon University

2. July 8, 2008In vivo experimentation: 2 So, you want to run an in vivo experiment? u What is in vitro? –Self-explanation (Chi, Bassok, Lewis, Reimann, & Glaser, 1989) u What is in vivo? –Example take from: u Booth, Siegler, Koedinger & Rittle-Johnson (2008) Booth, Siegler, Koedinger & Rittle-Johnson (2008)

3. July 8, 2008In vivo experimentation: 3 Step 0 u Become an expert in a content area (e.g., electrodynamics). –Find a textbook –Talk to an expert –(Potentially the most time consuming step.)

4. July 8, 2008In vivo experimentation: 4 Step 1 u Hypothesis –Self-explanation: A self-generated explanation of presented instruction that integrates the presented information with background knowledge and fills in tacit inferences. (from pslc wiki) –Correct examples u Facilitate construction of high-feature validity knowledge components. –Incorrect examples u Weaken low-feature validity knowledge components.

5. July 8, 2008In vivo experimentation: 5 Step 2 u Select a domain –Algebra, geometry, physics, chemistry, Chinese, English u Select a LearnLab site –Riverview High School –Central Westmoreland Career & Technology Center u Talk with instructors –Collaboratively identify “hot spots” in curriculum.

6. July 8, 2008In vivo experimentation: 6 Step 3 u Develop materials –Assessments –Tutor implementation u Get IRB approval.

7. July 8, 2008In vivo experimentation: 7 Corrective self-explanation explanation of incorrect worked example

8. July 8, 2008In vivo experimentation: 8 Typical self-explanation explanation of correct worked example

9. July 8, 2008In vivo experimentation: 9 Step 4 Self-Explanation of Correct Examples NoYes No “As-is” Control Typical YesCorrective Typical + Corrective (half of each) Self-Explanation of Incorrect Examples Design Study

10. July 8, 2008In vivo experimentation: 10 Step 5 u Formulate a Procedure –Pretest u Two forms of the assessment –Intervention u Solve problems with Cognitive Tutor Algebra –Post-test u Talk alternate form of assessment

11. July 8, 2008In vivo experimentation: 11 Step 6 u Run experiment –Collect log files –Collect paper-based assessments u Send Log Data to DataShop –Test Hypotheses

12. July 8, 2008In vivo experimentation: 12 Step 7 u Report your results to the scientific community –Create a wiki page –Give a conference talk –Write a journal paper

13. July 8, 2008In vivo experimentation: 13 Number of correctly answered problems

14. July 8, 2008In vivo experimentation: 14 Number of conceptual errors

15. July 8, 2008In vivo experimentation: 15 Goto Step 1 u Using the results from your in vivo experiment: –Replicate with a new population –Replicate with a new domain –Replicate and Extend in a lab experiment

16. July 8, 2008In vivo experimentation: 16 Summary u Step: 0.Become an expert 1.Generate a hypothesis (in vitro => in vivo) 2.Select domain, site, instructors 3.Develop materials 4.Design study 5.Formulate a Procedure 6.Run experiment & log to DataShop 7.Report your results u Goto Step 1

17. July 8, 2008In vivo experimentation: 17 Any Questions?