1. Strategies for Research Methods Tutoring using Latent Semantic Analysis David AllbrittonPeter Wiemer-Hastings Elizabeth ArnottJesse Efron DePaul University http://reed.cs.depaul.edu/peterwh/RMT

2. Talk presented at the annual meeting of the Society for Computers in Psychology, November 2003, Vancouver BC, Canada.

3. Abstract The concepts and skills taught in psychology research methods courses are fundamental to our field, yet are often difficult for students to master. RMT (Research Methods Tutor, a descendent of Autotutor) is an intelligent tutoring system that will supplement classroom instruction by engaging students in conversation about course topics, using LSA (latent semantic analysis) to understand student responses. Pedagogical strategies implemented in RMT are described, along with plans for evaluation and deployment.

4. The Need for Tutoring in Psychology Research Methods The skills taught in research methods are fundamental to psychological science Students have difficulty mastering these fundamental concepts in the classroom Tutoring can help

5. Advantages of Expert Human Tutoring Flexibility Effectiveness –2 Sigma (Bloom, 1984) –the “Holy Grail” of ITS Research

6. Advantages of an ITS (Intelligent Tutoring System) Practicality 24/7 Availability Lowering Social Barriers Low Cost (after development, that is)

7. Theoretical Issues in ITS Architecture Selection Representation of Domain Knowledge –Deep or Shallow? Student Model –Deep or Shallow? Interface –Constrained (Menus) or Natural Language?

8. Theoretical Issues in Choice of Pedagogical Style Didactic vs. Socratic Conceptual vs. Problem-Oriented Conversational vs. Pedagogical Collaborative vs. Competitive Authentic vs. Artificial Task Setting

9. Effects Sought on the User Comprehension (Clarity) Motivation Appropriately Challenging (ZPD)

10. Our Approach with RMT Architecture Pedagogy Implementation Evaluation Deployment

11. RMT: Architecture LSA for semantic representation –Word meanings are vectors in a semantic space –Sentences are “bags of words” –Cosine between vectors defines similarity of meaning Dialog Transition Network to guide dialog moves –Example of DTNExample of DTN Shallow knowledge of domain Shallow student model

12. RMT: Pedagogy Conversational Authentic Task –“You have just been hired as a research assistant. Your first task is to design a study..” Three Levels of Questions / Problems –Conceptual –Analytic –Synthetic

13. Conceptual Question Why are studies conducted outside of the lab more likely to have problems with internal validity? Model Answer: –Because the experimenter has more control over conditions in the lab. –Unanticipated confounds are more likely in a field setting.

14. Analytic Question Now I’ll give you a description of an experiment, and ask you to evaluate the validity of the experiment. (..description of experiment…) The concept of internal validity in research means that changes in the dependent variable are a direct result of changes in the independent variable, and not of some other variable. What are some ways that other things besides the lighting may have influenced the test scores?

15. Analytic Question (Cont.) Model Answer –Students took the exam at different times of the day. –One group is older than the other. –The two grades have different teachers.

16. Synthetic Question Now, I am going to describe an experiment, and you are going to help me set it up. An experimental psychologist believes that plant owners who interact with their plants will be happier than plant owners who do not. She identifies individuals who have never owned a plant and tests their happiness level using the Happiness Inventory. She then divides the non-plant owners into two groups: treatment and control. Describe how she should assign the participants to the two groups, what she should do to each group, and what each group should be told to do.

17. Synthetic Question (Cont.) Model Answer –Participants should be randomly assigned to two groups. –The treatment group should receive a plant. –The treatment group should be instructed to interact with the plant. –Control group should receive a plant but not be instructed to interact with it. –Both groups should be asked to return after a period of time to be re-tested on the Happiness Inventory.

18. RMT: Implementation Interface –Tutor is animated avatar –Text-only version for comparison –Student types responses in a text box Curriculum Script

19. Topics –Validity, reliability, ethics, types of research, experimental design XML schema defines structure of script –Questions –Model Answers –Prompts, hints, and splices –Dependencies

20. RMT: Evaluation Usability and Pilot Testing Experimental Evaluation of Tutor Features –RMT with avatar vs. text-only interface –RMT with authentic task vs. traditional tutoring –RMT vs. web-based tutorials

21. Pilot Testing: Agents One of our avatars’ voice needs work: –“The animation was fine, but the woman’s voice was annoying.” (“Miyako” avatar) The “Merlin” avatar was well-liked: –“I liked his facial expressions.” –“He was funny looking. Relieved stress in a way.” –“The animation was cool. The speech was monotonous.”

22. Pilot Testing: Tutor “It was easy because he asked a question and in case you got it wrong, he will help you to get to the right answer.” “It didn’t just give you the right answer if you got it wrong.” “It was creative and forces you to be clear with your ideas.”

23. Pilot Testing: Summary (N=26) Things they particularly liked: –The feedback –The hints and prompts –Multiple chances to get the right answer –Various ways of asking questions –Being led to the right answer –The Merlin animation Suggestions for improvement: –Mostly technical problems (esp. speech, agents)

24. RMT: Deployment Usability Testing (now) Experimental implementation in selected course sections (soon) Available for all research methods courses at DePaul (eventually)

25. Advantages of an LSA-based System Lower development cost than expert system “Cheap” way of doing NLP Works well for lists, on an all-or-nothing basis

26. Disadvantages of an LSA-based System No domain knowledge – only as good as the curriculum script Natural language understanding is only approximate –No syntax: “IV causes DV” = “DV causes IV” –“Heavy” words can overpower rest of sentence Not good at figuring out how much two sets overlap (only good at all-or-nothing matches)

27. RMT at DePaul: http://reed.cs.depaul.edu/peterwh/RMT http://reed.cs.depaul.edu/peterwh/RMT