1. The Impact of Conversational Navigational Guides on the Learning, Use, and Perceptions of Users of a Web Site Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.

2. Outline (The next 20 minutes) HURAA Experiment Results Conclusions

3. What is HURAA? HURAA (Human Use Regulatory Affairs Advisor ) is a web-based facility that provides help and training on the ethical use of human subjects in research. HURAA is based on documents and regulations from United States Federal agencies.

4. HURAA (Features) Hypertext Multimedia Animated navigational guide Lessons with case-based and explanation-based reasoning Help modules Context-sensitive FAQs (Point & Query) Glossaries Archives Natural language queries

5. HURAA (Modules) Introduction Historical Overview Lessons Explore Issues Explore Cases Decision Consequences Query IRB Documents

7. Outline HURAA Experiment Results Conclusions

8. Experiment 155 students Between subjects: –Navigational Guides Full Voice Text None Within subjects: –Two Phases Acquisition phase Test phase

9. Experiment (Guides) Full guide (agent and voice)

11. Experiment (Guides) Full guide (agent and voice) Voice guide (synthesized speech only)

13. Experiment (Guides) Full guide (agent and voice) Voice guide (synthesized speech only) Print guide (text message only)

15. Experiment (Guides) Full guide (agent and voice) Voice guide (synthesized speech only) Print guide (text message only) No guide (no navigational guidance)

17. Experiment (Predictions) If guidance is important: –Full, Print, Voice > None Given guidance: –If speech medium is most effective Full, Voice > Print –If text medium is most effective Print > Full, Voice –If the agent/persona effect is most effective Full > Voice –If the agent/persona effect is distracting Voice > Full

18. Experiment (Acquisition phase) Introduction - Flash intro used to hook the user and acquaint them with the system. Lessons – 4 example cases used to teach/test the seven critical issues.

19. Lessons 1

20. Lessons 2

21. Lessons 3

22. Experiment (Acquisition phase) Introduction – Flash intro used to hook the user and acquaint them with the system. Lessons – 4 example cases used to teach/test the seven critical issues. Search Task – Users answer 4 specific questions, designed to use different modules in the system (specifically the query documents module).

23. Search IRB Documents

24. Experiment (Test phase) Memory – Free and cued recall tested core ideas from the Introduction and Lesson material. A Cloze procedure was used to test memory for key words. Issue Comprehension – As a transfer test from the Lesson module, users read two sample cases and provided respective ratings of problematic issues. Perception Ratings – Users provided ratings on the system (e.g. “You learned a lot about human subjects protections.”).

25. Outline HURAA Experiment Results Conclusions

26. Results (Core ideas) Table 1. Proportion correct

27. Results (Problematic issues) Table 2. Proportion correct

28. Results (Search task) Table 3. Proportion correct Table 4. Completion Time (min)

29. Results (Time on task) Table 5. Mean minutes on task

30. Results (Perception ratings) Table 6. Mean ratings (1 to 6, higher is better)

31. Outline HURAA Experiment Results Conclusions

32. No significant results from any of the dependent measures. This null result is incompatible with any of our previously stated predictions. A practical implication of the result is that the animated conversational agent did not facilitate learning, usage, or perceptions of the interface.

33. Conclusions Perhaps, due to the well structured nature of the site, a navigational guide was superfluous. An agent could possibly provide help in a more complex environment. The value of a navigational guide may increase as a function of the complexity, ambiguity, and perplexity of the system.

34. Conclusions Animated conversational agents have proven to be effective when they deliver information and learning material in monologues and tutorial dialogues. Perhaps there are special conditions when a navigational guide of some form will be helpful, whether it be print, voice, or a talking head. However, these precise conditions have yet to be discovered and precisely specified in the literature.

35. The Impact of Conversational Navigational Guides on the Learning, Use, and Perceptions of Users of a Web Site Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.