Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron

Overview Background of AI in Education and Intelligent Tutoring Systems On human learning AutoTutor RMT Architecture Techniques Deployment Evaluation

Overview of AIEd The goal: Interact with student to increase learning The ideal system: Models domain knowledge Models student knowledge Adapts to student Adapts its approaches to optimize pedagogy

Some influential systems Algebra tutor Project Listen Scenario-based interaction (e.g. Wolf) Inquiry-based learning

What affects learning? Motivation Being “constructive” Doing example problems On the other hand: the Hawthorne effect

Bloom's hierarchy (1956) Knowledge: recall facts Comprehension: understanding info Application: ability to use info Analysis: see patterns, organization Synthesis: use old ideas to create new Evaluation: compare, discriminate, assess

Human tutors Much better than classroom practice: up to 2 standard deviations Rely on doing problems with their students Don’t actually have deep understanding of their students Use basic dialog moves: question, pump, prompt, hint, summarize Have a script of topics to cover

AutoTutor Started in 1997 Models (non-expert) human tutors Talking head “incarnates” the tutor Uses dialog moves above Evaluates student answers by comparing to expected answers

Latent Semantic Analysis Take a large corpus of texts Make a matrix of words in documents (paragraphs) Weight by inverse frequency Do singular value decomposition to “re- orient” the dimensions, reduce the noise Use vectors to represent “meaning” of words and texts. Use cosine to compare meanings.

RMT Follows the AutoTutor approach Re-implemented in Lisp Interacts via the internet Includes spell-checking and synonym substitution

Architecture Dialog manager handles logins, providing tutor utterances, collecting student responses. Responses evaluated with NLU components (LSA, aspell) Dialog Transition Network determines tutor’s next response

DATN Graphical representation of tutor’s behavior Directly controls the tutor’s choice of response.

Curriculum Script 5 topics: Ethics, types of studies, variables, reliability, validity Over 4000 lines Each item contains: Question (Picture) Target good answers Associated prompts, hints, questions Bad answers Summary

RM Curriculum Ethics of psychological experiments privacy, informed consent, ethics code, costs versus benefits Types of studies experiments and correlational (observational, archival, case studies, survey research) Variables independent and dependent, operationalization, levels, causality, confounding, categorical and quantitative variables, scales Reliability systematic and random error, test-retest, internal consistency, factors that affect reliability Validity types of validity (internal, construct, external, statistical), threats to validity.

Types of student tasks Conceptual: facts, definitions, issues Analytic: Given a scenario, evaluate weaknesses, or fill in missing slots Synthetic: Given a task, create an approach (frame and slots)

Example conceptual dialog Tutor: What does the principle of justice imply about how research should be done? Student: No group should have to take more risk than another group. Tutor: Good. What else? Student: The benefits of the research should be available to everyone

Analytical example Tutor: I'm going to tell you about a research project someone conducted that has some ethical problems, and then you will tell me what the researcher should have done differently. OK? Here is how the project was done. Joe, a graduate student, did a study of high school students' attitudes about racism. He created a survey about racism and gave it out to 100 high school students. Before he gave out the survey, he got permission from his dissertation advisor and from the principal of the high school to give out an anonymous survey to the students. He also made sure each student read and signed a consent form that was attached to the survey. What should Joe have done differently to make the project comply with the requirements for ethical practices in research? Student: Huh?

Synthetic example Now let's practice designing a study. I'll give you a hypothesis to test, and tell you what resources you have available. Then you design a study to test the hypothesis. Let's say that you want to find out whether reading a humorous greeting card causes people to be more likely to buy humorous magazines - that is your hypothesis. You have an agreement with a local magazine shop that allows you to use some of their customer purchase records for your research. … How will you set up the study?

Demo

Cool stuff on the way Shallow thinking traps Dialog Chains Research supervisor scam

Leading students astray Failure to “get it right” strengthens learning Shallow thinking traps (STTs) in content material provide opportunity for failure Dialog chains (DCs) should get students back on track A DC has a trigger and a network of student responses and tutor questions

Different tutoring style Traditional: Tutor (usually) asks the questions and the student (hopefully) answers. Pseudo-real world: The tutor is disguised as a research supervisor Asks questions to assess student knowledge (conceptual, analytic) Assigns research tasks (analytic, synthetic) Evaluates student’s work performance

To the classroom Use in association with two core courses in Psychology next term Students use the system after they’ve covered the topic in the course

In-course evaluation Pilot testing on various aspects 2x2 design: Tutor or static information Traditional tutor or research supervisor Pretests and posttests

Pilot testing Questions: 1. How easy/difficult was it for you to use the tutor? VERY EASY 1——2——3——4——5——6 VERY DIFFICULT 2. Was the tutor’s speech easy/difficult to understand? 1. How easy/difficult was the material presented by the tutor? N. Did you feel like you may learn/may not learn something during a tutoring session?

Early results In general, subjects disliked the computerized speech - -Every subject in the "Miyako" mentioned her voice as something they did not like, and something that could be improved. "It was a female tutor with a male voice. Made it annoying." "Try to use a less computerized voice." "The animation was fine, but the woman's voice was annoying.“

More results In general, subjects liked the animation - the only problem seemed to be that Miyako appears on top of the response box. Ss have to keep moving her back to the center of the screen. Merlin seemed to be the favorite. "I liked his facial expressions." "He was funny looking. Relieved stress in a way." "Liked his outfit."

Even more In general, subjects liked the feedback given and the hints and prompts and felt that the level of the material was appropriate "It was easy because he asked a question and in case you got it wrong, he will help you to get to the right answer. "The feedback was helpful." "It didn't just give you the right answer if you got it wrong."

Some suggestions -Keep questions on the screen in the dialog box in case the student misses what the tutor asks. -Improve voices on the animated characters. -Move Miyako so that she appears in the center of the screen. -Provide instructions at the beginning; a longer intro. -More explanation of right answer. -Sometimes the tutor asks the question before giving the details of the experiment.

Things they liked: -The feedback -The hints and prompts -Multiple chances to get the right answer -Various ways of asking questions -Lead me to the right answer -Merlin animation

Issues For the research supervisor, does text- only make more sense? If not, what head do we give it? Is the supervisor scenario realistic enough to motivate students? Will it work?