PIER Research Methods Protocol Analysis Module Hua Ai Language Technologies Institute/ PSLC

Questions? "There was a significant negative correlation between the first and third metrics used to compute a Power score for the partner's conversational contributions and this question's numeric value, and a marginal negative correlation in the case of the second metric.“  What are we supposed to interpret/learn from this statement?

Rose at el. Automatic Analysis  By Machine Learning

What is machine learning? Machine learning is about automatically finding meaningful patterns in data Example for medical data: Rule predicts who is more likely to have problems with their teeth as they get older.

Why Machine Learning? We use machine-learning products every day  Weather forecast  Spelling checker  Automated voice response system  … It has been successfully applied to many research areas  Natural language processing  Market analysis  Bioinformatics  … Note: Search engines use machine learning to personalize search results and suggest related sites or queries.

How does machine learning work? The simplest rule learner will learn to predict whatever is the most frequent result class. This is called the majority Class. What will the rule be in this case? It will always predict yes. A slightly more sophisticated rule learner will find the feature that gives the most information about the result class. What do you think that would be in this case? Outlook: Sunny -> No Overcast -> Yes Rainy-> Yes : -> …

What is machine learning? Automatically or semi-automatically  Inducing concepts (i.e., rules) from data  Finding patterns in data  Explaining data  Making predictions Data Learning Algorithm Model New Data Prediction Classification Engine

What will be the prediction? Outlook: Sunny -> No Overcast -> Yes Rainy-> Yes Model New Data Yes

Terminology Concept: the rule you want to learn Instance: one data point from your training or testing data (row in table) Attribute: one of the features that an instance is composed of (column in table) * Compute the predicted value. bad

Task Assign labels to a collaborative learning corpus using the Weinberger and Fischer’s coding scheme  Text classification task

Two approach categories The Feature Based Approach  Basic feature  Thread Structure feature – depth  FSM features  LIWC features (Linguistic Inquiry Word Count) The Algorithm Based Approach  Cascaded Binary Classification  Confidence Restricted Cascaded Binary Classification  Supervised/Unsupervised

Methodology issues related to automatic corpus analysis Validity  Whether the automatic coding accomplished by the computer captures human analysts’ intention Reliability  How faithfully the automatic codes match those of human experts’ Efficiency  Are we saving time by using the automatic classifier?

Ai et al. Manual Analysis  By statistical tests Interaction between tutor’s social behaviors and tutor’s bias towards one student’s stance or the other Topic modeling: sub-dialog level instead of utterance level

Study design 3*3  Social: how, low, none  Goal Match: yes, no, neutral

The ccLDA model Green Collection Power Collection Topic 1 Topic 2 Topic 3 Topic 1Topic 2 Topic 3 A topic is a distribution of words. We computed a score for each utterance based on the words contained in the utterance. This score stands for to what extend this utterance is biased towards this topic.

Results Learning Gain  Student learned most in Low-social + Yes- matched Perception (Questionnaire & Topic detection)  Effect on Goal-match manipulation Conversational Data  More social turns in social conditions  More off-task turns in non-social condition  More Jokes on tutor in the high social condition

Important message Understand the data is important Design good features  Both in automatic and manual analysis