InfoMagnets : Making Sense of Corpus Data Jaime Arguello Language Technologies Institute

Outline InfoMagnets Applications Topic Segmentation Conclusions Q/A

Outline InfoMagnets Applications Topic Segmentation Conclusions Q/A

Defining Exploratory Corpus Analysis Getting a “sense” of your data How does it relate to: –Information retrieval Need to understand the whole corpus –Data mining Need rich interface to support serendipitous search –Text classification Need to find the “interesting” classes

InfoMagnets

InfoMagnets Applications Behavioral Research –2 Publishable results (submitted to CHI) CycleTalk Project, LTI –New findings on mechanisms at work in guided exploratory learning Robert Kraut’s Netscan Group, HCII Conversational Interfaces Corpus organization makes authoring conversational agents less intimidating. Rose, Pai, & Arguello (2005); Gweon et al., (2005)

InfoMagnets Applications Behavioral Research –2 Publishable results (submitted to CHI) CycleTalk Project, LTI –New findings on mechanisms at work in guided exploratory learning Robert Kraut’s Netscan Group, HCII Conversational Interfaces Corpus organization makes authoring conversational agents less intimidating. Rose, Pai, & Arguello (2005); Gweon et al., (2005)

Authoring Conversational Interfaces Goal: Make Authoring CI’s easier Solution: –Guide development with pre-processed sample human-human conversations Addresses different issues –Accessible to non-computational linguists –Developers ≠ domain experts –Consistent with user-centered design: “The user is not like me!”

Authoring Conversational Interfaces Topic Segmentation Transcribed human-human conversations A C B CA B Constructing a Master Template

Topic Segmentation Preprocess for InfoMagnets But, an important computational linguistics problem in its own right! Previous Work –Marti Hearst’s TextTiling (1994) –Beeferman, Berger, and Lafferty (1997) –Barzilay and Lee (2004) NAACL best paper award! – ….. But, should it all fall under “topic segmentation”?

Topic Segmentation of Dialogue Dialogue is Different: –Very little training data –Linguistic Phenomena Ellipsis Telegraphic Content –Coherence is organized around a shared task, not primarily around a single flow of information

Lots of places where there is no overlap in “meaningful” content Coherence Defined Over Shared Task

Multiple topic shifts in regions w/ zero lexical cohesion

Experimental Condition 22 student-tutor pairs Conversation captured through mainstream chat client Thermodynamics domain Training and test data coded by one coder Results shown in terms of p_k (Lafferty & Beeferman, 1999) Significant tests: 2-tailed, t-tests

1 st Attempt: TextTiling TextTiling (Hearst, 1997) –Slide two adjacent “windows” down the text –At each state calculate cosine correlation –Use correlation values to calculate “depth” –“Depth” values higher than a threshold correspond to topic shifts w1 w2

TextTiling Results Algorithm(avg) P k NONE ALL EVEN TextTiling TT T-test p-value TT (NONE) TT (ALL) TT (EVEN) Trend for TextTiling to perform worse than degenerate baselines Difference not statistically significant Why doesn’t it work?

Lots of gaps where the correlation = 0 Must select boundary heuristically And, still a heuristical improvement on original TextTiling Results

But, topic shifts tend NOT to occur where corr > 0. TextTiling Results

Cluster utterances Treat each cluster as a “state” Construct HMM –Emission probabilities: state-specific language models –Transition probabilities: based on location and cluster-membership of the utterances Viterbi re-estimation until convergence 2 nd Attempt: Barzilay and Lee (2005)

B&L statistically better than TT, but not better than degenerate algorithms B&L Results Algorithm(avg) P k NONE ALL EVEN TextTiling B&L B&L T-test p-value B&L (NONE) B&L (ALL) B&L (EVEN) B&L (TextTiling)

B&L Results Too fine grained topic boundaries Most clusters based on “fixed expressions” (e.g. “ok”, “yeah”, “sure” ) Remember: cohesion based on shared task Are state-based language models sufficiently different?

Incorporating Dialogue Dynamics Dialogue Act coding scheme Not originally developed for segmentation, but for discourse analysis of human-tutor dialogues 4 main dimensions: –Action: open question, closed question, negation, etc. –Depth: (yes/no) is utterance accompanied with explanation or elaboration –Focus: (binary) is focus on speaker or other agent –Control: Initiation, Response, Feedback Dialogue Exchange (Sinclair and Coulthart, 1975)

(Donmez, 2004) Use estimated labels on some dimensions to learn other dimensions 3 types of Features: –Text (discourse cues) –Lexical coherence (binary) –Dialogue Acts labels 10-fold cross-validation Topic Boundaries learned on estimated labels, not hand coded ones! 3 rd Attempt: Cross-Dimensional Learning

X-Dimensional Learning Results X-DIM statistically better than TT and degenerate algorithms! Algorithm(avg) P k NONE ALL EVEN TextTiling B&L X-DIM X-DIM T-test p-value X-DIM (NONE) X-DIM (ALL) X-DIM (EVEN) X-DIM (TextTiling)

Statistically Significant Improvement TTB&LX-DIM NONE NON-SIG SIG ALL NON-SIG SIG EVEN NON-SIG SIG TT SIG B&L SIG

Future Directions Merge cross-dimensional learning (w/ dialogue act features) with B&L content modeling HMM approach. Explore other work in topic segmentation of dialogue

Recap InfoMagnets and applications Corpus exploration and authoring of CI’s Challenges of topic segmentation of dialogue Description of TextTiling, Barzilay & Lee, X-DIM vs. degenerate methods and each other

Q/A Thank you!