Explorative Thread-based Analysis of Patterns of Collaborative Interaction in Chat Nan Zhou Murat Cakir

Overview Motivation for thread based sequential analysis Information about the data and the coding scheme A simple mathematical model for performing thread based sequential analysis of chat 4 research questions Extra: A possible method based on our model to support identification of the type of interaction (Exploratory vs Expository)

Motivation Analysis of fine-grained patterns of interaction is important for understanding collaborative learning Content based analysis cannot reveal the sequential unfolding of interaction A sequential analysis of postings is necessary to study how actions of participants unfold through interaction A naïve sequential analysis based on observed ordering of postings could be misleading due to artificial turn orderings produced by the quasi-synchronous chat medium By considering threads we aim to address this issue. The idea is to take into account the more complex linking structures while conducting sequential analysis.

Data 6 coded transcripts In 3 cases the problem was announced in advance In the latter cases the problem was announced at the beginning of the session We will consider Powwow2a, 2b and 18 in this presentation We will focus on 4 dimensions: Conversation, Conversation Thread, Problem Solving, and Problem Solving Thread

The Coding Scheme Math collaboration dimensions scheme_version 15_ doc Math collaboration dimensions scheme_version 15_ doc

What is captured by the Conversation Thread? The conversation thread aims… To tie postings of a single member together which span multiple lines Setup, elaboration and extension codes are used for this purpose To tie two postings where the latter one conversationally replies to the first one. Each statement is posted by different users. E.g. a question-explain pair, an offer-agree pair Such pairs can be considered as an approximation of adjacency pairs in chat The shape of the graph induced by the c-thread is a tree

Threaded view (based on C-thread)

What is captured by the Problem Solving Thread? PB-Thread links a posting to an earlier one from a problem solving perspective. Some parts are inherited from the conversation thread: Explain, Elaboration, Critique, Agree, Disagree, Follow Restatements are linked to the original statement, a summary is linked to the statements that are summarized having multiple links are possible in these cases, which is not the case in the conversation thread The graph induced by the problem solving thread need not be a tree

The computational model I Input: Coded transcript 2 graph representations based on c- thread and pb-thread dimensions are generated Each node of a graph corresponds to an object capturing properties of each posting (e.g. its author, statement, codes etc.)

The computational model II Traversals are performed over graphs to identify frequently occurring sequential patterns. We consider two postings P i -P j as a sequential pattern if they are linked by the thread First, we analyzed dyads and triads to study local organization of interaction Dyads and triads are the most frequently occurring patterns in a group of 3-4 participants We also consider the unfolding of local structures in the whole discussion

The computational model III Triad patterns Dyad pattern A variable C i can be replaced by 1. the author name, 2. the conversation code, 3. the problem solving code, 4. a combination of conversation and problem solving codes.

Research Questions Research Question 1: What patterns of interaction are frequently observed in a synchronous, collaborative math problem solving environment? Research Question 2: How can patterns of interaction be used to identify: (a) each member’s level of participation; (b) the distribution of contributions among participants; and, (c) whether participants are organized into subgroups through the discussion? Research Question 3: What are the most frequent patterns related to the main activities of the math problem solving? How do these patterns sequentially relate to each other? Research Question 4: What are the (most frequent) minimal building blocks observed during “local” interaction? How are these local structures sequentially related together yielding larger interactional structures?

RQ1: Interactional Patterns a percentage matrix for dyads based on conversation codes

RQ1: Interactional Patterns a row-based percentage matrix for the conversation dyads

RQ1: frequent conversational patterns Source vs. Sink Request-Response: 16%, 7%, 9%, 9%, 10%, 8% Response-Response: 12%, 5%, 2%, 4%, 10%, 11% State-Response: 8%, 6%, 4%, 2%, 5%, 16% Setup-X :8%, 14%, 12%, 2%, 3%, 4% X-Extension: 14%, 15%, 9%, 7%, 9%, 6%

RQ1 cont’d Implication of Setup-X and X-Extension Pruning: combine the fragmented statements into a single node

RQ2: identify participation -- dyads

RQ2: identify participation -- triads Percentage of triads having the same author: 15% for group A vs. 42% for group B Elaboration Anti-symmetry patterns: MCP to REA 23% vs. REA to MCP 14% AVR-PIN: 17%, 18%; AVR-SUP: 13%, 13%

RQ2: identify participation -- triads (cont’d) Who initiated the triads? the percentage of triads initiated by each member: AVR PIN SUP OFF 41% 29% 20% 7%

RQ3: Problem Solving Patterns Problem solving activities are defined by clustering problem solving codes. Orientation, Tactic, Strategy: Signaled action: understanding the problem statement and/or proposing strategies to approach the problem Perform, Result: Signaled action: proposed strategies are being executed Restate, Summary: Signaled action: Help a member to catch up and/or producing a reformulation of the problem Check, Reflect: These are interposed among the 3 activities described above (not considered as an individual cluster)

RQ3: Problem Solving Patterns Problem Solving ActivityPowwow2aPowwow2b Orientation/Tactic/Strategy/Check/Reflect Perform/Result/Check/Reflect Restate/Summarize/Check/Reflect 28% 21% 4% 5% 50% 0%

RQ4 Maximal Patterns Previous slide showed the difference between the types of problem solving activities performed by each group Maximal structure (the sequential unfolding of problem solving activities) can be used to contrast both groups at a global scale

RQ4 Maximal Patterns Powwow2a 1.Orientation activity in which the group identified a relevant sub-problem to work on 2. The group executed the strategy to find the altitudes and areas of each triangle. 3. Reflective activity in which they tried to relate the solution of the sub- problem to the general problem. They realized they made a mistake in the formula 4. Session ended without a solution Powwow2b 1. Each member solved the problem individually, no orientation activity. 2. Each member revealed their solution steps (captured as perform and result codes). 3. When they started working on the extra credit portion a short orientation activity was performed 4. They individually executed their strategies and reported their steps. 5. Session ended with a correct solution (both to the original and to the extra credit portion), which was not co-constructed

Extra: Using thread-analysis for studying the type of interaction. Alan and Fatos presented two different types of interaction in powwows (expository versus exploratory interaction) based on their CA analysis of powwow2a and powwow18 We will try to present additional evidence supporting this observation For this purpose we will contrast powwow2a and powwow18 in terms of their sequential organizations

Observations on powwow2a As concluded by Alan & Fatos before, interaction is organized in an ‘explorative’ way Suggesting tactics, co-constructing ideas for approaching the problem Interesting pattern: High percentage of Strategy, Orientation, Tactic codes

Observations on powwow18 There are 2 episodes in which one member shares his/her solution with other members. Solution steps are revealed step-by-step Other members’ reaction to each step with Agree, Follow statements suggesting the presenter to continue with the next step Interesting pattern: ‘Offer’s followed by ‘Follow’ and ‘Agree’s, high percentage of execution actions (i.e. perform, result) As concluded by Alan and Fatos, interaction is organized in an ‘expository’ way

Contrasting problem solving activities Problem Solving ActivityPowwow2aPowwow18 Orientation/Tactic/Strategy/Check/Reflect Perform/Result/Check/Reflect Restate/Summarize/Check/Reflect 28% 21% 4% 16% 43% 0% Less exploration work is done in powwow18, high percentage of execution steps

Contrasting local organization of conversation O F F El F F F Offer – Follow – Follow Elaboration – Follow – Follow These are called triads

Triad Pattern: (total 276)FreqPerc Offer - Follow - Response3 Offer - Follow - Follow13 Offer - Follow - Elaboration6 Offer - Follow - Extension3 Offer - Follow - Agree1 Offer - Follow - Critique1 Offer - Elaboration - Follow8 Offer - Elaboration - Agree1 Offer - Elaboration - Disagree1 Offer - Extension - Follow2 Offer - Extension - Agree1 Offer - Extension - Disagree0 Offer - Agree - Response2 Offer - Agree - Follow1 Offer - Agree - Elaboration1 Offer - Agree - Extension1 Offer - Disagree - Agree1 Elaboration - Follow - Follow3 Elaboration - Critique - Follow0 49%18 Triad Pattern: (total 350)FreqPerc Offer - Follow - Response1 Offer - Follow - Follow1 Offer - Follow - Elaboration2 Offer - Follow - Extension2 Offer - Follow - Agree3 Offer - Follow - Critique1 Offer - Elaboration - Follow2 Offer - Elaboration - Agree3 Offer - Elaboration - Disagree0 Offer - Extension - Follow2 Offer - Extension - Agree2 Offer - Extension - Disagree1 Offer - Agree - Response0 Offer - Agree - Follow1 Offer - Agree - Elaboration2 Offer - Agree - Extension1 Offer - Disagree - Agree1 Elaboration - Follow - Follow0 Elaboration - Critique - Follow1 26%7 Powwow18Powwow2a Triads that fit to the pattern we observed in powwow18

Author1Author2Author3Freq KILKOHAME3 KILAMEROB1 KILAMEKOH5 ROBKILKOH1 ROBKILAME2 ROBKOHKIL1 ROBKOHAME2 ROBAMEKIL1 ROBAMEKOH4 KILROB5 KOHKILAME11 KOHROBKIL3 KOHROBAME6 KOHAMEKIL10 KOHAMEROB6 AMEKILROB3 AMEKILKOH6 AMEROBKOH3 AMEKOHKIL4 AMEKOHROB2 AME MFP1 tot=27680%29 Author1Author2Author3Freq PINSUPAVR1 PINAVROFF1 PINAVRSUP3 OFFPINSUP1 OFFPINAVR1 OFFSUPAVR1 SUPPINAVR5 SUPOFFAVR2 SUPAVRPIN3 AVRPINSUP1 AVROFFSUP1 AVRSUPPIN6 AVRSUPOFF1 tot=35027%7 Group 1 Group 2 Triads linking 3 distinct authors, which is another property of the patterns we observed in powwow18

Contrasting powwow2a vs powwow18: Summary More effort is devoted to explorative activities in powwow2a as opposed to powwow18 More effort is devoted to exposing solution steps in powwow18, which is an indication of expository interaction. Powwow18 has a higher percentage of triad patterns that fit to expository interaction This short analysis shows that thread analysis could be useful in studying the type of the interaction

Conclusion We presented an alternative way for doing sequential analysis of interaction in chat, based on threads. We presented methods for making assessments about Each participant’s level of participation The conversational structure of discussion Problem solving activities performed by the group This is an ongoing work: We will study more factors (e.g. expository vs exploratory) We will build a statistical model