1. Text Mining for Assessment of Writing and Social Positioning
Carolyn Penstein Rosé
Language Technologies Institute
and Human-Computer Interaction Institute
Carnegie Mellon University

2. Outline What are Text Mining and Discourse Analytics?
How can Discourse Analytics inform impactful design?
Caveats
Deep Dive: Tansactivity and Team Formation
Resources

3. Historical View
Earliest approaches to automated essay scoring (Page, 1966)
Work towards greater validity (Shermis & Burstein, 2003)
Work towards triggering feedback from assessment (Wade-Stein & Kintch, 2004)
Competition across industry and academia (Shermis & Hammer, 2012)
*** Free off-the-shelf approach just as good as industry options
LightSIDE (Mayfield & Rosé, 2013)

4. Bridging insights from linguistics to computation, and vice versa
Specifically Sociolinguistics and Discourse Analysis
Which modeling assumptions are not consistent with the theory?
What faulty assumptions about language in models lead to errors?
In what ways do models challenge theory?
In what ways are theories brittle or overly simplistic?
Linguistics
Computation

5. LightSIDE Tool Bench

6. Nguyen, D. , Dogruöz, A. S. , Rosé, C. P. , de Jong, F. (2016)
Nguyen, D., Dogruöz, A. S., Rosé, C. P., de Jong, F. (2016). Computational Sociolinguistics: An Emerging Area for Language Technologies, Computational Linguistics, Vol. 42, No. 3: 537–593.

7. Methodology
Theory
Research Questions
Interpretation
Data
Patterns

8. What is a theory?

9. Why do we need theory to inform our computational models?
VanLehn, K., Graesser, A., Jackson, G. T., Jordan, P., Olney, A., Rosé, C. P., (2007). Natural Language Tutoring: A comparison of human tutors, computer tutors, and text. Cognitive Science 31(1), 3-52
Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning 3(3), pp
General indicators of interactivity
Turn length
Conversation length
Number of student questions
Student to tutor word ratio
Student initiative

10. Theoretical Framework
Basic concepts of power and social distance explain social processes operating in interactions
Social processes are reflected through patterns of language variation
If we understand this connection, we can model language more effectively
Models that embody these structures will be able to predict social processes from interaction data
Language
Technologies
Sociolinguistics
Psychology

11. Linguistic Agency
“The expressiveness of an individual appears to involve two radically different kinds of sign activity: the expression that he gives and the expression that he gives off.”
What he gives refers to the strategies employed in order to achieve social goals
What he gives off refers to aspects of language that reflect the internal state of the speaker that are beyond his control
Erving Goffman
Sociologist and Author of
Presentation of Self in Everyday Life
Originally published in 1959

12. How can Discourse Analytics inform impactful design?

13. Process and State Measures
Help exchange
(Gweon et al., 2007; Cui et al., 2008)
Collaborative Knowledge Construction
(Rosé et al., 2008; Ai et al., 2010; Mu et al., 2012; Gweon et al., 2013)
Openness and Authoritativeness
(Mayfield & Rosé, 2011; Howley et al., 2013)
Attitude
(Wen et al., 2014a)
Cognitive Engagement and Expressed Motivation
(Wen et al., 2014b)
Coordinated Activity
(Kumar et al., 2014; Yang et al., 2014a)
Social Connection
(Yang et al., 2014b)
Confusion
(Yang et al., in press)

14. Connecting Processes with Outcomes
Information Transfer
(Gweon et al., 2011)
Learning
(Howley et al., 2011)
Attrition
(Rosé et al., 2014; Wen et al., 2014a; Wen et al., 2014b; Yang et al., 2014a; Yang et al., 2014b; Yang et al., in press)

15. Impact of Intensive Discussion on Commitment in MOOCs

16. Impact of Introducing Discussion Activities to MOOCs
Now a required activity in the course!!

17. Positive Impact on Retention in MOOCs
Reduction in attrition of between 40% and 70% when students chat together
Tested across 5 MOOC deployments

18. Continue No Dropout With baseline probability No Yes Click? Partner
2 day
time period
Continue No
Dropout With
baseline probability No
Yes
Click?
Partner
Present?
Yes
Dropout With
baseline probability
Yes No
Dropout With
70% lower than baseline
probability
Yes
Yes
Done

19. Caveats!

20. Souflé Framework (Howley et al., 2013)
Authority
Transactive
Knowledge Integration
Engagement
Person
Person

21. Example: MathTalk Personalized Agent condition vs Control condition
30 6th graders
Randomly assigned to pairs, conditions
Procedure
Day 1
Day 2
Lab session
Pretest Quiz
Posttest Quiz, Questionnaire
Social Dialogue Agent Study (Kumar et al, 2007)

22. Example: MathTalk

23. Main Results: Advantage for Social Condition
[Kumar et al, 2007]
Significant increase in perception of amount of help given and received
Significant increase in amount of help given per problem (Gweon et al., 2007)
Students marginally more likely to complete a step on their own after receiving help (Cui et al., 2009)
Marginally higher learning gains
But why?

24. Measuring Authority K2 A2 K1 A1 Primary Secondary Knowledge Action
Elijah Mayfield and Carolyn Penstein Rosé Recognizing authority in dialogue with an integer linear programming constrained model. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (HLT '11), Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, K2
requesting knowledge, information, opinions, or facts
Primary Secondary
Source or Sink?
Knowledge Action
Type of Content? A2
Instructing, suggesting, or requesting non-verbal action K1
giving knowledge, information, opinions, or facts A1
Narrating or performing your own non-verbal action
Additionally…
ch (direct challenge to previous utterance)
o (all other moves, backchannels, etc.)
K1 + A2
K1 + K2 + A1 + A2
Authoritativeness:

25. Understanding the Effect of Social Climate on Positioning and Risk Taking
Coded chat logs for instances of aggressive behavior
Pushy behavior
Insults
Coded for Negotiation (especially K1 and K2)
Based on counts of K1 and K2, computed an authoritativeness score for each student per lab day
K1/[K1 + K2]
Computed a Shift score per student
Residual from linear regression predicting Day 2 authoritativeness from Day 1 authoritativeness
Binary Shift variable (within pair, which student shifted up to a more authoritative stance versus shifted down)

26. Aggressive Behavior
Significantly more aggressive behavior in Control condition
F(1,56) = 8.93, p < .005 **, effect size .63σ
Significantly more aggressive behavior on Day 2
F(1,56) = 15.61, p < **, effect size .87σ
Significant interaction with Condition
F(1,56) = 6.06, p < .05 **
Only significant increase in aggressive behavior on Day 2 in the Control condition
In each pair, identified student with higher amount of aggressive behavior on Day 2 as “the bully” for further analysis

27. Authoritativeness and Shift
Significant difference in Authoritativeness of Bullies and Non-Bullies in Control condition on Day 2
F(1,23) = 5.92, p < .05**
Visible Shift only in Control Condition
F(1,23) = 5.28, p < .05**, effect size .15σ
Bullies in Control condition shifted to more authoritative stance
Non-bullies in Control condition shifted to less authoritative stance
Authoritativeness
Shift

28. Learning No significant main effect of Aggressive behavior on learning
Bullied students in Control condition learned significantly less than Social Condition students
Recall that students respond differently to help in Control condition
Significant interaction between Shift and Condition on Learning: F(1,20) = 7.91, p = .01**
Opposite trend in Social Condition
Significant correlation between amount of shift and learning only within Control condition
Shifting down was associated with less learning
Bullying
Shift

29. Deep Dive: Transactivity and Team Formation

31. Building Reasoning Together
Transactivity

32. Souflé Framework (Howley et al., 2013)
Authority
Transactive
Knowledge Integration
Engagement
Person
Person

33. From Theory to Computation
Model connecting speech style accommodation and Transactivity
Findings from Sociolinguistics
DBM
Model
Reflecting
Evidence of
Consensus Building
Reflecting
Perspectives
And
Relationships
Findings from Developmental Psychology
Mahaveer Jain, John McDonough, Gahgene Gweon, Bhiksha Raj, and Carolyn Penstein Rosé An unsupervised dynamic Bayesian network approach to measuring speech style accommodation. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL '12). Association for Computational Linguistics, Stroudsburg, PA, USA,

34. Power, Relationships, and Transactivity
Piaget
Berkowitz
& Gibbs
Kruger &
Tomasello
Azmitia
& Montgomery
Socio-
Cognitive
Conflict and
Transactivity
Power,
Cognitive Conflict,
And Learning
Power Balance
And
Transactivity
Friendship,
Transactivity,
And Learning
1963
1983
1986
1993

35. From Theory to Computation
Model connecting speech style accommodation and Transactivity
Findings from Sociolinguistics
DBM
Model
Reflecting
Evidence of
Consensus Building
Reflecting
Perspectives
And
Relationships
Findings from Developmental Psychology
Mahaveer Jain, John McDonough, Gahgene Gweon, Bhiksha Raj, and Carolyn Penstein Rosé An unsupervised dynamic Bayesian network approach to measuring speech style accommodation. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL '12). Association for Computational Linguistics, Stroudsburg, PA, USA,

36. From Theory to Computation
# Other Oriented Transacts

37. Transactivity Findings Computational Work
Moderating effect on learning (Joshi & Rosé, 2007; Russell, 2005; Kruger & Tomasello, 1986; Teasley, 1995)
Moderating effect on knowledge sharing in working groups (Gweon et al., 2011)
Computational Work
Can be automatically detected in:
Threaded group discussions (Kappa .69) (Rosé et al., 2008)
Transcribed classroom discussions (Kappa .69) (Ai et al., 2010)
Speech from dyadic discussions (R = .37) (Gweon et al., 2012)
Predictable from a measure of speech style accommodation computed by an unsupervised Dynamic Bayesian Network (Jain et al., 2012)

38. Effective
Team Formation

39. Current Team-based MOOCs
Self-selection
based Team
Formation
Team Collaboration

40. Proposed Team Formation and Support Paradigm
Individual Work
Community
Deliberation
Team Formation with transactivity evidence
Transactivity-based Collaboration Suport

41. Transactivity Maximization Team Formation
Maximize how many times members in a team transactively interact with each other
overall benefits
Minimal Cost Network Flow Algorithm [Ahuja and Orlin, 1989]
network flow: Jigsaw constraint
minimal cost: maximized the transactivite communication that was observed among all the group members during community deliberation

42. Team Collaboration

43. Validation Study

44. Results 27 Random groups, 27 Transactivity Maximization groups
Teams in the Transactivity Maximization Condition demonstrate better team performance (F(1,52) = 9.26, p = 0.02).

45. Results
Teams in the Transactivity Maximization Condition contribute more transactive chat turns during team collaboration (F(2, 51) = 6.74, p = 0.03).

46. Deployment Study Team-based MOOC
Rise of the Superheroes and the Heroes of the Future

47. Week 3 Week 2 Week 1 Team formation, Small team collaboration
Post individual work,
Community deliberation
Team formation,
Small team
collaboration
Week 3
Week 2
Week 1

48. Success!!! Summary of Results
Correlational results are consistent with our previous experiments
Marginal effect of detected #transactive exchanges prior to collaboration on project completion status (r = 0.23, p = 0.09)
Significant effect of #transactive exchanges prior to collaboration on number of active students within groups (r = 0.32, p = 0.02)
Marginal effect of #transactive exchanges prior to collaboration on whether all the superheroes interacted in the story (r = 0.26, p = 0.06)

49. Resources

50. DANCE is a community of practice with many open source resources
We were invited to serve as a satellite of the edX opensource community
We have been invited to serve in an advisory role to the edX Teaching and Learning team
Thousands of visitors
Hundreds of return visitors each month

51. Conclusion Important partnership between theory and computation
For understanding and for motivating effective design
Intertwining of constructs within collaboration can be leveraged
Caveate! Intertwining can also obscure our ability to assess accurately
Join us: Open Source Resources
Let me know if you would like to collaborate