2. Charisma Perception from Text and Speech Andrew Rosenberg NLP Group Meeting 11/03/05

3. NLP Group Meeting2 Overview Background Speech Study Transcript Study Conclusion & Future Work

4. 11/03/05NLP Group Meeting3 Overview Background –What is charisma? –Does charismatic speech exist? –Why study charismatic speech? Speech Study Transcript Study Conclusion

5. 11/03/05NLP Group Meeting4 Background - What is charisma? (What do I mean by charisma?) Not “closed door”, face-to-face charisma. Rather, political (or religious) charisma –The ability to attract, and retain followers by virtue of personality as opposed to tradition or laws. (Weber ‘47) E.g. Ghandi, Hitler, Che Guevara. Charismatic speech: Speech that encourages listeners to perceive the speaker as “charismatic”.

6. 11/03/05NLP Group Meeting5 Background - Is there such a thing as charismatic speech? Pro: –Potential charismatic leaders must communicate with would-be followers. –Charismatic leaders have historically had a particular gift at public speaking Hitler, MLK Jr., Castro. Con: –Charisma as a relationship between leader and followers. –The mythologizing of a charismatic leader extends beyond public address.

7. 11/03/05NLP Group Meeting6 Background - Why study charismatic speech? General scientific interest. Feedback system for politicians and academic instructors. Identification of potential charismatic leaders Automatic generation of “charismatic- like” speech

8. 11/03/05NLP Group Meeting7 Overview Background Speech Study –Questions Addressed –Experiment Design –Analyses of Responses Transcript Study Conclusion

9. 11/03/05NLP Group Meeting8 Speech Study - Questions Do subjects agree about what is charismatic? What do subjects mean by charismatic? What makes speech charismatic?

10. 11/03/05NLP Group Meeting9 Speech Study - Experiment Design Subjects: Friends and colleagues, not compensated monetarily Interface: Presentation of 45 short speech segments (2-30secs) via a web form Dependent variables: 5-point Likert scale ratings of agreement on 26 statements. Duration: avg. 1.5 hrs, min 45m, max ~3hrs

11. 11/03/05NLP Group Meeting10 Speech Study - Experiment Design Interface –http://www1.cs.columbia.edu/~amaxwell/survey/

12. 11/03/05NLP Group Meeting11 Speech Study - Experiment Design Materials: 45 tokens of American political speech Speakers: 9 Candidates for Democratic Party’s nomination for President –Gen. Clark, Gov. Dean, Rep. Edwards, Rep. Gephardt, Sen. Kerry, Rep. Kucinich, Sen. Lieberman, Amb. Moseley Braun, Rev. Sharpton Topics: Postwar Iraq, Healthcare, Bush’s Tax plan, Reason for Running, Content-Neutral

13. 11/03/05NLP Group Meeting12 Speech Study - Analysis How much do subjects agree? –Using the weighted kappa statistic with quadratic weighting, mean kappa was 0.213 across all subject responses. Do subjects agree differently based on the stimuli? –No, there were no systematic differences across all tokens

14. 11/03/05NLP Group Meeting13 Do subject agree differently on the 26 statements? Most consistent statements Charisma: 0.224 (8 th ) Least consistent statements The speaker is accusatory0.512 The speaker is passionate0.458 The speaker is intense0.431 The speaker is angry0.404 The speaker is enthusiastic0.362 The speaker is trustworthy0.037 The speaker is reasonable0.070 The speaker is believable0.074 The speaker is desperate0.076 The speaker is ordinary0.115

15. 11/03/05NLP Group Meeting14 What do subjects mean by “charismatic”? Using kappa we determined which pairs of statements were most closely and consistently correlated with the charismatic statement. The speaker is enthusiastic0.606 The speaker is charming0.602 The speaker is persuasive0.561 The speaker is boring-0.513 The speaker is passionate0.512 The speaker is convincing0.503

16. 11/03/05NLP Group Meeting15 Are certain speakers more charismatic than others? Yes, there is a significant difference between speakers (p=1.75e-2) Most charismatic –Rep. Edwards (3.73) –Rev. Sharpton (3.40) –Gov. Dean (3.32) Least charismatic –Sen. Lieberman (2.38) –Rep. Kucinich (2.73) –Rep. Gephardt (2.77)

17. 11/03/05NLP Group Meeting16 Does the genre or topic of speech affect judgments of charisma? The tokens were taken from debates, interviews, stump speeches, and a campaign ad –Stump speeches were the most charismatic. (3.28) –Interviews the least. (2.90) Topic does not affect ratings of charisma.

18. 11/03/05NLP Group Meeting17 Does recognizing a speaker affect judgments of charisma? Subjects were asked to identify which, if any, speakers they recognized at the end of the study. Subjects rated recognized speakers (3.28) significantly more charismatic than those they did not (2.99).

19. 11/03/05NLP Group Meeting18 What makes speech charismatic? Acoustic/Prosodic and Lexical Properties Examined Duration (secs) Min, max, mean, stdev F0 –Raw and normalized by speaker Min, max, mean, stdev intensity Speaking rate (syls/sec) Length (words, syls) 1 st, 2 nd, 3 rd person pronoun density Function to content word ratio Mean syllables/word Number and ratio of disfluencies and repeated words Mean words per intermediate and intonational phrase Number of intonational, intermediate, and internal phrases Mean and stdev of normalized F0 and intensity across phrases Require manual labeling of phrase boundaries

20. 11/03/05NLP Group Meeting19 What makes speech charismatic? Properties showing positive correlation with charisma More Content –Length in secs, words, syllables, and phrases Higher and more dynamic raw F0 –Min, max, mean, std. dev. of F0 over male speakers Greater intensity –Mean intensity Higher in a speaker’s pitch range –Mean normalized F0 Faster speaking rate –Syllables per second

21. 11/03/05NLP Group Meeting20 What makes speech charismatic? Properties showing positive correlation with charisma Greater variation of F0 and intensity across phrases –Std. dev. of normalized phrase F0 and intensity The use of more first person pronouns –First person pronoun density The use of polysyllabic words –Lexical complexity (mean syllables per word) Speaking fluidly –Number and ratio of disfluencies negatively correlate Repeat yourself –Number and ratio of repeated words

22. 11/03/05NLP Group Meeting21 Overview Background Previous Work Speech Study Transcript Study –Questions Addressed –Experiment Design –Analyses of Responses Comparisons to Speech results Conclusion

23. 11/03/05NLP Group Meeting22 Transcript Survey - Questions When reading a transcript of speech, do subjects rate charisma consistently? What do subjects mean by charisma? –Do they mean the same thing when referring to text and speech? How does what is said influence subject ratings of charisma, compared to how it is said?

24. 11/03/05NLP Group Meeting23 Transcript Survey - Experiment Design Subjects: 24 paid participants found –http://newyork.craigslist.orghttp://newyork.craigslist.org –“Talent gigs” section Interface: Presentation of 60 short transcripts (words…) via a web form Dependent variables: 5-point Likert scale ratings of agreement on 26 statements. Duration: avg. 1.5 hrs, min 45m, max ~3hrs

25. 11/03/05NLP Group Meeting24 Transcript Survey - Design Interface: –http://www1.cs.columbia.edu/~amaxwell/textsurvey/A/http://www1.cs.columbia.edu/~amaxwell/textsurvey/A/

26. 11/03/05NLP Group Meeting25 Transcript Study - Design Materials: 60 of 90 tokens of American political speech –The 90 transcripts were the 45 used in the speech study, and 45 longer paragraphs –Each subject was presented with all 45 short (mean ~28 words) and a semi-random set of 15 long transcripts (mean ~130 words) Speakers: Identical to Speech Study Topics: Identical to Speech Study

27. 11/03/05NLP Group Meeting26 Transcript Study - Design Examples: –Token 1: We’re driving seniors out of medicare into HMOs. Every provision that would’ve brought down the cost of prescription drugs, the drug companies were against em all. They all came out.

28. 11/03/05NLP Group Meeting27 Transcript Study - Design Examples: –Token 2. …and I’d like to begin by, saying that I hope that, this afternoon’s talk will be an opportunity to challenge some underlying assumptions that we have about the world cause that’s why I’m uh running for President.

29. 11/03/05NLP Group Meeting28 Transcript Study - Design Examples –Token 3: …stabilize iraq because we occupy it. Yet he will not talk about the deficits in the fifty states we already occupy.

30. 11/03/05NLP Group Meeting29 Transcript Study - Design Examples –Token 4: …by two thousand five and then let their parents on a sliding scale based on income buy into medicaid at a price much below what they’d have to pay in the market.

31. 11/03/05NLP Group Meeting30 Transcript Study - Design Some tokens are rated very similarly whether presented as speech or a transcript. –Example 1 always charismatic –Example 2 always uncharismatic Others are rated very differently –Example 3 more charismatic in speech –Example 4 in text

32. 11/03/05NLP Group Meeting31 Transcript Study - Analyses How much do subjects agree? –Using the weighted kappa statistic with quadratic weighting, mean kappa was 0.149 Do subjects agree differently based on different stimuli? –No significant differences across all tokens

33. 11/03/05NLP Group Meeting32 Do subject agree differently on the 26 statements? Most consistent statements Charisma: 0.134 (18 th ) Least consistent statements The speaker is accusatory0.280 The speaker is angry0.263 The speaker’s message is clear0.206 The speaker is friendly0.197 The speaker is knowledgeable0.193 The speaker is spontaneous 0.039 The speaker is ordinary0.048 The speaker is boring0.050 The speaker is desperate0.064 The speaker is enthusiastic0.093

34. 11/03/05NLP Group Meeting33 What do subjects mean by “charismatic”? Using the kappa statistic determined which pairs of statements were most closely correlated with the charismatic statement. The speaker is charming0.576 The speaker is enthusiastic0.511 The speaker is persuasive0.503 The speaker is powerful0.485 The speaker is convincing0.483 The speaker is passionate0.446

35. 11/03/05NLP Group Meeting34 What do subjects mean by “charismatic”? Those statements that cooccur with the charismatic are in the speech and transcript study overlap greatly From this we conjecture that subjects employ a consistent functional definition of “charismatic” –Namely “charming, enthusiastic, persuasive, convincing and passionate”

36. 11/03/05NLP Group Meeting35 Does the identity of the speaker affect judgments of charisma? There is a significant difference between speakers (p=1.67e-10) Most Charismatic: –Gen. Clark (3.61) –Sen. Kerry (3.56) –Gov. Dean (3.54) Least Charismatic: –Sen. Lieberman (3.03) –Rep. Kucinich (3.12) –Amb. Mosley-Braun (3.23)

37. 11/03/05NLP Group Meeting36 Does the genre of a transcript affect judgments of charisma? Genre demonstrates a significant influence on charisma (p=9.18e-14) Stump (3.34) and debate (3.32) above mean (3.15) Interview below mean (2.85)

38. 11/03/05NLP Group Meeting37 Does the topic of a transcript affect judgments of charisma? Topic was significantly influenced ratings of charisma (p=1.5e-10) –In speech study, topic had no impact. Most charismatic topics: –Content-Neutral (3.64), Reason for running (3.53) mean:3.36 Least charismatic: –Taxes (3.12), Iraq (3.22), Healthcare (3.28)

39. 11/03/05NLP Group Meeting38 What makes a transcript charismatic? More Content –Length in words, or syllables Use of more function words –Density of function words Use of fewer first person pronouns –First person pronoun density is negatively correlated Speak fluidly –Number and ratio of disfluencies Repeat yourself –Number and ratio of repetitions Lexical complexity (syls/wd) doesn’t matter

40. 11/03/05NLP Group Meeting39 Overview Background Previous Work Speech Study Transcript Study Conclusion –Future Work

41. 11/03/05NLP Group Meeting40 Conclusion Despite not agreeing about what is “charismatic”, subjects employ a common definition of “charisma”. –“Enthusiasm, passion, charm, persuasion and being convincing” are consistently used to describe someone is “charismatic”. In general, what is said is a dominant force in whether speech is perceived as “charismatic” or not, with how it is said modifying this. Acoustic properties broadly reflect enthusiasm and passion

42. 11/03/05NLP Group Meeting41 Conclusion - Future Work Resynthesis Experiments –By modifying prosody of tokens can we make Lieberman charismatic? Sharpton uncharismatic? Investigating Palestinian Arabic –What are the similarities and differences between American and Palestinian notions of charisma? –What lexical and acoustic/prosodic properties are displayed by charismatic Palestinian speech?