1. Discovery, Analysis and Monitoring of Hidden Social Networks and their Evolution Malik Magdon-Ismail Rensselaer Polytechnic Institute

2. 2 Our Group  M. Goldberg  M-I  B. Szymanski  A. Wallace Students:  Mykola Hayvanovich  Apirak Hoonlor  Stephen Kelley  Konstantin Mertsalov

3. 3 Motivation Communications supporting IED planning have patterns and are correlated…. Analysis of the patterns can reveal the groups as well as their internal group structure.

4. 4 Communications Time: January 12, 2005, 09:35 From: joe@xyz.comjoe@xyz.com To: sue@abc.comsue@abc.com Subject: Hello Message: Where have you been? 16:06:31] Republicans were the worst pacifists before ww1 and ww2 [16:06:43] France Fries [16:06:50] As a generality, of course their were Republican Hawks. [16:07:13] Sweet, good pun but bad story! [16:07:18] yup [16:07:23] anyways, he's perpetually tormented by presidential actions [16:07:25] it aint good for no one [16:07:47] I think they knew it was commiing [16:07:51] Rossevelt met monthly in New York with mostly trusted Republicans to talk about how to get america into the war. [16:08:10] and he spent 2 year with Churchill meeting him sometimes secretly in the ocean to discuss the same topic. [16:08:22] Exchanging a lot of letters. [16:08:25] telegrams [16:08:28] There really is nothing like a shorn scrotum. It's breathtaking, I suggest you try it. [16:08:55] Well they didnt literally meet in the ocean, they were on ships.

5. 5 Streaming Example Time From To Message 10:00 Alice Charlie Golf tomorrow? Tell everyone. 10:05 Charlie Felix Alice mentioned golf tomorrow. 10:06 Alice Bob Hey, golf tomorrow. Spread the word. 10:12 Alice Bob Tee off: 8am at Pinehurst. 10:13 Felix Grace Hey guys, golf tomorrow. 10:13 Felix Harry Hey guys, golf tomorrow. 10:15 Alice Charlie Pinehurst Tee time: 8am. 10:20 Bob Elizabeth We’re playing golf tomorrow. 10:20 Bob Dave We’re playing golf tomorrow. 10:22 Charlie Felix Tee time 8am at Pinehurst 10:25 Bob Elizabeth We tee off 8am at Pinehurst. 10:25 Bob Dave We tee off 8am at Pinehurst. 10:31 Felix Grace Tee time 8am, Pinehurst. 10:31 Felix Harry Tee time 8am, Pinehurst.

6. 6 Streaming Example Time From To Message 10:00 Alice Charlie Golf tomorrow? Tell everyone. 10:05 Charlie Felix Alice mentioned golf tomorrow. 10:06 Alice Bob Hey, golf tomorrow. Spread the word. 10:12 Alice Bob Tee off: 8am at Pinehurst. 10:13 Felix Grace Hey guys, golf tomorrow. 10:13 Felix Harry Hey guys, golf tomorrow. 10:15 Alice Charlie Pinehurst Tee time: 8am. 10:20 Bob Elizabeth We’re playing golf tomorrow. 10:20 Bob Dave We’re playing golf tomorrow. 10:22 Charlie Felix Tee time 8am at Pinehurst 10:25 Bob Elizabeth We tee off 8am at Pinehurst. 10:25 Bob Dave We tee off 8am at Pinehurst. 10:31 Felix Grace Tee time 8am, Pinehurst. 10:31 Felix Harry Tee time 8am, Pinehurst.

7. 7 Streaming Example Time From To 10:00 Alice Charlie 10:05 Charlie Felix 10:06 Alice Bob 10:12 Alice Bob 10:13 Felix Grace 10:13 Felix Harry 10:15 Alice Charlie 10:20 Bob Elizabeth 10:20 Bob Dave 10:22 Charlie Felix 10:25 Bob Elizabeth 10:25 Bob Dave 10:31 Felix Grace 10:31 Felix Harry

8. 8 Overview: SIGHTS & RDM buy,trade...buy 2trade...2trade 3,sell...3,hell Pattern id = 2 Pattern = “buy,” Pattern id = 3 Pattern = “2trade”bb Level 0 Level 1 Level 2 Higher ranked leaders Group leader Subgroup leaders Members

9. 9 Communications  Email, Telephone, Newsgroup, Weblog, Chatrooms, … Time: January 12, 2005, 09:35 From: joe@xyz.com To: sue@abc.com Subject: Hello Message: Where have you been lately? Time: January 12, 2005, 09:35 From: joe@xyz.com To: sue@abc.com Subject: Hello Message: Where have you been lately?

10. 10 Communication Graph January 12, 2005, 09:35 sue@abc.com joe@xyz.com

11. 11 Communication Graph What are the social groups/coalitions?

12. 12 Social Groups are Clusters  Clusters may overlap.

13. 13 Social Groups are Clusters  Clusters may overlap.  A cluster is a locally defined object.

14. 14 Social Groups are Clusters  Clusters may overlap.  A cluster is a locally defined object.  Group members are more introverted than extroverted. YESNO

15. 15 Social Groups are Clusters  Clusters may overlap.  A cluster is a locally defined object.  Group members are more introverted than extroverted.  Social groups (clusters) persist

16. 16 SIGHTS Statistical Identification of Groups Hidden in Time and Space - System for statistical analysis of social coalitions in communication networks Data Sources Blogs Emails (Enron) Chatroom Synthetic data Coalition Discovery Overlapping Clustering Streaming groups Persistent groups. Coalition Analysis Leaders Opposing groups Topic matching Visualizations Size-Density plots Static coalitions Dynamic coalitions Groups matching analyst topic in red Size vs. Density Plot Visualization options Choose time window Group members Different analyses on dataset Leader index

17. 17 Examples Two clusters: Electric circuit design; Optimization of Neural Networks: Intersection: “Sensitivity analysis in degenerate quadratic programming” Citeseer ENRON GROUND TRUTH  Group A  Dog  Vulture  Camel  Yassir Hussein  Bird  (6 others)  Group B  Ahmet  Saleh Sarwuk  Shaid  Pavlammed Pavlah  Osan Domenik SIGHTS  Group A  Dog  Vulture  Camel  Gopher  Group B  Ahmet  Saleh Sarwuk  Shaid  Dajik Ali Baba Data Set (DoD)

18. 18 Build a classifier to identify the relationship between sender and receiver of a message EXAMPLE: “ Do you have time to meet some time this week? ” “ Lets meet 2pm today, ok? ” Which is advisor, which is student? Recursive Data Mining (RDM)

19. 19 Hierarchical Pattern Construction (recursive definition) Captures patterns; patterns of patterns; patterns of patterns of patterns… (can even capture long-range patterns) Pattern Definition Larger patterns buy,trade...buy 2trade...2trade 3,sell...3,hell Pattern id = 2 Pattern = “buy,” Pattern id = 3 Pattern = “2trade” Pattern id = 4 Pattern = “3,_ell” Level 0 Level 1 Level 2

20. 20  Ensemble of classifiers  Classifier for each level in the hierarchical approach  Features gathered from the training messages  Global features include average length and number of sentences  Approximate matching allows treatment of noise A Classifier – Joining the Pieces

21. 21 Binary classification: for a given message m, is m sent by a person with role r? r є {CEO, Manager, Trader, Vice-President} Multi-classification: for a given message m, which role r is the most likely for the sender? r є {CEO, Manager, Trader, Vice-President} The bars show the error of classification. Universally RDM_SVM outperforms other classifiers Results on Enron

22. 22 Summing Up  SIGHTS:  Structural; non-semantic; language independent  Finds groups, their dynamics and structure; visual analytic capabilities.  RDM  Uses statistical semantics; language independent  Identifies roles within the group

23. 23 Thank You http://www.cs.rpi.edu/~magdon