Collective Intelligence Week 1 Old Dominion University Department of Computer Science CS 795/895 Spring 2009 Michael L. Nelson <mln@cs.odu.edu> 1/14/09

Goals We will: learn to program in Python survey mathematical techniques for mining the web and be able to use them in actual applications cover each chapter in the text book note that each chapter is a survey; we could spend a semester or more on every topic…

Administrivia This is a programming class! You should be able to teach yourself a new programming language from the book examples You will work in teams of one or two people mixes (g/u, g/g, u/u) ok assignments are the same regardless of group size

Administrivia 2 Pick teams wisely teams will exist by mutual consent only at any time, teams can split up, but no new teams will be formed after the first assignment is due no team member swaps ex-team members will have access to their shared code base

Administrivia 3 Important URLs Class homepage: http://www.cs.odu.edu/~mln/teaching/cs895-s09/ http://list.odu.edu/listinfo/cs895-s09/ Class homepage: readings are posted demo days are posted

Grading 10 problems, 10 points each Days of in class demo are posted 1 problem each from chapters 2-11 points recvd for each problem at the discretion of the instructor each group must email to the class list which problems they are planning to cover instructor has the right to prohibit exercises deemed “too easy” and to approve exercises not included in book since len(groups) > len(problems), overlap is ok (but variety is encouraged) Days of in class demo are posted demo day 1: chapters 2,3,4 demo day 2: chapters 5,6,7 demo day 3: chapters 8,9,10,11

Development Environment Assignments must be demoed on the machine mln-web.cs.odu.edu same uid/passwd as departmental unix machines; I will have accounts set up shortly Assignments will be written in Python

Extra Credit 10 points available for each person (not group) for: asking a question on the email list answering a question on the email list sharing tips Only “significant” messages will be counted (instructor’s discretion) e.g., “how do I login?” will not count

Why do we care about mining the Web? People in aggregate can be very smart… sometimes referred to as the “wisdom of crowds” http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds …and sometimes not. http://en.wikiquote.org/wiki/List_of_misquotations http://www.youtube.com/watch?v=h6sj89xgnl4 pagerank image from: http://hamletbatista.com/wp-content/uploads/2007/10/pagerank.jpg

Chapter 2: Recommendations

Comparing Reviews of 2 Movies

Comparing 2 Reviewers

2 Highly Correlated Reviewers

Correlation A measure of a linear relationship between two independent variables from: http://en.wikipedia.org/wiki/Correlation

Correlation In the book, we use “Pearson’s Product Moment Correlation Coefficient” http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient Many, many, many other coefficients possible e.g, for rank data, we often use: Spearman’s Rho Kendall’s Tau http://en.wikipedia.org/wiki/Rank_correlation

Look at data first!!! Pearson’s only works for linear (i.e., normal distributions) non parametric correlation methods needed for non-normal distributions http://en.wikipedia.org/wiki/Correlation#Non-parametric_correlation_coefficients Example: Anscombe’s Quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet

Recommendations Based on Past Rankings Note not all reviewers have seen all movies!

“If you like X, you probably won’t like Y.”

Item Based Recommendations (i.e., movies instead of critics)

To Do for Next Time… Subscribe to the class email list Submit group info to class list I’ll email to the list when accounts on mln-web.cs.odu.edu are set up