1. MIS 111: Computers and the Inter-networked Society Class 11: Data Mining July 25th, 2011

3. This Week  Data Mining (Today):  humans and machines generate knowledge from data  Decision Support Systems (Tuesday)  combining models and data in an attempt to solve semistructured and some unstructured problems with extensive user involvement  Expert Systems (Machine’s that make decisions)  Computer systems that attempt to mimic human experts by applying expertise in a specific domain.

4. Learning Objectives  List a few current events in information systems news  Recap of quiz 2 learning objectives  Use Google analytics to perform data mining and make business decisions  List 3 practical applications of data mining  Explain the difference between Descriptive and Predictive data mining  Compare and contrast classification, association rule, deviation detection data mining  List a tool that can help you perform data mining

5. Administrative Trivia  Quiz  Some people didn’t put their name on their quiz  If you have a zero, come talk with me  We’ll go over it together today  Assignment 3 due Wednesday morning before class

6. Quiz 2 Recap  http://eller.qualtrics.com/SE/?SID=SV_eaGLk1JGq9kczBy http://eller.qualtrics.com/SE/?SID=SV_eaGLk1JGq9kczBy

7. Data Mining

8. What exactly IS Data Mining? Roughly speaking, Data Mining is the process by which humans and machines generate knowledge from data. Data Warehouse Data Processing Data Analytics

9. What is Data Mining?  Many Definitions  Non-trivial extraction of implicit, previously unknown and potentially useful information from data  Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns

11. Knowledge Discovery in Databases  Data Mining is only a small part of the knowledge discovery process.  Which part of the process do you think is most critical?  Which part of the process do you think takes longest?

12. What is (not) Data Mining? l What is Data Mining? – Certain names are more prevalent in certain US locations (O’Brien, O’Rurke, O’Reilly… in Boston area) – Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,) l What is not Data Mining? – Look up phone number in phone directory – Query a Web search engine for information about “Amazon”

13.  Lots of data is being collected and warehoused  Web data, e-commerce  purchases at department/ grocery stores  Bank/Credit Card transactions  Computers have become cheaper and more powerful  Competitive Pressure is Strong  Provide better, customized services for an edge (e.g. in Customer Relationship Management) Why Mine Data? Commercial Viewpoint

14. Why Mine Data? Scientific Viewpoint  Data collected and stored at enormous speeds (GB/hour)  remote sensors on a satellite  telescopes scanning the skies  microarrays generating gene expression data  scientific simulations generating terabytes of data  Traditional techniques infeasible for raw data  Data mining may help scientists  in classifying and segmenting data  in Hypothesis Formation

15. Evolution of Data Analysis Evolutionary Step Data Collection (1960s) Data Access (1980s) Data Warehousing & Decision Support (1990s) Data Mining (2000s) Business Question "What was my total revenue in the last five years?" "What were unit sales in New England last March?" "What were unit sales in New England last March? Drill down to Boston." "What’s likely to happen to Boston unit sales next month? Why?" Enabling Technologies Computers, tapes, disks Relational databases (RDBMS) On-line analytic processing (OLAP), multidimensional databases, data warehouses Advanced algorithms, multiprocessor computers, massive databases information delivery

16. In what disciplines do people use data mining?

17. Google Analytics

18. AnimalLingo.com  Google Analytics  What in here is data mining:  Map Overlay  Time series analysis  New visits  Bounce rate  Time on site ……  What changes should I make to my Web site (this is getting into the role of decision support systems)?

19. E-commerce

20.  http://www.amazon.com/

21. Finance

22.  Basic: Finance.yahoo.com  Can be much, much more complex (I should have a finance PhD student come in)  IS data mining and finance are a great mix!  Some examples ahead (you don’t have to know these unless you want to)

23. Finance: Portfolio Management (FYI)

25. Interpretable Trading Rules (FYI)  Categorical rules predict a categorical attribute, such as increase/decrease, buy/sell.

26. Discovering Fraud (FYI)  http://www.picalo.org/ http://www.picalo.org/

27. Sports

28. Sports and Data Mining  Go for it on forth down!  http://www.advancednflstats.com/2009/09/4th-down-study- part-1.html

29. Moneyball: The Art of Winning an Unfair Game

30. The Main Message of Moneyball  By analyzing baseball statistics you could see through a lot of baseball nonsense.  For instance, when baseball managers talked about scoring runs, they tended to focus on team batting average, but if you ran the analysis you could see that the number of runs a team scored bore little relation to that team's batting average. It correlated much more exactly with a team's on-base and slugging percentages.

31. Other applications

32. Just some examples…  Linguistics  Economics  Farming Farming  Government  Defense / homeland ssecurity  Education  Production forecasting  Sales forecasting  Fast food  Just about ANY DISCIPLINE can benefit from data mining

33. Data Mining Tasks

34.  Prediction Methods  Use some variables to predict unknown or future values of other variables.  Description Methods  Find human-interpretable patterns that describe the data. From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996

35. Data Mining Tasks...  Classification [Predictive]  Clustering [Descriptive]  Association Rule Discovery [Descriptive]  Sequential Pattern Discovery [Descriptive]  Regression [Predictive]  Deviation Detection [Predictive]

36. Lots of tools to help:  Weka  R  SPSS  SAS  Google  Picalo  Google Correlate / Graphs

37. Classification

38. Classification: Definition  Given a collection of records (training set )  Each record contains a set of attributes, one of the attributes is the class.  Find a model for class attribute as a function of the values of other attributes.  Goal: previously unseen records should be assigned a class as accurately as possible.  A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.

39. Classification Example categorical continuous class Test Set Training Set Model Learn Classifier

40. FYI: LOTS of different Classification Algorithms  Neural network  Mimics the way that humans Learn with NEURONS  Decision trees  K-means clustering

41. Classification: The Iris Flower Data Set  Which factors help us determine which Iris type a flower will be?  Petal Length  Petal Width  Sepal Length  Sepal Width  We can make the machine “learn” which attributes = which iris types.

42. Personal Equity Plan  Weka Example

43. Classification: A Business Application  Direct Marketing  Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product.  Approach:  Use the data for a similar product introduced before.  We know which customers decided to buy and which decided otherwise. This {buy, don’t buy} decision forms the class attribute.  Collect various demographic, lifestyle, and company-interaction related information about all such customers.  Type of business, where they stay, how much they earn, etc.  Use this information as input attributes to learn a classifier model. From [Berry & Linoff] Data Mining Techniques, 1997

44. Association Rule

45. Association Rule Discovery: Definition  Given a set of records each of which contain some number of items from a given collection;  Produce dependency rules which will predict occurrence of an item based on occurrences of other items. Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer} Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer}

46. Association Rule Discovery: A Business Application  Supermarket shelf management.  Goal: To identify items that are bought together by sufficiently many customers.  Approach: Process the point-of-sale data collected with barcode scanners to find dependencies among items.  A classic rule --  If a customer buys diaper and milk, then he is very likely to buy beer.  So, don’t be surprised if you find six-packs stacked next to diapers!

47. Application of Association Rule Discovery:  Shelf-Management

48. Correlations and time  http://correlate.googlelabs.com/tutorial http://correlate.googlelabs.com/tutorial  Tree  IPad and Apple stock IPadApple stock

49. Any problems with the association rule?  Correlation does not cause causation

50. Deviation/Anomaly Detection

51.  Detect significant deviations from normal behavior  Applications:  Credit Card Fraud Detection  Network Intrusion Detection Typical network traffic at University level may reach over 100 million connections per day

52. Deviation/Anomaly Detection  What kinds of deviations from normal behavior might you want to mine for?  For Credit Card Fraud Detection  Picalo!  For Network Intrusion Detection

53. Applications and Issues

54. Recommender Systems  Use Data Mining techniques to recommend products/services based on user behavior  Use Data Mining techniques to recommend products/services based on user specification

55. Data Mining: Some Concerns  Is your interpretation valid?  Do you have enough data to process for “good” results?  Finally… should the government be able to data mine?

56. Data Mining is NOT MAGIC Data Mining will not do any of the following: Automatically find answers to questions you do not ask Constantly monitor your database for new and interesting relationships Eliminate the need to understand your business and your data Remove the need for good data analysis skills

57. “The Great Giveaway”  1. In what way did Amazon, Ebay, and Google “expose themselves?”  2. Why did they make the decision to open their doors?  3. How might this move “change business in ways they don’t understand?”  4. How are “hacker entrepreneurs” taking advantage of this move?  5. Do you think that this move was a smart one for Amazon, Google, and Ebay? Why or why not?  6. Pretend that you’re a “hacker entrepreneur. What kind of application are you going to develop with all of that data?

58. Tomorrow’s class  Read: http://en.wikipedia.org/wiki/Decision_support_system http://en.wikipedia.org/wiki/Decision_support_system  Bring your computer and excel (meet here and we’ll move to a room with computers halfway through class)