Guest Lecture Introduction to Data Mining Dr. Bhavani Thuraisingham September 17, 2010

4-2 6/3/ :51 Objective of the Unit 0 This unit provides an introduction to data mining

4-3 6/3/ :51 Outline of Data Mining 0 What is Data Mining? 0 Data warehousing vs data mining 0 Steps to Data Mining 0 Need for Data Mining 0 Example Applications 0 Technologies for Data Mining 0 Why Data Mining Now? 0 Preparation for Data Mining 0 Data Mining Tasks, Methodology, Techniques 0 Commercial Developments 0 Status, Challenges, and Directions

4-4 6/3/ :51 What is Data Mining? Data Mining Knowledge Mining Knowledge Discovery in Databases Data Archaeology Data Dredging Database Mining Knowledge Extraction Data Pattern Processing Information Harvesting Siftware The process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data, often previously unknown, using pattern recognition technologies and statistical and mathematical techniques (Thuraisingham 1998)

4-5 6/3/ :51 Data Warehouses vs Data Mining 0 Goal: Improved business efficiency -Improve marketing (advertise to the most likely buyers) -Inventory reduction (stock only needed quantities) 0 Information source: Historical business data -Example: Supermarket sales records -Size ranges from 50k records (research studies) to terabytes (years of data from chains) -Data is already being warehoused 0 Sample question – what products are generally purchased together? The answers are in the data, need to MINE the data

4-6 6/3/ :51 What Does Warehousing do for Data Mining? 0 Difficult to mine disparate data sources 0 Data warehouse integrates the disparate data sources into a single logical entity 0 Maintains integrity of the data -Scrubbing and Cleaning 0 Formats the data for querying and mining -Multidimensional data

4-7 6/3/ :51 Is it Necessary to Have a Data Warehouse for Data Mining? 0 Key to successful data mining is having good data 0 Data warehousing integrates heterogeneous data sources, formats the data, and facilitates interactive query processing 0 Having a data warehouse is good for data mining, but perhaps not essential 0 Data mining tools could be used directly on good/clean databases

4-8 6/3/ :51 What’s going on in data mining? 0 What are the technologies for data mining? -Database management, data warehousing, machine learning, statistics, pattern recognition, visualization, parallel processing 0 What can data mining do for you? -Data mining outcomes: Classification, Clustering, Association, Anomaly detection, Prediction, Estimation,... 0 How do you carry out data mining? -Data mining techniques: Decision trees, Neural networks, Market-basket analysis, Link analysis, Genetic algorithms,... 0 What is the current status? -Many commercial products mine relational databases 0 What are some of the challenges? -Mining unstructured data, extracting useful patterns, web mining, Data mining, national security and privacy

4-9 6/3/ :51 Steps to Data Mining Data Sources Integrate data sources Clean/ modify data sources Mine the data Examine Results/ Prune results Report final results Take Actions

4-10 6/3/ :51 Knowledge Directed to Data Mining Data Sources Integrate data sources Clean/ modify data sources Mine the data Examine Results/ Prune results Report final results Expert System Take Actions

4-11 6/3/ :51 0 Large amounts of current and historical data being stored 0 As databases grow larger, decision-making from the data is not possible; need knowledge derived from the stored data 0 Data for multiple data sources and multiple domains -Medical, Financial, Military, etc. 0 Need to analyze the data -Support for planning (historical supply and demand trends) -Yield management (scanning airline seat reservation data to maximize yield per seat) -System performance (detect abnormal behavior in a system) -Mature database analysis (clean up the data sources) Need for Data Mining

4-12 6/3/ :51 Example Applications 0 Medical supplies company increases sales by targeting certain physicians in its advertising who are likely to buy the products 0 A credit bureau limits losses by selecting candidates who are likely not to default on their payment 0 An Intelligence agency determines abnormal behavior of its employees 0 An investigation agency finds fraudulent behavior of some people

4-13 6/3/ :51 Integration of Multiple Technologies Machine Learning Database Management Data Warehousing Statistics Data Mining Visualization Parallel Processing

4-14 6/3/ :51 Why Data Mining Now? 0 Large amounts of data is being produced 0 Data is being organized 0 Technologies are developing for database management, data warehousing, parallel processing, machine intelligent, etc. 0 It is now possible to mine the data and get patterns and trends 0 Interesting applications exist

4-15 6/3/ :51 Preparation for Data Mining 0 Getting the data into the right format 0 Data warehousing 0 Scrubbing and cleaning the data 0 Some idea of application domain 0 Determining the types of outcomes -e.g., Clustering, classification 0 Evaluation of tools 0 Getting the staff trained in data mining

4-16 6/3/ :51 Some Data Mining Tasks/Outcomes 0 Data Mining Tasks -Classification -Estimation -Prediction -Affinity Grouping -Clustering -Description -Other =Deviation detection, Anomaly detection, Association 0 Note: Different text and papers use different terms to mean different tasks -e.g., Association and Affinity Groups have been used interchangeably

4-17 6/3/ :51 Some Types of Data Mining (Data Mining Tasks/Outcomes) 0 Classification – grouping records into meaningful subclasses -e.g., Marketing organization has a list of people living in Manhattan all owning cars costing over 20K 0 Sequence Detection -John always buys groceries after going to the bank 0 Data dependency analysis – identifying potentially interesting dependencies or relationships among data items -If John, James, and Jane meet, Bill is also present 0 Deviation detection – discovery of significant differences between an observation and some reference -Anomalous instances -Discrepancies between observed and expected values

4-18 6/3/ :51 Data Mining Methodology (or Approach) 0 Top-down -Hypothesis testing =Validate beliefs 0 Bottom-up -Discover patterns -Directed =Some idea what you want to get -Undirected =Start from fresh

4-19 6/3/ :51 Outline of Data Mining Techniques 0 Data Mining Techniques -Market Basket Analysis -Memory-based Detection -Automatic Cluster Detection -Link Analysis -Decision Trees and Rule Induction -Neural Networks -Inductive Logic Programming -Other techniques 0 Some Observations

4-20 6/3/ :51 Commercial Developments in Data Mining: Some Early Products 0 Information Discovery-IDIS 0 WizSoft - WhizWhy 0 Hugin - Hugin 0 IBM - Intelligent Miner 0 Red Brick – DataMind (became part of Informix and now part of IBM) 0 Neo Vista - Decision Series 0 Reduct Systems - Datalogic/R 0 Lockheed Martin - Recon 0 Nicesoft – Nicel 0 SAS – Enterprise Miner 0 Recent products will be discussed in Unit #9

4-21 6/3/ :51 Current Status, Challenges and Directions 0 Status -Data Mining is now a technology -Several prototypes and tools exist; Many or almost all of them work on relational databases 0 Challenges -Mining large quantities of data; Dealing with noise and uncertainty; False positives and negatives 0 Directions -Mining multimedia and text databases, Web mining (structure, usage and content), Data mining, national security and privacy