Data and Applications Security Introduction to Data Mining Dr. Bhavani Thuraisingham Guest Lecture February 25, 2008

Objective of the Unit This unit provides an introduction to data mining

Outline of Data Mining What is Data Mining? Data warehousing vs data mining Steps to Data Mining Need for Data Mining Example Applications Technologies for Data Mining Why Data Mining Now? Preparation for Data Mining Data Mining Tasks, Methodology, Techniques Commercial Developments Status, Challenges , and Directions

What is Data Mining? Information Harvesting Knowledge Mining Knowledge Discovery in Databases Data Archaeology Data Dredging Database Mining Knowledge Extraction Data Pattern Processing Information Harvesting Siftware The process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data, often previously unknown, using pattern recognition technologies and statistical and mathematical techniques (Thuraisingham 1998)

Data Warehouses vs Data Mining Goal: Improved business efficiency Improve marketing (advertise to the most likely buyers) Inventory reduction (stock only needed quantities) Information source: Historical business data Example: Supermarket sales records Size ranges from 50k records (research studies) to terabytes (years of data from chains) Data is already being warehoused Sample question – what products are generally purchased together? The answers are in the data, need to MINE the data

What Does Warehousing do for Data Mining? Difficult to mine disparate data sources Data warehouse integrates the disparate data sources into a single logical entity Maintains integrity of the data Scrubbing and Cleaning Formats the data for querying and mining Multidimensional data

Is it Necessary to Have a Data Warehouse for Data Mining? Key to successful data mining is having good data Data warehousing integrates heterogeneous data sources, formats the data, and facilitates interactive query processing Having a data warehouse is good for data mining, but perhaps not essential Data mining tools could be used directly on good/clean databases

What’s going on in data mining? What are the technologies for data mining? Database management, data warehousing, machine learning, statistics, pattern recognition, visualization, parallel processing What can data mining do for you? Data mining outcomes: Classification, Clustering, Association, Anomaly detection, Prediction, Estimation, . . . How do you carry out data mining? Data mining techniques: Decision trees, Neural networks, Market-basket analysis, Link analysis, Genetic algorithms, . . . What is the current status? Many commercial products mine relational databases What are some of the challenges? Mining unstructured data, extracting useful patterns, web mining, Data mining, national security and privacy

Steps to Data Mining Clean/ Mine Integrate modify the data data data sources Mine the data Integrate data sources Report final results Take Actions Examine Results/ Prune results Data Sources

Knowledge Directed to Data Mining Mine the data Clean/ modify data sources Integrate data sources Expert System Report final results Take Actions Examine Results/ Prune results Data Sources

Need for Data Mining Large amounts of current and historical data being stored As databases grow larger, decision-making from the data is not possible; need knowledge derived from the stored data Data for multiple data sources and multiple domains Medical, Financial, Military, etc. Need to analyze the data Support for planning (historical supply and demand trends) Yield management (scanning airline seat reservation data to maximize yield per seat) System performance (detect abnormal behavior in a system) Mature database analysis (clean up the data sources)

Example Applications Medical supplies company increases sales by targeting certain physicians in its advertising who are likely to buy the products A credit bureau limits losses by selecting candidates who are likely not to default on their payment An Intelligence agency determines abnormal behavior of its employees An investigation agency finds fraudulent behavior of some people

Integration of Multiple Technologies Data Warehousing Machine Learning Database Management Statistics Parallel Processing Visualization Data Mining

Why Data Mining Now? Large amounts of data is being produced Data is being organized Technologies are developing for database management, data warehousing, parallel processing, machine intelligent, etc. It is now possible to mine the data and get patterns and trends Interesting applications exist

Preparation for Data Mining Getting the data into the right format Data warehousing Scrubbing and cleaning the data Some idea of application domain Determining the types of outcomes e.g., Clustering, classification Evaluation of tools Getting the staff trained in data mining

Some Types of Data Mining (Data Mining Tasks/Outcomes) Classification – grouping records into meaningful subclasses e.g., Marketing organization has a list of people living in Manhattan all owning cars costing over 20K Sequence Detection John always buys groceries after going to the bank Data dependency analysis – identifying potentially interesting dependencies or relationships among data items If John, James, and Jane meet, Bill is also present Deviation detection – discovery of significant differences between an observation and some reference Anomalous instances Discrepancies between observed and expected values

Data Mining Methodology (or Approach) Top-down Hypothesis testing Validate beliefs Bottom-up Discover patterns Directed Some idea what you want to get Undirected Start from fresh

Some Data Mining Techniques Market Basket analysis Decision Trees Neural networks Rough sets and fuzzy logic Inductive logic programming

Commercial Developments in Data Mining: Some Early Products Information Discovery-IDIS WizSoft - WhizWhy Hugin - Hugin IBM - Intelligent Miner Red Brick – DataMind (became part of Informix and now part of IBM) Neo Vista - Decision Series Reduct Systems - Datalogic/R Lockheed Martin - Recon Nicesoft – Nicel SAS – Enterprise Miner Recent products will be discussed in Unit #9

Current Status, Challenges and Directions Data Mining is now a technology Several prototypes and tools exist; Many or almost all of them work on relational databases Challenges Mining large quantities of data; Dealing with noise and uncertainty Directions Mining multimedia and text databases, Web mining (structure, usage and content), Data mining, national security and privacy