Business Intelligence: A Managerial Approach (2nd Edition) Chapter 4: Data Mining for Business Intelligence

4.1 DATA MINING CONCEPTS AND DEFINITIONS It used to understanding customers, vendors, business processes, and the extended supply chain very well. Although the term data mining is relatively new, the ideas behind it are not. Why, then, has it suddenly gained the attention of the business world??

4.1 DATA MINING CONCEPTS AND DEFINITIONS WHY DATA MINING ? More intense competition at the global scale Recognition of the value in data sources Availability of quality data on customers, vendors, transactions, Web, etc. Consolidation and integration of data repositories into data warehouses The exponential increase in data processing and storage capabilities; and decrease in cost Movement toward conversion of information resources into nonphysical form

Definitions, Characteristics, and Benefits is a term used to describe discovering or "mining" knowledge from large amounts of data. “knowledge mining” Technically speaking, data mining is a process that uses statistical, mathematical, and artificial intelligence techniques to extract and identify useful information and subsequent knowledge (or patterns) from large sets of data.

Definition The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structured databases Keywords in this definition: Process, nontrivial , valid, novel, potentially useful, understandable Other names: knowledge extraction, pattern analysis, knowledge discovery, information harvesting, pattern searching, data dredging

Data Mining Characteristics/Objectives Source of data for DM is often a consolidated data warehouse (not always!). DM environment is usually a client-server Sophisticated new tools, including advanced visualization tools, help to remove the information ore buried in corporate files or archival public records The miner is often an end user. Striking it rich requires creative thinking. Data mining tools are readily combined with spreadsheets and other software development tools. Because of the large amounts of data and massive search efforts, it is sometimes necessary to use parallel processing for data mining.

Data Mining at the Intersection of Many Disciplines

What Does DM Do? How Does it Work? DM extracts patterns from data  Pattern? A mathematical (numeric and/or symbolic) relationship among data items Types of patterns  Association  Prediction  Cluster (segmentation)  Sequential (or time series) relationships

A Taxonomy for Data Mining Tasks.

Data Mining Tasks. PREDICTION :is commonly referred to as the act of telling about the future. prediction can be named more specifically as classification: (where the predicted thing, such as tomorrow's forecast, is a class label such as "rainy" or "sunny") or regression: (where the predicted thing, such tomorrow's temperature, is a real number such as "65°F"). Classification, is perhaps the most common of all data mining tasks. The objective of classification is to analyze the historical data stored in a database and automatically generate a model that can predict future behavior. (Neural networks OR Decision trees )

Data Mining Tasks. Clustering: partitions a collection of things (e.g. , objects and events presented in a structured dataset) into segments (or natural groupings) whose members share similar characteristics. market segmentation with cluster analysis. OR segmenting customers.

Data Mining Tasks. Associations: is a popular and well-researched technique for discovering interesting relationships among variables in large databases In the context of the retail industry , association rule mining is often called market-basket analysis

Data Mining Applications Customer Relationship Management Banking & Other Financial Retailing and Logistics Manufacturing and Maintenance Brokerage and Securities Trading Insurance Computer hardware and software Science and engineering Government and defense Homeland security and law enforcement Travel industry Healthcare Medicine Entertainment industry Sports Etc

Data Mining Process: CRISP-DM Cross-Indust1y Standard Process proposed in the mid-1990s by a European consortium of companies

Data Mining Process: CRISP-DM Step 1: Business Understanding know what the study is for like ("What are the common characteristics of the customers we have) Step 2: Data Understanding identify the relevant data from many available databases. quantitative OR qualitative Step 3: Data Preparation (!) (table 4.4) called as data preprocessing . take the data identified in the previous step and prepare them for analysis by data mining data preprocessing consumes the most time and effort; most believe that this step accounts for roughly 80% of the total time.

Data Mining Process: CRISP-DM Step 4: Model Building use a variety of data mining methods and algorithms Step 5: Testing and Evaluation a critical and challenging task Step 6: Deployment The deployment step may also include maintenance activities

Data Preparation – A Critical DM Task (see table 4.4)

Data Mining Process: SEMMA

Source: KDNuggets.com, August 2007 Data Mining Process Source: KDNuggets.com, August 2007

Data Mining Methods: Classification Most frequently used DM method Part of the machine-learning family Employ supervised learning Learn from past data, classify new data The output variable is categorical (nominal or ordinal) in nature

Classification Techniques Decision tree analysis Statistical analysis Neural networks Support vector machines Case-based reasoning Bayesian classifiers Genetic algorithms Rough sets

Decision Trees Employs the divide and conquer method Recursively divides a training set until each division consists of examples from one class Create a root node and assign all of the training data to it Select the best splitting attribute Add a branch to the root node for each value of the split. Split the data into mutually exclusive subsets along the lines of the specific split Repeat the steps 2 and 3 for each and every leaf node until the stopping criteria is reached A general algorithm for decision tree building

Decision Tree

Cluster Analysis for Data Mining Used for automatic identification of natural groupings of things Part of the machine-learning family Employ unsupervised learning Learns the clusters of things from past data, then assigns new instances There is not an output variable Also known as segmentation

Cluster Analysis for Data Mining Clustering results may be used to Identify natural groupings of customers Identify rules for assigning new cases to classes for targeting/diagnostic purposes Provide characterization, definition, labeling of populations Decrease the size and complexity of problems for other data mining methods Identify outliers in a specific domain (e.g., rare-event detection)

Cluster Analysis for Data Mining Analysis methods Statistical methods (including both hierarchical and nonhierarchical), such as k-means, k-modes, and so on Neural networks (adaptive resonance theory [ART], self-organizing map [SOM]) Fuzzy logic (e.g., fuzzy c-means algorithm) Genetic algorithms Divisive versus Agglomerative methods

Cluster Analysis for Data Mining k-Means Clustering Algorithm k : pre-determined number of clusters Algorithm (Step 0: determine value of k) Step 1: Randomly generate k random points as initial cluster centers Step 2: Assign each point to the nearest cluster center Step 3: Re-compute the new cluster centers Repetition step: Repeat steps 3 and 4 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable)

Cluster Analysis for Data Mining- k-Means Clustering Algorithm

Association Rule Mining A very popular DM method in business Finds interesting relationships (affinities) between variables (items or events) Part of machine learning family Employs unsupervised learning There is no output variable Also known as market basket analysis Often used as an example to describe DM to ordinary people, such as the famous “relationship between diapers and beers!”

Association Rule Mining Input: the simple point-of-sale transaction data Output: Most frequent affinities among items Example: according to the transaction data… “Customer who bought a laptop computer and a virus protection software, also bought extended service plan 70 percent of the time." How do you use such a pattern/knowledge? Put the items next to each other for ease of finding Promote the items as a package (do not put one on sale if the other(s) are on sale) Place items far apart from each other so that the customer has to walk the aisles to search for it, and by doing so potentially seeing and buying other items

Association Rule Mining A representative applications of association rule mining include In business: cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, product pricing, and sales/promotion configuration In medicine: relationships between symptoms and illnesses; diagnosis and patient characteristics and treatments (to be used in medical DSS); and genes and their functions (to be used in genomics projects)…

Association Rule Mining Are all association rules interesting and useful? A Generic Rule: X  Y [S%, C%] X, Y: products and/or services X: Left-hand-side (LHS) Y: Right-hand-side (RHS) S: Support: how often X and Y go together C: Confidence: how often Y go together with the X Example: {Laptop Computer, Antivirus Software}  {Extended Service Plan} [30%, 70%]

Association Rule Mining Apriori Algorithm Finds subsets that are common to at least a minimum number of the itemsets uses a bottom-up approach frequent subsets are extended one item at a time (the size of frequent subsets increases from one-item subsets to two-item subsets, then three-item subsets, and so on), and groups of candidates at each level are tested against the data for minimum support see the figure…

Association Rule Mining Apriori Algorithm

Problem Decomposition – Example X  Y Support = X and Y / T Confidence = supp ( x,y) / supp ( X) For the rule Shoes  Jacket Support = Sup({Shoes,Jacket)}= 2/4= .5 =50% Confidence =0.5 / 0.75 =66.6% Jacket  Shoes has 50% support and 66.6% confidence

Element of ANN PROCESSING ELEMENTS (PE) The PE of an ANN are essentially artificial neurons. Similar to biological neurons. INFORMATION PROCESSING The inputs received by a neuron go through a two-step process to turn into outputs: summation function and transformation function NETWORK STRUCTURE Each ANN is composed of a collection of neurons (or PE) that are grouped into layers

Text and Web Data mining

Text Mining Concepts 85-90 percent of all corporate data is in some kind of unstructured form (e.g., text) Unstructured corporate data is doubling in size every 18 months Tapping into these information sources is not an option, but a need to stay competitive Answer: text mining A semi-automated process of extracting knowledge from unstructured data sources a.k.a. text data mining or knowledge discovery in textual databases

Data Mining versus Text Mining Both seek for novel and useful patterns Both are semi-automated processes Difference is the nature of the data: Structured versus unstructured data Structured data: in databases Unstructured data: Word documents, PDF files, text excerpts, XML files, and so on Text mining – first, impose structure to the data, then mine the structured data

Text Mining Concepts Benefits of text mining are obvious especially in text-rich data environments e.g., law (court orders), academic research (research articles), finance (quarterly reports), medicine (discharge summaries), biology (molecular interactions), technology (patent files), marketing (customer comments), etc. Electronic communization records (e.g., Email) Spam filtering Email prioritization and categorization Automatic response generation

Text Mining Application Area Information extraction Topic tracking Summarization Categorization Clustering Concept linking Question answering

Web Mining Overview Web is the largest repository of data Data is in HTML, XML, text format Challenges (of processing Web data) The Web is too big for effective data mining The Web is too complex The Web is too dynamic The Web is not specific to a domain The Web has everything Opportunities and challenges are great!

Web Mining Web mining (or Web data mining) is the process of discovering intrinsic relationships from Web data (textual, linkage, or usage)