1. Promising “Newer” Technologies to Cope with the
Information Flood
Knowledge Discovery and Data Mining (KDD)
Agent-based Technologies
Ontologies and Knowledge Brokering
Non-traditional data analysis techniques
Model Generation
As an Example
To Explain /
Discuss Technologies
As I mentioned in the introduction the goal of this talk is to introduce and describe newer technologies that in my opinion show some promise to cope with the information flood in health care. In this talk, I will focus on 3 particular technologies, namely, ...
Moreover, during the course of the talk I will not only introduce the technologies but also analyze how they can fertilize each other’s application.

2. Knowledge Discovery in Data [and Data Mining] (KDD)
Let us find something interesting!
Definition := “KDD is the non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data” (Fayyad)
Frequently, the term data mining is used to refer to KDD.
Many commercial and experimental tools and tool suites are available (see
Field is more dominated by industry than by research institutions
The first technology I like to …
The above picture is, in my opinion, a good description of the task of knowledge discovery in that it illustrates a huge search space that contains very very few interesting things, and if applied in practice, KDD is frequently like finding a needle in a hay stack, except that you are not sure what you are looking for...

3. Making Sense of Data --- Knowledge Discovery and Data Mining
Introduction to KDD (1 class)
Data Warehouses and OLAP (2 classes)
Association Rule Mining (1.5 classes)
Learning to Classify (1 class)
Other Techniques: Clustering, Deviation Detection, Sequential Pattern Analysis (0.5 classes)

4. Data Mining: Confluence of Multiple Disciplines
Database
Technology
Statistics
Data Mining
Machine
Learning
Visualization
Information
Science
Other
Disciplines

5. Interpret/Evaluate/Assimilate
General KDD Steps
Data sources
Selected/Preprocessed data
Transformed data
Extracted information
Knowledge
Select/preprocess
Transform
Data mine
Interpret/Evaluate/Assimilate
Data preparation

6. Popular KDD-Tasks Classification (try to learn how to classify)
Clustering (finding groups of similar object)
Estimation and Prediction (try to learn a function that predicts an th value of a continuous output variable based on a set of input variables)
Bayesian and Dependency Networks
Deviation and Fraud Detection
Text Mining
Web Mining
Visualization
Transformation and Data Cleaning

7. KDD and Classical Data Analysis
KDD is less focused than data analysis in that it looks for interesting patterns in data; classical data analysis centers on analyzing particular relationships in data. The notion of interestingness is a key concept in KDD. Classical data analysis centers more on generating and testing pre-structured hypothesis with respect to a given sample set.
KDD is more centered on analyzing large volumes of data (many fields, many tuples, many tables, …).
In a nutshell the the KDD-process consists of preprocessing (generating a target data set), data mining (finding something interesting in the data set), and post processing (representing the found pattern in understandable form and evaluated their usefulness in a particular domain); classical data analysis is less concerned with the the preprocessing step.
KDD involves the collaboration between multiple disciplines: namely, statistics, AI, visualization, and databases.
KDD employs non-traditional data analysis techniques (neural networks, association rules, decision trees, fuzzy logic, evolutionary computing,…).

8. Generating Models as an Example
The goal of model generation (sometimes also called predictive data mining) is the creation, evaluation, and use of models to make predictions and to understand the relationships between various variables that are described in a data collection. Typical example application include:
generate a model to that predicts a student’s academic performance based on the applicants data such as the applicant’s past grades, test scores, past degree,…
generate a model that predicts (based on economic data) which stocks to sell, hold, and buy.
generate a model to predict if a patient suffers from a particular disease based on a patient’s medical and other data.
Model generation centers on deriving a function that can predict a variable using the values of other variables: v=f(a1,…,an)
Neural networks, decision trees, naïve Bayesian classifiers and networks, regression analysis and many other statistical techniques, fuzzy logic and neuro-fuzzy systems, association rules are the most popular model generation tools in the KDD area.
All model generation tools and environments employ the basic train-evaluate-predict cycle.
In the next section of this talk, I will describe how agent-based architectures could be used to support knowledge discovery and data mining, centering on the task of model generation.
The goal of this section is to give you an idea what it means to use for intelligent agents for software development, and to illustrate how using agents is different from a more traditional software development approach.

9. Why Do We Need so many Data Mining / Analysis Techniques?
No generally good technique exists.
Different methods make different assumptions with respect to the data set to be analyzed (to be discussed on the next transparency)
Cross fertilization between different methods is desirable and frequently helpful in obtaining a deeper understanding of the analyzed dataset.

10. Motivation: “Necessity is the Mother of Invention”
Data explosion problem
Automated data collection tools and mature database technology lead to tremendous amounts of data stored in databases, data warehouses and other information repositories
We are drowning in data, but starving for knowledge!
Solution: Data warehousing and data mining
Data warehousing and on-line analytical processing
Extraction of interesting knowledge (rules, regularities, patterns, constraints) from data in large databases

11. Why Data Mining? — Potential Applications
Database analysis and decision support
Market analysis and management
target marketing, customer relation management, market basket analysis, cross selling, market segmentation
Risk analysis and management
Forecasting, customer retention, improved underwriting, quality control, competitive analysis
Fraud detection and management
Other Applications
Text mining (news group, , documents) and Web analysis.
Intelligent query answering

12. Market Analysis and Management
Where are the data sources for analysis?
Credit card transactions, loyalty cards, discount coupons, customer complaint calls, plus (public) lifestyle studies
Target marketing
Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc.
Determine customer purchasing patterns over time
Conversion of single to a joint bank account: marriage, etc.
Cross-market analysis
Associations/co-relations between product sales
Prediction based on the association information

13. Fraud Detection and Management
Applications
widely used in health care, retail, credit card services, telecommunications (phone card fraud), etc.
Approach
use historical data to build models of fraudulent behavior and use data mining to help identify similar instances
Examples
auto insurance: detect a group of people who stage accidents to collect on insurance
money laundering: detect suspicious money transactions (US Treasury's Financial Crimes Enforcement Network)
medical insurance: detect professional patients and ring of doctors and ring of references

14. Other Applications Sports
IBM Advanced Scout analyzed NBA game statistics (shots blocked, assists, and fouls) to gain competitive advantage for New York Knicks and Miami Heat
Astronomy
JPL and the Palomar Observatory discovered 22 quasars with the help of data mining
Internet Web Surf-Aid
IBM Surf-Aid applies data mining algorithms to Web access logs for market-related pages to discover customer preference and behavior pages, analyzing effectiveness of Web marketing, improving Web site organization, etc.

15. Data Mining and Business Intelligence
Increasing potential
to support
business decisions
End User
Making
Decisions
Data Presentation
Business
Analyst
Visualization Techniques
Data Mining
Data
Analyst
Information Discovery
Data Exploration
Statistical Analysis, Querying and Reporting
Data Warehouses / Data Marts
OLAP, MDA
DBA
Data Sources
Paper, Files, Information Providers, Database Systems, OLTP

16. Architecture of a Typical Data Mining System
Graphical user interface
Pattern evaluation
Data mining engine
Knowledge-base
Database or data warehouse server
Filtering
Data cleaning & data integration
Data
Warehouse
Databases

17. An OLAM Architecture Mining query Mining result OLAM Engine OLAP
Layer4
User Interface
User GUI API
OLAM
Engine
OLAP
Engine
Layer3
OLAP/OLAM
Data Cube API
Layer2
MDDB
MDDB
Meta Data
Database API
Filtering&Integration
Filtering
Layer1
Data Repository
Data cleaning
Data
Warehouse
Databases
Data integration

18. Example: Decision Tree Approach

19. Decision Tree Approach2

20. Decision Trees Example:
Conducted survey to see what customers were interested in new model car
Want to select customers for advertising campaign
training
set

21. One Possibility age<30 city=sf car=van likely unlikely likely

22. Another Possibility car=taurus city=sf age<45 likely unlikely

23. Example: Nearest Neighbor Approach

24. Clustering
income
education
age

25. Another Example: Text Each document is a vector
e.g., < > contains words 1,4,5,...
Clusters contain “similar” documents
Useful for understanding, searching documents
sports
international
news
business

26. Issues Given desired number of clusters? Finding “best” clusters
Are clusters semantically meaningful?
e.g., “yuppies’’ cluster?
Using clusters for disk storage

27. Association Rule Mining
transaction id
customer id
products
bought
sales
records:
market-basket
data
Example Rules:
age(X, “20..29”) ^ income(X, “20..29K”)  buys(X, “PC”) [support = 2%, confidence = 60%]
buys(x, p2) ^ buys(x,p5)  bus(x,p8) [1%, 85%]

28. Characteristics and Assumptions of Popular Data Mining/Analysis Techniques
Distance based approaches (assume that a distance function with respect to the objects in the dataset exists) vs. order-based approaches (just use the ordering of values in their decision making; 3>2>1 is indistinguishable from 2.01>2>1.99)
Approaches that make no assumptions / assume a particular distribution of the data in the underlying dataset.
Differences in employed approximation techniques
Rectangular vs. other approximation
Linear vs. non-linear approximations
Sensitivity to redundant attributes (variables)
Sensitivity to irrelevant attributes
Sensitivity to attributes of different degrees of importance
Different Training Performance / Testing Performance
What does the learnt function tell us about the analyzed data set? How difficult is it to understand the learnt function?
Deterministic / non-deterministic approaches
Stability of the obtained results

29. Summary KDD
KDD: discovering interesting patterns from large amounts of data
A natural evolution of database technology, in great demand, with wide applications
A KDD process includes data cleaning, data integration, data selection, transformation, data mining, pattern evaluation, and knowledge presentation
Mining can be performed in a variety of information repositories
Data mining functionalities: characterization, discrimination, association, classification, clustering, outlier and trend analysis, etc.
Multi-disciplinary activity
Important Issues: KDD-methodologies and user-interactions, scalability, tool use and tool integration, preprocessing, interpretation of results, finding good parameter settings when running data mining tools,…

30. Where to Find References?
Data mining and KDD (SIGKDD member CDROM):
Conference proceedings: KDD, and others, such as PKDD, PAKDD, etc.
Journal: Data Mining and Knowledge Discovery
Database field (SIGMOD member CD ROM):
Conference proceedings: ACM-SIGMOD, ACM-PODS, VLDB, ICDE, EDBT, DASFAA
Journals: ACM-TODS, J. ACM, IEEE-TKDE, JIIS, etc.
AI and Machine Learning:
Conference proceedings: Machine learning, AAAI, IJCAI, etc.
Journals: Machine Learning, Artificial Intelligence, etc.
Statistics:
Conference proceedings: Joint Stat. Meeting, etc.
Journals: Annals of statistics, etc.
Visualization:
Conference proceedings: CHI, etc.
Journals: IEEE Trans. visualization and computer graphics, etc.