1. Business Intelligence
Lecture 9
Chapter 6 – Artificial Neural Networks for Data Mining
1/31/2011
Business Intelligence

2. Learning Objectives
Understand the concept and definitions of artificial neural networks (ANN)
Know the similarities and differences between biological and artificial neural networks
Learn the different types of neural network architectures
Learn the advantages and limitations of ANN
Understand how backpropagation learning works in feedforward neural networks
1/31/2011
Business Intelligence

3. Learning Objectives
Understand the step-by-step process of how to use neural networks
Appreciate the wide variety of applications of neural networks; solving problem types of
Classification
Regression
Clustering
Association
Optimization
1/31/2011
Business Intelligence

4. Opening Vignette: (Page 242)
“Predicting Gambling Referenda with Neural Networks”
Using NeuroSolutions, this study developed and tested models to predict community support for commercial gaming.
The study examined the role of factors that contribute to legalization and/or probation of gambling activities using neural networks.
It attempted to use Neural Network technology to predict various counties voting outcome on this subject.
1/31/2011
Business Intelligence

5. Opening Vignette:
On average, the models accurately predicted the voting results for 4 out of every 5 counties (approximately 82% accuracy) on a sample data set. (1287 records of data)
Interestingly, and contrary to popular belief, the counties financial characteristics and age distribution were not found to be significant factors in determining ballot outcome. Dominant factors are identified on Page 244
The study demonstrates that demographic data can be used to accurately predict voting outcomes on controversial issues.
1/31/2011
Business Intelligence

6. Opening Vignette: Predicting Gambling Referenda…
1/31/2011
Business Intelligence

7. Opening Vignette:
NeuroSolutions is offered by NeuroDimension and offers algorithms for the in the field of artificial intelligence.
NeuroDimension offers NeuroSolutions, NeuroSolutions for Excel, and a Custom Solution Wizard each of which can be downloaded for a free eval.
1/31/2011
Business Intelligence

8. Opening Vignette:
An very good video is offered by the company that explains Neural Network algorithms and the field in general.
Pricing is relatively reasonable for the product. - NS for Excel costs $295
1/31/2011
Business Intelligence

9. Neural Network Concepts (Page 245)
Neural networks (NN): a brain metaphor for information processing– uses artificial neurons (programming constructs that mimic the properties of biological neurons).
Neural computing - pattern recognition methodology for machine learning
Artificial neural network (ANN) – resulting model from neural computing
Many uses for ANN for
pattern recognition, forecasting, prediction, and classification
finance, marketing, manufacturing, operations, information systems, and so on
1/31/2011
Business Intelligence

10. ANN Video
Here is an excellent video offered by NeuroSolutions that provides a good overview of ANN
1/31/2011
Business Intelligence

11. Biological Neural Networks (Page 246)
Two interconnected brain cells (neurons)
An axon is a long, slender projection of a nerve cell, or neuron, that conducts electrical impulses away from the neuron's cell body or soma.
Dendrites are branched filaments in nerve cells (neurons). The word dendrite derives from the Greek word for tree which describes their branching tree-like structure.
1/31/2011
Business Intelligence

12. Biological Neural Networks (Page 246)
Synapse – able to increase or decrease the strength of the connection between neurons and cause excitation or inhibition of a subsequent neuron.
The word "soma" comes from the the Greek word “body”; the soma of a neuron is often called the cell body.
1/31/2011
Business Intelligence

13. Processing Information in ANN (Page 247 Figure 6.3)
A single neuron (processing element – PE) with inputs and outputs
1/31/2011
Business Intelligence

14. Biology Analogy (Page 247)
1/31/2011
Business Intelligence

15. Elements of ANN (Page 248-250)
Processing element (PE) – organized in different ways to form the networks structure.
Network architecture
Hidden layers - takes input from the previous layer and converts into outputs for more processing (used in complex problems)
Parallel processing – resembles the way the brain works – different than serial processing in conventional computing
Not this ANN
1/31/2011
Business Intelligence

16. Elements of ANN (Page 248-250)
Network information processing
Inputs – single attribute such as age, income level, etc
Outputs – solution to the problem – ie – loan app “yes” or “no”
Connection weights – relative strength of input data (how important)
Summation function – weighted sums of all input elements entering a PE.
1/31/2011
Business Intelligence

17. Elements of ANN (Figure 6.4 Page 249)
Neural Network with One Hidden Layer
1/31/2011
Business Intelligence

18. Elements of ANN
Summation Function for a Single Neuron (a) and Several Neurons (b)
1/31/2011
Business Intelligence

19. Elements of ANN (Page 251) Threshold value
Transformation (Transfer) Function – activation level of a neuron (based on this level the neuron may or may not produce an output).
Computed via Sigmoid (logical activation) function – YT=1/(1+e-Y)
Y is computed via weighted summation
Any value less than threshold will not be passed to output (0); anything above does (1)
Threshold value
1/31/2011
Business Intelligence

20. Neural Network Architectures (Page 251-252)
Several ANN architectures exist
Feedforward - figure 6.4 page 249 (see previous slide)
Recurrent - - figure 6.7 page 252 (next slide)
Associative memory
Self-organizing feature maps
Hopfield networks, etc
1/31/2011
Business Intelligence

21. Neural Network Architectures Recurrent Neural Networks (Page 252, figure 6.7)
1/31/2011
Business Intelligence

22. Neural Network Architectures (Page 252)
Architecture of a neural network is driven by the task it is intended to address
Most popular architecture: Feedforward, multi-layered perceptron with backpropagation learning algorithm
Ie – Feedforward Perceptron is the architecture and backpropagation is the learning algorithm.
1/31/2011
Business Intelligence

23. Neural Network Architectures
Frank Rosenblatt (1957)
The perceptron is a type of artificial neural network invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. Frank Rosenblatt was a computer scientist born in 1928 in New York City.  He helped to create the Perceptron, a.k.a. the Mark 1, computer in 1960 at Cornell University.  This was the first computer that could learn skills by trial and error in an attempt to mimic human thought processes through the use of a neural network.  (Died 1971)
Backpropagation is a common, supervised method for teaching artificial neural networks how to perform a given task. It was first described by Arthur E. Bryson and Yu-Chi Ho in 1969.
1/31/2011
Business Intelligence

24. Neural Network Architectures
Original Mark 1 (Automatic Sequence Controlled Calculator (ASCC)
The building elements of the ASCC were switches, relays, rotating shafts, and clutches.
1/31/2011
Business Intelligence

25. Learning in ANN (Page 252)
A process by which a neural network learns the underlying relationship between input and outputs, or just among the inputs
Supervised learning
For prediction type problems
E.g., backpropagation
Unsupervised learning
For clustering type problems
Self-organizing
E.g., adaptive resonance theory
1/31/2011
Business Intelligence

26. A Taxonomy of ANN Learning Algorithms (Page 253, Figure 6.8)
Most popular
1/31/2011
Business Intelligence

27. Read Application Case (Page 254)
Microsoft used BrainMaker Neural Network software from California Scientific to maximize return on direct Mail
Some of the variables considered (25 in total)
Recency (how long since last registration / product purchase)
First date to file – loyal over time?
Number of products bought and filed
Value of products bought and registered
Number of days from product release to purchase
Improved response rate from 4.9% to 8.2% - 35% cost savings on 40 Millions pieces of direct mailings
1/31/2011
Business Intelligence

28. A Supervised Learning Process (Pages 255-256, figure 6.9)
Three-step process:
1. Compute temporary outputs
2. Compare outputs with desired targets
3. Adjust the weights and repeat the process
1/31/2011
Business Intelligence

29. How a Network Learns (Page 256)
Example: single neuron that learns the inclusive OR operation
Learning parameters:
Learning rate
Momentum
* See page 257 for step-by-step progression of the learning process
1/31/2011
Business Intelligence

30. Backpropagation Learning (Page 258)
Errors are used to correct weights – called Back-error propagation
The (supervised) learning algorithm procedure:
Initialize weights with random values and set other network parameters
Read in the inputs and the desired outputs
Compute the actual output (by working forward through the layers)
Compute the error (difference between the actual and desired output)
Change the weights by working backward through the hidden layers
Repeat steps 2-5 until weights stabilize
1/31/2011
Business Intelligence

31. Backpropagation Learning (Figure 6.10 Page 258)
Backpropagation of Error for a Single Neuron
1/31/2011
Business Intelligence

32. Development Process of an ANN (Page 259)
Similar to structured design for traditional IS, with some new elements
See page 253
1/31/2011
Business Intelligence

33. An MLP ANN Structure for the Box-Office Prediction Problem (Page 262, Fig 6.12) This is the vignette at the start of Chapter 5 of page 191
1/31/2011
Business Intelligence

34. Data Collection and Testing (Page 261)
Data is split into three parts
Training (~60%)
Validation (~20%)
Testing (~20%)
1/31/2011
Business Intelligence

35. Sensitivity Analysis on ANN Models (Page 264-265)
A common criticism for ANN: The black-box syndrome!
Answer: sensitivity analysis
Conducted on a trained ANN
The inputs are changed while the relative change on the output is measured/recorded
Results illustrates the relative importance of input variables
1/31/2011
Business Intelligence

36. Sensitivity Analysis on ANN Models (Page 265, Figure 6.13)
See and read example Application Case 6.5 (Page 266)
Sensitivity analysis reveals the most important injury severity factors in traffic accidents
1/31/2011
Business Intelligence

37. Sensitivity Analysis on ANN Models (Page 266)
Application Case 6.5 – see here
41,000 die in 6M US traffic accidents
Analyze the factors that elevate the risk of severe injury
Factors include behavior, environment, technical, etc.
Used series of ANN models to estimate the significance of the crash factors on the level of severity sustained by the driver.
Two step process used (1) prediction models, (2) sensitivity analysis on trained neural network
Results shows significant differences among models built for different injury severity levels. (The most influential factors HIGHLY depend on the level of injury).
1/31/2011
Business Intelligence

38. A Sample Neural Network Project Bankruptcy Prediction (Pg 267-270)
A comparative analysis of ANN versus logistic regression (LR) (a statistical method)
Inputs
X1: Working capital/total assets
X2: Retained earnings/total assets
X3: Earnings before interest and taxes/total assets
X4: Market value of equity/total debt
X5: Sales/total assets
1/31/2011
Business Intelligence

39. A Sample Neural Network Project Bankruptcy Prediction
Data was obtained from Moody's Industrial Manuals
Time period: 1975 to 1982
129 firms (65 of which went bankrupt during the period and 64 nonbankrupt)
Different training and testing propositions are used/compared
90/10 versus 80/20 versus 50/50
Resampling is used to create 60 data sets
1/31/2011
Business Intelligence

40. A Sample Neural Network Project Bankruptcy Prediction
Network Specifics
Feedforward MLP
Backpropagation
Varying learning and momentum values
5 input neurons (1 for each financial ratio),
10 hidden neurons,
2 output neurons (1 indicating a bankrupt firm and the other indicating a nonbankrupt firm)
1/31/2011
Business Intelligence

41. A Sample Neural Network Project Bankruptcy Prediction – Results (Page 269 figure 6.2)
1/31/2011
Business Intelligence

42. Bottomline - Advantages of ANN (Pages 274-276)
Able to deal with (identify/model) highly nonlinear relationships
Can handle variety of problem types (loan apps, forecast profitability / finances, sports – team success, fraud prevention, time-series forecasting, health care and medicine – diagnose breast cancer – see Case 6.4 on page 276)
Usually provides better results (prediction and/or clustering) compared to its statistical counterparts
1/31/2011
Business Intelligence

43. Disadvantages of ANN
They are deemed to be black-box solutions, lacking expandability
It is hard to find optimal values for large number of network parameters
Optimal design is still an art: requires expertise and extensive experimentation
It is hard to handle large number of variables (especially the rich nominal attributes)
Training may take a long time for large datasets; which may require case sampling
1/31/2011
Business Intelligence

44. ANN Software (Page 263) Standalone ANN software tool
NeuroSolutions
BrainMaker
NeuralWare
NeuroShell, … for more (see pcai.com) …
Part of a data mining software suit
PASW (formerly SPSS Clementine)
SAS Enterprise Miner
Statistica Data Miner, … many more …
1/31/2011
Business Intelligence

45. Chapter 7 - Text and Web Mining
Next lecture
Chapter 7 - Text and Web Mining
1/31/2011
Business Intelligence