1. Chapter 5: Introduction to Predictive Modeling: Neural Networks and Other Modeling Tools
5.2 Input Selection
5.3 Stopped Training
5.4 Other Modeling Tools (Self-Study)

2. Chapter 5: Introduction to Predictive Modeling: Neural Networks and Other Modeling Tools
5.2 Input Selection
5.3 Stopped Training
5.4 Other Modeling Tools (Self-Study)

3. Model Essentials – Neural Networks
Prediction
formula
Predict new cases.
None
Select useful inputs.
Stopped
training
Optimize complexity.
...

4. Model Essentials – Neural Networks
Prediction
formula
Predict new cases.
None
None
Select useful inputs.
Select useful inputs
Stopped
training
Stopped
training
Optimize complexity
Optimize complexity.
...

5. Model Essentials – Neural Networks
Prediction
formula
Predict new cases.
None
Select useful inputs.
Stopped
training
Optimize complexity.
...

6. Neural Network Prediction Formula
hidden unit
prediction
estimate
bias
estimate
weight
estimate 1 5 -5 -1
tanh
activation
function
...
...

7. Neural Network Prediction Formula
hidden unit
prediction
estimate
bias
estimate
weight
estimate 1 5 -5 -1
tanh
activation
function
...

8. Neural Network Binary Prediction Formula1 5 -5
logit
link function 1 5 -5 -1
tanh
...

9. Neural Network DiagramH1 H2 H3
hidden
layer x2
input
layer x1 y
target
layer
...

10. Neural Network DiagramH1 H2 H3
hidden
layer x2
input
layer x1 y
target
layer
...

11. Prediction Illustration – Neural Networks
logit equation
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x2
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x1
...

12. Prediction Illustration – Neural Networks
logit equation
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x2
Need weight estimates.
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x1
...

13. Prediction Illustration – Neural Networks
logit equation
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x2
Weight estimates found by maximizing:
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x1
Log-likelihood Function
...

14. Prediction Illustration – Neural Networks
logit equation
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x2
0.50
0.70
0.60
0.30
0.40
0.60
0.50
0.50
0.40
0.60
Probability estimates are obtained by solving the logit equation for p for each (x1, x2). ^
0.0
0.5
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0 x1
...

15. Neural Nets: Beyond the Prediction Formula
Manage missing values.
Handle extreme or unusual values.
Handle extreme or unusual values
Use non-numeric inputs
Use non-numeric inputs.
Account for nonlinearities.
Account for nonlinearities
Interpret the model.
Interpret the model
...

16. Neural Nets: Beyond the Prediction Formula
Manage missing values.
Handle extreme or unusual values.
Use non-numeric inputs.
Account for nonlinearities.
Interpret the model.

17. Training a Neural Network
This demonstration illustrates using the Neural Network tool.

18. Chapter 5: Introduction to Predictive Modeling: Neural Networks and Other Modeling Tools
5.2 Input Selection
5.3 Stopped Training
5.4 Other Modeling Tools (Self-Study)

19. Model Essentials – Neural Networks
Prediction
formula
Predict new cases.
Sequential
selection
None
Select useful inputs.
Select useful inputs
Best model
from sequence
Optimize complexity.

21. 5.01 Multiple Answer Poll
Which of the following are true about neural networks in SAS Enterprise Miner?
Neural networks are universal approximators.
Neural networks have no internal, automated process for selecting useful inputs.
Neural networks are easy to interpret and thus are very useful in highly regulated industries.
Neural networks cannot model nonlinear relationships.
Type answer here

22. 5.01 Multiple Answer Poll – Correct Answers
Which of the following are true about neural networks in SAS Enterprise Miner?
Neural networks are universal approximators.
Neural networks have no internal, automated process for selecting useful inputs.
Neural networks are easy to interpret and thus are very useful in highly regulated industries.
Neural networks cannot model nonlinear relationships.
Type answer here

23. Selecting Neural Network Inputs
This demonstration illustrates how to use a logistic regression to select inputs for a neural network.

24. Chapter 5: Introduction to Predictive Modeling: Neural Networks and Other Modeling Tools
5.2 Input Selection
5.3 Stopped Training
5.4 Other Modeling Tools (Self-Study)

25. Model Essentials – Neural Networks
Prediction
formula
Predict new cases.
Sequential
selection
Select useful inputs.
Stopped training
Optimize complexity.
...

26. Fit Statistic versus Optimization Iteration
initial hidden unit weights
logit( p ) = ^
+ 0·H1 + 0·H2 + 0·H3
logit(0.5)
logit(ρ1) ^
H1 = tanh( x x2)
H2 = tanh( x x2)
H3 = tanh( x1 +.35x2 )
...

27. Fit Statistic versus Optimization Iteration
logit( p ) = ^
+ 0·H1 + 0·H2 + 0·H3
H1 = tanh( x x2)
H1 = tanh( x x2)
H2 = tanh( x x2)
H2 = tanh( x x2)
H3 = tanh( x1 +.35x2 )
H3 = tanh( x1 +.35x2 )
random initial
input weights and biases
...

28. Fit Statistic versus Optimization Iteration
logit( p ) = ^
+ 0·H1 + 0·H2 + 0·H3
H1 = tanh( x x2)
H1 = tanh( x x2)
H2 = tanh( x x2)
H2 = tanh( x x2)
H3 = tanh( x1 +.35x2 )
H3 = tanh( x1 +.35x2 )
random initial
input weights and biases
...

29. Fit Statistic versus Optimization Iteration5 15 20
Iteration 10
...

30. Fit Statistic versus Optimization Iteration
ASE
training
validation 1 5 10 15 20
Iteration
...

31. Fit Statistic versus Optimization Iteration
ASE
training
validation 2 5 10 15 20
Iteration
...

32. Fit Statistic versus Optimization Iteration
ASE
training
validation 3 5 10 15 20
Iteration
...

33. Fit Statistic versus Optimization Iteration
ASE
training
validation 4 5 10 15 20
Iteration
...

34. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20
Iteration
...

35. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 6 10 15 20
Iteration
...

36. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 7 10 15 20
Iteration
...

37. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 8 10 15 20
Iteration
...

38. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 9 10 15 20
Iteration
...

39. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20
Iteration
...

40. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 11 15 20
Iteration
...

41. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 12 15 20
Iteration
...

42. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 13 15 20
Iteration
...

43. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 14 15 20
Iteration
...

44. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20
Iteration
...

45. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 16 20
Iteration
...

46. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 17 20
Iteration
...

47. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 18 20
Iteration
...

48. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 19 20
Iteration
...

49. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20
Iteration
...

50. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20 21
Iteration
...

51. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20 22
Iteration
...

52. Fit Statistic versus Optimization Iteration
ASE
training
validation 5 10 15 20 23
Iteration
...

53. Fit Statistic versus Optimization Iteration
0.50
0.70
0.60
0.30
ASE
0.40
0.60
0.50
0.50
0.40
0.60 5 10 12 15 20
Iteration
...

54. Increasing Network Flexibility
This demonstration illustrates how to further improve neural network performance.

55. Using the AutoNeural Tool (Self-Study)
This demonstration illustrates how to use the AutoNeural tool.

56. Chapter 5: Introduction to Predictive Modeling: Neural Networks and Other Modeling Tools
5.2 Input Selection
5.3 Stopped Training
5.4 Other Modeling Tools (Self-Study)

57. Model Essentials – Rule Induction
Prediction rules /
prediction formula
Predict new cases.
Split search /
none
Select useful inputs.
Ripping /
stopped training
Optimize complexity.

58. Rule Induction Predictions
[Rips create prediction rules.]
A binary model sequentially classifies and removes correctly classified cases.
[A neural network predicts remaining cases.]
1.0
0.74
0.9
0.8
0.7
0.6 x2
0.5
0.4
0.3
0.2
0.39
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 x1

59. Model Essentials – Dmine Regression
Prediction formula
Predict new cases.
Forward selection
Select useful inputs.
Stop R-square
Optimize complexity.

60. Dmine Regression Predictions
Interval inputs binned, categorical inputs grouped
Forward selection picks from binned and original inputs

61. Model Essentials – DMNeural
Stagewise
prediction formula
Predict new cases.
Principal
component
Select useful inputs.
Max stage
Optimize complexity.

62. DMNeural Predictions
Up to three PCs with highest target R square are selected.
One of eight continuous transformations are selected and applied to selected PCs.
The process is repeated three times with residuals from each stage.
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 x1

63. Model Essentials – Least Angle Regression
Prediction formula
Predict new cases.
Generalized
sequential selection
Select useful inputs.
Penalized best
fit
Optimize complexity.

64. Least Angle Regression Predictions
Inputs are selected using a generalization of forward selection.
An input combination in the sequence with optimal, penalized validation assessment is selected by default.
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 x1

65. Model Essentials – MBR Predict new cases. Select useful inputs.
Training data
nearest neighbors
Predict new cases.
None
Select useful inputs.
Number of
neighbors
Optimize complexity.

66. MBR Prediction Estimates
Sixteen nearest training data cases predict the target for each point in the input space.
Scoring requires training data and the PMBR procedure.
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 x1

67. Model Essentials – Partial Least Squares
Prediction formula
Predict new cases.
VIP
Select useful inputs.
Sequential factor
extraction
Optimize complexity.

68. Partial Least Squares Predictions
Input combinations (factors) that optimally account for both predictor and response variation are successively selected.
Factor count with a minimum validation PRESS statistic is selected.
Inputs with small VIP are rejected for subsequent diagram nodes.
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 x1

69. Exercises
This exercise reinforces the concepts discussed previously.

70. Neural Network Tool Review
Create a multi-layer perceptron on selected inputs. Control complexity with stopped training and hidden unit count.