Computational Intelligence John Sum Institute of Technology Management National Chung Hsing University Taichung, Taiwan ROC

Computational Intelligence OUTLINE Historical Background Computational Intelligence Example Problems Methodology Model Structure Model Parameters Parametric Estimation Discussion Conclusion John Sum Computational Intelligence

Computational Intelligence HISTORY John Sum Computational Intelligence

Computational Intelligence HISTORY 1940 – First computing machine 1957 – Perceptron (First NN model) 1965 – Fuzzy Logic (Rules) 1960s – Genetic Algorithm 1970s – Evolutionary Computing John Sum Computational Intelligence

Computational Intelligence HISTORY 1980s Neural Computing Swarm Intelligence 1990s (Hybrid) Fuzzy Neural Networks NFG, FGN, GNF, etc John Sum Computational Intelligence

Computational Intelligence HISTORY Beyond 1990s: Research areas converge Computational Intelligence Softcomputing Intelligent Systems Covering Adaptive Systems Fuzzy Systems Neural Networks Evolutionary Computing Data Mining CI IS SC AS FS DM EC NN SA PSO GA MCMC SL John Sum Computational Intelligence

COMPUTATIONAL INTELLIGENCE Heuristic algorithms (or models) such as in fuzzy systems, neural networks and evolutionary computation. Techniques that use Simulated annealing, Swarm intelligence, Fractals and Chaos Theory, Artificial immune systems, Wavelets, etc. John Sum Computational Intelligence

COMPUTATIONAL INTELLIGENCE Goal: Problem Solving Financial forecast Customer segmentation (CRM) Supply chain design (SCM) Business process re-engineering System control Pattern recognition Image compression Homeland security John Sum Computational Intelligence

COMPUTATIONAL INTELLIGENCE Underlying structure of the model is unknown, or the model is known but it is too complicated Example: DJI versus HIS (Time Series) Define system structure NL model (NN, ODE, etc.) Rule-based system Parametric estimation Deterministic search (Gradient descent or Newton’s method) Stochastic search (SA or MCMC) John Sum Computational Intelligence

COMPUTATIONAL INTELLIGENCE Underlying model structure is known Example: Manufacturing process (SCM) Define the objective to be maximized Examples: Completion time, Cost, Profit Optimization Linear programming, ILP, NLP Deterministic search (Gradient descent or Newton’s method) Stochastic search (SA or MCMC) John Sum Computational Intelligence

EG1: Nonlinear Dynamic System John Sum Computational Intelligence

EG2: Nonlinear Function John Sum Computational Intelligence

Computational Intelligence EG3: Car Price Predict the price of a car based on Specification of an auto in terms of various characteristics Assigned insurance risk rating Normalized losses in use as compared to other cars Number of attributes: 25 Missing values: Yes! John Sum Computational Intelligence

Computational Intelligence EG3: Car Price John Sum Computational Intelligence

EG4: Purchasing Preference Structural Equation Model Bayesian Network Feedforward Network Fuzzy Logic John Sum Computational Intelligence

EG5: Financial Time Series John Sum Computational Intelligence

EG5: Financial Time Series What would happen in the next trading day? (Time series prediction problem) Closing value Open value UP or DOWN Time series prediction + trading rules What should I do tomorrow? HOLD, SELL or BUY When should I BUY and SELL? John Sum Computational Intelligence

Remarks on EG1 ~ EG5 System Structure Data Types Model Dynamic System Unknown Continuous RNN, Fuzzy NN Nonlinear Function BPN, RBF, Fuzzy NN Car Price Discrete Purchasing Preference Known (SEM) SEM Bayesian Net Financial Time Series John Sum Computational Intelligence

COMPUTATIONAL INTELLIGENCE Statement of Problem Given a set of data collected (or measured) from a system (probably an unknown system), devise a model (by whatever structure, technique, method in CI) that mimics the behavior of that system as ‘good’ as possible. Making use of the devised model to (1) interpret the behavior of the system, (2) predict the future behavior of the system, (3) control the behavior of the system, (4) make money. John Sum Computational Intelligence

Computational Intelligence METHODOLOGY Step 1: Data Collection Experiments or measurements Questionnaire Magazine Public data sets Step 2: Model Structure Assumption IF it is known, SKIP this step. ELSE, DEFINE a model structure John Sum Computational Intelligence

Computational Intelligence METHODOLOGY Step 3: Parametric Estimation Gradient descent Newton’s method Exhaustive search Genetic algorithms (*) Evolutionary algorithms (*) Swarm intelligence Simulated annealing (*) Markov Chain Monte Carlo (*) John Sum Computational Intelligence

Computational Intelligence METHODOLOGY Step 4: Model Validation (is it a reasonable good model) Hypothesis test Validation/Testing set Leave one out validation Step 5: Model Reduction (would there be a simpler model that is also reasonable good) AIC, BIC, MDL Pruning (using testing set) John Sum Computational Intelligence

Computational Intelligence METHODOLOGY Beyond Model Reduction Any redundant input Any redundant sample (or outlier) Any better structure (alternative) How do we determine a ‘good’ model John Sum Computational Intelligence

Computational Intelligence NN MODEL STRUCTURES Perceptron Multilayer Perceptron (MLP or BPN) Adaptive Resonance Theory Model (ART) Competitive Learning (CL) Hopfield Network, Associative Network Bidirectional Associative Model (BAM) Recurrent Neural Network (RNN) Boltzmann Machine Brain-State-In-A-Box (BSB) Radial Basis Function Network (RBF Net) Bayesian Networks Self Organizing Map (SOM or Kohonen Map) Learning Vector Quantization (LVQ) Support Vector Machine (SVM) Support Vector Regression (SVR) PCA, ICA, MCA Winner-Take-All Network (WTA) Spike neural networks Remarks Not all of them is able to learn, eg BSB, WTA Might need to combine two structures to solve a single problem Multiple definitions on the ‘neuron’ John Sum Computational Intelligence

Computational Intelligence NN MODEL STRUCTURES Supply Chain Management (Optimization Problem) Hopfield Network Customer Segmentation (Clustering Problem) CL, SOM, LVQ, ART Dynamic Systems Modeling RNN, Recurrent RBF Car Price/NL Function (Function Approximation) MLP, RBF Net, Bayesian Net, SVR, +SOM/LVQ Financial TS (FA or Time Series Prediction) RNN, SVR, MLP, RBF Net, + SOM/LVQ John Sum Computational Intelligence

Computational Intelligence FUZZY MODEL STRUCTURE John Sum Computational Intelligence

Computational Intelligence FUZZY MODEL STRUCTURE John Sum Computational Intelligence

Computational Intelligence NN MODEL PARAMETERS MLP Input Weights Output Weights Neuron model RNN Recurrent Weights John Sum Computational Intelligence

Computational Intelligence NN MODEL PARAMETERS John Sum Computational Intelligence

Computational Intelligence NN MODEL PARAMETERS John Sum Computational Intelligence

Computational Intelligence NN MODEL PARAMETERS John Sum Computational Intelligence

FUZZY MODEL PARAMETERS John Sum Computational Intelligence

PARAMETRIC ESTIMATION John Sum Computational Intelligence

PARAMETRIC ESTIMATION Gradient Descent John Sum Computational Intelligence

PARAMERTIC ESTIMATION Genetic Algorithm John Sum Computational Intelligence

PARAMERTIC ESTIMATION Genetic Algorithm John Sum Computational Intelligence

PARAMERTIC ESTIMATION Genetic Algorithm John Sum Computational Intelligence

Computational Intelligence DISCUSSIONS CI is not the only method (or structure) to solve a problem. Even it can solve, its performance might not be better than other methods. Should compare with other well-known or existing methods John Sum Computational Intelligence

Computational Intelligence DISCUSSIONS SCM Problem LP, LIP, NLP Lagrangian Relaxation Cutting Plane CPLEX Function Approximation Polynomial Series Trigonometric Series B-Spline John Sum Computational Intelligence

Computational Intelligence CONCLUSIONS IF The problem to be solved has been well formulated The structure has been selected The objective function to evaluation the goodness of a parametric vector has been defined THEN Every problem is just an optimization problem John Sum Computational Intelligence

JOHN SUM (pfsum@nchu.edu.tw) Taiwan HK-Chinese, PhD (98) and MPhil (95) from CUHK, BEng (92) from PolyU HK. Taught in HK Baptist University (98-00), OUHK (00) and PolyU HK (00-04), Chung Shan Medical University (05-07) Adj. Associate Prof., Institute of Software, CAS Beijing (99-02) Short visit: CityU HK, Griffith University in Australia, FAU, Boca Raton FL US, CAS in Beijing, Ching Mai University in Thailand. Assist. Prof., IEC (07-09), Asso. Prof., ITM (09-) NCHU Taiwan 2000 Marquis Who's Who in the World. Senior Member of IEEE, CI Society, SMC Society (05-) GB Member, Asia Pacific Neural Network Assembly (09-) Associate Editor of the IJCA (05-09) Research Interests include NN, FS, SEM, EC, TM John Sum Computational Intelligence