Seoul National University Neural Network Modeling for Intelligent Novelty Detection 제 2 차 뇌신경정보학 Workshop 일시 : 2002 년 2 월 27 일 ( 수 ) 10:00-18:00 장소 : KAIST LG Semicon Hall 1 층 강당

제 2 차 뇌신경 정보학 Workshop Introduction Concept of Novelty Detection – Typical Classification Class A Class B Classifier A New Instance Class A or Class B Training Classification Class A Novelty Detector A New Instance Class A or NOT Training Classification – Novelty Detection

제 2 차 뇌신경 정보학 Workshop Introduction Applications of Novelty Detection – Authentication of computer system – Detection of counterfeit – Detection of phase transformation in a financial market – Fault detection in a mechanical system Algorithms for Novelty Detection – Principal Component Analysis (PCA) – Auto-Associative Multi-Layer Perceptron (AAMLP) – Probability Density Estimator : SOM, Mixture of Kernels, etc.

제 2 차 뇌신경 정보학 Workshop Characteristics of 2-Layer AAMLP Structure – – Auto-Association  Input vectors = Target vectors  Normal patterns  Smaller errors Novel patterns  Larger errors – Training Algorithms  Same as a typical MLP  Back-propagation,  Gradient-descent,  Levenberg-Marquadt, etc. Hidden Layer Output Layer nodesMLP with

제 2 차 뇌신경 정보학 Workshop Characteristics of 2-Layer AAMLP Properties (Lee, Hwang, Cho; submitted) – A 2-layer AAMLP defines an output-constrained hyperplane – The hyperplane is bounded – The hyperplane lies within the training input vector area – It is trained so that the association error, the sum of distances between the hyperplane and the input vectors, is minimized

제 2 차 뇌신경 정보학 Workshop Experiments with 2-Layer AAMLP Comparison with 2-Layer AAMLP (Linear Ftn.) (=PCA) – AAMLP with Linear Ftn. – AAMLP with Linear Ftn.

제 2 차 뇌신경 정보학 Workshop Experiments with 2-Layer AAMLP Comparison with 2-Layer AAMLP (Linear Ftn.) (=PCA) – AAMLP with Linear Ftn.

제 2 차 뇌신경 정보학 Workshop Experiments with 2-Layer AAMLP 2-Layer AAMLP with Saturated Linear Ftn. Equivalent to 2-layer AAMLP with sigmoid ftn.

제 2 차 뇌신경 정보학 Workshop Experiments with 2-Layer AAMLP Limitations of 2-Layer AAMLP Inability to model a non-linear data

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP 4-Layer AAMLP (Kramer, 1991) – – Non-Linear PCA – Limitations of NLPCA (Malthouse, 1998)  Good at interpolation  But not at extrapolation  A desirable property as a novelty detector Mapping Layer De-Mapping Layer Bottleneck Layer Output Layer

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Novelty Detection in A Non-Linear Data – 4-Layer AAMLP with Sigmoid Ftn. Ability to model a non-linear data

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Novelty Detection in A Non-Linear Data – 4-Layer AAMLP with Saturated Linear Ftn. Ability to model a non-linear data

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Novelty Detection in A Non-Linear Data – Similarity of Sigmoid and Saturated Linear Function – AAMLP with Sigmoid and Saturated Linear Function Assumption AAMLP-SatLinis similar to 2-layer AAMLP in the output characteristics &is similar to 4-layer AAMLP in the shape of output vectors Hypothesis AAMLP-Sigmoiddefines an output-constrained hypersurface &can minimizes the association error, at least, locally

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Novelty Detection in A Multi-Modal Data – 4-Layer AAMLP with Sigmoid Ftn. Inability to model multi-modal data

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Novelty Detection in A Multi-Modal Data – Clustering + 4-Layer AAMLP with Sigmoid Ftn. Ability to model a multi-modal data

제 2 차 뇌신경 정보학 Workshop Characteristics of 4-Layer AAMLP Properties – A 4-layer AAMLP with sigmoid ftn. defines a hypersurface – A one with saturated linear ftn. defines a set of segments – A one with saturated linear ftn. can model a non-linear data in a way similar to that of 2-layer AAMLP – It can be argued that a one with sigmoid ftn. can model a non-linear data & minimizes the association error – A one with sigmoid ftn. cannot model a multi-modal data – A one with sigmoid ftn. can model a multi-modal data if it is preceded by proper clustering algorithms

제 2 차 뇌신경 정보학 Workshop Conclusion – A 2-layer AAMLP can be a novelty detector for a linear data – A 4-layer one can be a novelty detector for some non-linear data, which a 4-layer one cannot – A 4-layer one can be a novelty detector for a multi-modal data, if the data is pre-clustered

제 2 차 뇌신경 정보학 Workshop Future Works – Probability Density Estimators : SOM, Mixture of Kernels, etc.  Comparison of AAMLP and the density estimators in performances  Sophisticated clustering methods using the density estimators – AAMLP Ensemble  The unstableness of AAMLP  An AAMLP ensemble may overcome  Bagging & Majority voting