KULIAH II JST: BASIC CONCEPTS Amer Sharif, S.Si, M.Kom

INTRODUCTION REVIEW Neural Network definition: A massively parallel distributed processor of simple processing units (neuron) Store experiential knowledge and make it available for use Knowledge is acquired from the environment through learning process Knowledge is stored as internerneuron connection strengths (synaptic weights)

INTRODUCTION REVIEW Benefits: Nonlinearity Input Output Mapping Adaptivity Evidential Response Contextual Information Fault Tolerance/Graceful Degrading VLSI Implementability Uniform Analysis & Design

NEURON MODELLING Basic elements of neuron: A set of synapses or connecting links Each synapse is characterized by its weight Signal xj at synapse j connected to neuron k is multiplied by synaptic weight wkj Bias is bk An adder for summing the input signals An activation function for limiting the output amplitude of the neuron

NEURON MODELLING Block diagram of a nonlinier neuron

NEURON MODELLING Note x1, x2,…, xm are input signals wk1, wk2,…, wkm are synaptic weights of neuron k uk is the linier combiner output bk is bias is the activation function yk is the output signal of the neuron

NEURON MODELLING If and bias is substituted for a synapse where x0 = + 1 with weight wk0 = bk then and

NEURON MODELLING Modified block diagram of a nonlinier neuron

ACTIVATION FUNCTIONS Activation Function types: Threshold Function and also known as the McCulloch-Pitts model -2 -1 0 1 2 v 1.2 1 0.8 0.6 0.4 0.2

ACTIVATION FUNCTIONS Piecewise-Linear Function

ACTIVATION FUNCTIONS Sigmoid Function S-shaped Sample logistic a is the slope parameter: the larger a the steeper the function Differentiable everywhere increasing a

NEURAL NETWORKS AS DIRECTED GRAPHS Neural networks maybe represented as directed graphs: Synaptic links (linier I/O) Activation links (nonlinier I/O) Synaptic convergence Synaptic divergence wkj xj yk= wkj xj xj yk=yi + yj yi yj xj

NEURAL NETWORKS AS DIRECTED GRAPHS Architectural graph: partially complete directed graph Output yk x0 =+1 xm x2 x1 .

FEEDBACK Output of a system influences some of the input applied to the system One or more closed paths of signal transmission around the system Feedback plays an important role in recurrent networks

FEEDBACK x’j (n) yk(n) xj(n) Sample single-loop feedback system w z-1 Output signal yk(n) is an infinite weighted summation of present and past samples of input signal xj(n) x’j (n) xj(n) yk(n) w z-1 w is fixed weight z-1 is unit-delay operator is sample of input signal delayed by l time units

FEEDBACK Dynamic system behavior is determined by weight w w < 1 yk(n) wxj(0) 0 1 2 3 4 n Dynamic system behavior is determined by weight w w < 1 w < 1 System is exponentially convergent/stable System posses infinite memory: Output depends on input samples extending into the infinite past Memory is fading: influence of past samples is reduced exponentially with time n

FEEDBACK w = 1 w > 1 System is linearly divergent yk(n) wxj(0) 0 1 2 3 4 n w = 1 w = 1 System is linearly divergent w > 1 System is exponentially divergent yk(n) wxj(0) 0 1 2 3 4 n w > 1

NETWORK ARCHITECTURES Single-Layered Feedforward Networks input layer of source nodes output layer of neurons Neurons are organized in layers “Single-layer” refers to output neurons Source nodes supply to output neurons but not vice versa Network is feedforward or acyclic

NETWORK ARCHITECTURES Multilayer Feedforward Networks Input layer of source nodes Layer of hidden neurons Layer of output One or more hidden layers Hidden neurons enable extractions of higher-order statistic Network acquires global perspective due to extra set of synaptic connections and neural interactions 7-4-2 fully connected network: 7 source nodes 4 hidden neurons 2 output neurons

NETWORK ARCHITECTURES Recurrent Networks z-1 Unit-delay operators Inputs Outputs At least one feedback loop Feedback loops affect learning capability and performance of the network

KNOWLEDGE REPRESENTATION Definition of Knowledge: Knowledge refers to stored information or models used by a person or a machine to interpret, predict, and appropriately respond to the outside world Issues: What information is actually made explicit How information is physically encoded for subsequent use Knowledge representation is goal-directed Good solution depends on good representation of knowledge

KNOWLEDGE REPRESENTATION Challenges faced by Neural Networks: Learn the model of the world/environment Maintain the model to be consistent with the real world to achieve the goals desired Neural Networks may learn from a set of observations data in form of input-output pairs (training data/training sample) Input is input signal and output is the corresponding desired response

KNOWLEDGE REPRESENTATION Handwritten digit recognition problem Input signal: one of 10 images of digits Goal: to identify image presented to the network as input Design steps: Select the appropriate architecture Train the network with subset of examples (learning phase) Test the network with presentation of data/digit image not seen before, then compare response of network with actual identity of the digit image presented (generalization phase)

KNOWLEDGE REPRESENTATION Difference with classical pattern-classifier: Classical pattern-classifier design steps: Formulate mathematical model of the problem Validate model with real data Build based on model Neural Network design is: Based on real life data Data may “speak for itself” Neural network not only provides model of the environment but also process the information

ARTIFICIAL INTELLIGENCE AND NEURAL NETWORKS AI systems must be able to: Store knowledge Use stored knowledge to solve problem Acquire new knowledge through experience AI components: Representation Knowledge is presented in a language of symbolic structures Symbolic representation makes it relatively easy for human users

ARTIFICIAL INTELLIGENCE AND NEURAL NETWORKS Reasoning Able to express and solve broad range of problems Able to make explicit and implicit information known to it Have a control mechanism to determine which operation for a particular problem, when a solution is obtained, or when further work on the problem must be terminated Rules, Data, and Control: Rules operate on Data Control operate on Rules The Travelling Salesman Problem: Data: possible tours and cost Rules: ways to go from city to city Control: which Rules to apply and when

ARTIFICIAL INTELLIGENCE AND NEURAL NETWORKS Learning Inductive learning: determine rules from raw data and experience Deductive learning: use rules to determine specific facts Environment Learning element Knowlegdge Base Performance element

ARTIFICIAL INTELLIGENCE AND NEURAL NETWORKS Parameter Artificial Intelligence Neural Networks Level of Explanation Symbolic representation with sequential processing Parallel distributed processing (PDP) Processing Style Sequential Parallel Representational Structure Quasi-linguistic structure Poor Summary Formal manipulation of algorithm and data representation in top down fashion Parallel distributed processing with natural ability to learn in bottom up fashion