1. CS621 : Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 20: Neural Net Basics: Perceptron

2. Turing Machine & Von Neumann Machine

3. Challenges to Symbolic AI Motivation for challenging Symbolic AI A large number of computations and information process tasks that living beings are comfortable with, are not performed well by computers! The Differences Brain computation in living beings TM computation in computers Pattern Recognition Numerical Processing Learning oriented Programming oriented Distributed & parallel processing Centralized & serial processing Content addressable Location addressable

4. Perceptron

5. The Perceptron Model A perceptron is a computing element with input lines having associated weights and the cell having a threshold value. The perceptron model is motivated by the biological neuron. Output = y wnwn W n-1 w1w1 X n-1 x1x1 Threshold = θ

6. θ 1 y Step function / Threshold function y = 1 for Σw i x i >=θ =0 otherwise ΣwixiΣwixi

7. Features of Perceptron Input output behavior is discontinuous and the derivative does not exist at Σw i x i = θ Σw i x i - θ is the net input denoted as net Referred to as a linear threshold element - linearity because of x appearing with power 1 y= f(net): Relation between y and net is non-linear

8. Computation of Boolean functions AND of 2 inputs X1 x2 y 0 0 0 0 10 1 00 1 11 The parameter values (weights & thresholds) need to be found. y w1w1 w2w2 x1x1 x2x2 θ

9. Computing parameter values w1 * 0 + w2 * 0 = 0; since y=0 w1 * 0 + w2 * 1 <= θ  w2 <= θ; since y=0 w1 * 1 + w2 * 0 <= θ  w1 <= θ; since y=0 w1 * 1 + w2 *1 > θ  w1 + w2 > θ; since y=1 w1 = w2 = = 0.5 satisfy these inequalities and find parameters to be used for computing AND function.

10. Other Boolean functions OR can be computed using values of w1 = w2 = 1 and = 0.5 XOR function gives rise to the following inequalities: w1 * 0 + w2 * 0 = 0 w1 * 0 + w2 * 1 > θ  w2 > θ w1 * 1 + w2 * 0 > θ  w1 > θ w1 * 1 + w2 *1 <= θ  w1 + w2 <= θ No set of parameter values satisfy these inequalities.

11. Threshold functions n # Boolean functions (2^2^n) #Threshold Functions (2 n^2 ) 1 4 4 2 16 14 3 256 128 4 64K 1008 Functions computable by perceptrons - threshold functions #TF becomes negligibly small for larger values of #BF. For n=2, all functions except XOR and XNOR are computable.

12. Computational Capacity of Perceptrons

13. Separating plane ∑ w i x i = θ defines a linear surface in the (W,θ) space, where W= is an n- dimensional vector. A point in this (W,θ) space defines a perceptron. y x1x1... θ w1w1 w2w2 w3w3 wnwn x2x2 x3x3 xnxn

14. The Simplest Perceptron θ y x1x1 w1w1 Depending on different values of w and θ, four different functions are possible

15. Simplest perceptron contd. 10101 11000 f4f3f2f1x θ≥0 w≤0 θ≥0 w>0 θ<0 w≤0 θ<0 W<0 0-function Identity Function Complement Function True-Function

16. Counting the #functions for the simplest perceptron For the simplest perceptron, the equation is w.x=θ. Substituting x=0 and x=1, we get θ=0 and w=θ. These two lines intersect to form four regions, which correspond to the four functions. θ=0 w=θ R1 R2 R3 R4

17. Fundamental Observation The number of TFs computable by a perceptron is equal to the number of regions produced by 2 n hyper- planes,obtained by plugging in the values in the equation ∑ i=1 n w i x i = θ Intuition: How many lines are produced by the existing planes on the new plane? How many regions are produced on the new plane by these lines?

18. The geometrical observation Problem: m linear surfaces called hyper-planes (each hyper-plane is of (d-1)-dim) in d-dim, then what is the max. no. of regions produced by their intersection? i.e. R m,d = ?

19. Concept forming examples Max #regions formed by m lines in 2-dim isR m,2 = R m-1,2 + ? The new line intersects m-1 lines at m-1 points and forms m new regions. R m,2 = R m-1,2 + m, R 1,2 = 2 Max #regions formed by m planes in 3 dimensions is R m,3 = R m-1,3 + R m-1,2, R 1,3 = 2

20. Concept forming examples contd.. Max #regions formed by m planes in 4 dimensions is R m,4 = R m-1,4 + R m-1,3, R 1,4 = 2 R m,d = R m-1,d + R m-1,d-1 Subject to R 1,d = 2 R m,1 = 2

21. General Equation R m,d = R m-1,d + R m-1,d-1 Subject to R 1,d = 2 R m,1 = 2 All the hyperplanes pass through the origin.

22. Method of Observation for lines in 2-D R m,2 = R m-1,2 + m R m-1,2 = R m-2,2 + m-1 R m-2,2 = R m-3,2 + m-2 : R 2,2 = R 1,2 + 2 Therefore, R m,2 = R m-1,2 + m = 2 + m + (m-1) + (m-2) + …+ 2 = 1 + ( 1 + 2 + 3 + … + m) = 1 + [m(m+1)]/2

23. Method of generating function R m,2 = R m-1,2 + m f(x) = R 1,2 x + R 2,2 x 2 + R 3,2 x 3 + … + R i,2 x m + … + α –>Eq1 xf(x) = R 1,2 x 2 + R 2,2 x 3 + R 3,2 x 4 + … + R i,2 x m+1 + … + α –>Eq2 Observe that R m,2 - R m-1,2 = m

24. Method of generating functions cont… Eq1 – Eq2 gives (1-x)f(x) = R 1,2 x + (R 2,2 - R 1,2 )x 2 + (R 3,2 - R 2,2 )x 3 + … + (R m,2 - R m-1,2 )x m + … + α (1-x)f(x) = R 1,2 x + (2x 2 + 3x 3 + …+mx m +..) = 2x 2 + 3x 3 + …+mx m +.. f(x) = (2x 2 + 3x 3 + …+mx m +..)(1-x) -1

25. Method of generating functions cont… f(x) = (2x 2 + 3x 3 + …+mx m +..)(1+x+x 2 +x 3 …)  Eq3 Coeff of x m is R m,2 = (2 + 2 + 3 + 4 + …+m) = 1+[m(m+1)/2]

26. The general problem of m hyperplanes in d dimensional space c(m,d) = c(m-1,d) + c(m-1,d-1) subject to c(m,1)= 2 c(1,d)= 2

27. Generating function f(x,y) = R 1,1 xy + R 1,2 xy 2 + R 1,3 xy 3 + … + R 2,1 x 2 y + R 2,2 x 2 y 2 + R 2,3 x 2 y 3 +... + R 3,1 x 3 y + R 3,2 x 3 y 2 + ……… f(x,y) = ∑ m≥1 ∑ n≥1 R m,d x m y d

28. # of regions formed by m hyperplanes passing through origin in the d dimensional space c(m,d)= 2.Σ d-1 i=0 m-1 c i