Instructor: Prof. Pushpak Bhattacharyya 13/08/2004 CS-621/CS-449 Lecture Notes CS621/CS449 Artificial Intelligence Lecture Notes Set 6: 22/09/2004, 24/09/2004, 1/10/2004, 6/10/2004

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Contents Complexity of Feedforward Network (FFN) training Positive Linear Confinement (PLC) problem Computational complexity basics PLC is NP-Complete

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Complexity of Feedforward Network (FFN) Training Observation –Almost all problems that use FFN with BP require a very large amount of time. Rivest and Blum (1992) –Training of 3-layer FFN is NP-complete

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Decision Problem for FFN Training Given a set of labeled patterns, does there exist two hyperplanes such that patterns of only one kind are confined to a quadrant? In other words, Can a 3-node NN be trained on the given positive and negative examples? Does there exist two hyperplanes defined by h 0 and h 1 which confine ALL and ONLY points of one kind? (Linear Confinement Problem)

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Positive Linear Confinement(PLC) problem The PLC decision problem is computationally hard i.e. as the number of points grows, the time taken to find the planes grows exponentially. Equivalent statement: –Training of a 3-node NN will take time

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Computational Complexity Basics Time Complexity –The number of steps taken by a Turing Machine (TM) to complete the computation as a function of the length of the input Infinite tape Finite state head TURING MACHINE

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Computational Complexity Basics Space Complexity –The number of cells on the tape needed to complete the computation, as a function of the length of the input Kolmogorov Complexity –The length of the shortest program (as a function of input length) to express the computation

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Complexity Expressions for AI Problems Most problems in AI are pattern recognition problems AI problems involve enumerating large number of cases Numerical problems are described in terms of space and time complexity, whereas, AI problems are described using Kolmogorov complexity

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Complexity Classes Class P –It is the class of decision problems which can be computed in polynomial time by a deterministic TM Class NP –It is the class of decision problems which can be computed in polynomial time by a non- deterministic TM –For this class of problems, once the solution is given, it is easy to verify it

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Complexity Classes Since Deterministic TMs are a subset of Non- deterministic TMs, Complexity Class: –This is a class of problems which share the complexity measure –There exists “transformations” of lesser complexity among the problems

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Complexity Classes Completeness within a complexity class –If one of the problems within the class is solved in less time than indicated by the complexity, all of them can be solved in that amount of ‘less’ time. –This happens because of the existence of transformations of less complexity among the problems

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Transformation Procedure A known Complexity problem is chosen –The transformation procedure should take less time than the inherent complexity of the known complexity problem Arbitrary instance of “known complexity” problem Transformation procedure Constructed instance of problem whose complexity is desired

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Set Splitting Problem Here, we introduce Set Splitting problem which is later used in proving that PLC is NP-Complete Given a set S= {s 1,s 2,s 3,…s n } and subsets C i of S such that, can S be split into two subsets S 1 and S 2 such that and

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes PLC is NP-Complete 1.PLC belongs to NP Given two hyperplanes, one can check easily (in polynomial time) if they confine all and only the positive points 2.Transformation: Transformation of ‘Set Splitting’ problem to PLC in polynomial time Corresponding to n elements in S, set up a coordinate system with n axes S = {s 1,s 2,s 3,…s n } xnxn X n-1 X n-2 X n-3 X1X1

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes PLC is NP-Complete Transformation of SS problem instance to an instance of PLC: 1.Mark origin as positive 2.Mark unit-distance points on the axes as negative Ex: S={s 1,s 2,s 3 } C1={s 1, s 3 } C2={s 2 } + (0,0,0) - (1,0,0) - (0,0,1) - (0,1,0)

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes PLC is NP-Complete For each c i, mark as positive, where + (0,0,0) - (1,0,0) + (1,1,0) + (0,1,1) - (0,0,1) - (0,1,0)

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes PLC is NP-Complete The transformation implies –If part: If the set S can be split, two hyperplanes can be found which can confine all and only positive points –Then part: If two hyperplanes can confine ALL and only positive points, S can be split

Prof. Pushpak Bhattacharyya IIT Bombay 22/09/ /10/2004 CS-621/CS-449 Lecture Notes Example for SS to PLC transformation For the previous set-splitting example, one possible solution for the corresponding instance of PLC is P1: -x 1 +3x 2 -x 3 = -1/2 P2: 3x 1 -x 2 +3x 3 =-1/2 (see that all positive points are on the same side as origin)