1. CPSC 5307 Search and Discrete Optimization Good results with problems that are too big for people or computers to solve completely http://mathworld.wolfram.com/TravelingSalesmanProblem.html

2. Course outline  textbook – Michalewicz and Fogel (reasonable price, valuable book) ppt  lectures, notes and ppt presentations  evaluation assignments5@10% 50% project10% + 15% + 25% 50% (documents plus presentations)

3. Difficult computing problems  hard to represent what information, what data structures e.g., image organization  no known algorithms e.g., face recognition  no known efficient algorithms e.g., travelling salesperson This course: discrete variable problems

4. Good results with problems too big to solve completely  cannot formally optimize results cannot be proven best usually results are not best  how to evaluate results? statistical probability  “within 4%, 19 times out of 20” satisficing  surpassing a threshhold that is “good enough”

5. Examples  practical examples scheduling (transportation, timetables,…)  puzzles crosswords, Sudoku, n Queens  classic examples SAT: propositional satisfiability problem (independent parameters) CSP: constraint satisfaction problem (dependent parameters) TSP: travelling salesman problem (permutations)

6. SAT: propositional satisfiability problem P1P1 P2P2 P 1 ^P 2 FFF FTT TFF TTT n propositions, P 1, P 2, P 3, …, P n What combination of truth values makes a sentence true? Table has 2 n rows. n=50, 2 50 = 1,125,899,906,842,624 n=2; 2 2 = 4 rows

7. CSP: constraint satisfaction problem  example – map colouring n countries – 4 possible colours -constraints: adjacent countries different colours -4 n combinations n=13; 4 13 = 67,108,864 combinations; 25 constraints

8. TSP: traveling salesman (sic) problem  n cities: what is shortest path visiting all cities, C 1, C 2, C 3, …, C n once?  (n-1)! routes from home city on complete graph n = 16; (n-1)! = 1,307,674,368,000 C1C1 n = 5; (n-1)! = 24

9. Simple Example – one variable  maximize a function, f(n) over a range n min ≤ n ≤ n max  f (n) is linear f(n) is monotonic  f (n) is quadratic with f’’(n) < 0 f(n) is convex upward  f (n) is differentiable function f(n) has multiple (local) maxima

11. Simple Example – one variable  maximize a function, f(n) over a range n min ≤ n ≤ n max  f (n) is linear f(n) is monotonic  f (n) is quadratic with f’’(n) < 0 f(n) is convex upward  f (n) is differentiable function f(n) has multiple (local) maxima

13. Simple Example – one variable  maximize a function, f(n) over a range n min ≤ n ≤ n max  f (n) is linear f(n) is monotonic  f (n) is quadratic with f’’(n) < 0 f(n) is convex upward  f (n) is differentiable function f(n) has multiple (local) maxima

15. Problem description 1.fitness function (optimization function, evaluation) – e.g., m = n 3 mod 101 2.constraints (conditions) – e.g., n is odd find global optimum of fitness function without violating constraints OR getting stuck at local optimum  small space: complete search  large space: ?????

16. Large problems  more possible values  more parameters, n = {n 1, n 2, n 3, …}  more constraints  more complex fitness functions - takes significant time to calculate m = f(n)  too big for exhaustive search

17. Searching without searching everywhere How to search intelligently/efficiently using information in the problem: -hill climbing -simulated annealing -constraint satisfaction -A* -genetic algorithms - …

18. Focusing search  assumption – some pattern to the distribution of the fitness function (otherwise: random distribution of fitness) finding the height of land in a forest - can only see ‘local’ structure - easy to find a hilltop but are there other higher hills?

19. Fitness function distribution  convex – easy – start anywhere, make local decisions

20. Fitness function distribution  many local maxima (conve in local region) make local decisions but don’t get trapped

21. Performance – O ( f(n) ) What is the problem?

22. Quality software  Correctness  Performance == Efficiency Space – how much memory must be allocated? Time – how much time elapses during execution?

23. Measuring performance  Testing  Analysis of algorithms How much space / time is required for an algorithm? How does one algorithm compare to another? How do requirements change depending on input?  Need a scale for describing performance

24. Measuring performance  “Big Oh” notation O ( f(n) )  Efficiency of performance expressed as an approximate function of the number of data elements processed

25. Three simple methods public void s0(int n) { int val = n; } public void s1(int n) { int[] val = new int[n]; } public void s2(int n) { int[][] val = new int[n][n]; } n = 6 Memory space space

26. We will mainly consider TIME efficiency

27. Three simple methods public void m0(int n) { System.out.println(n); } public void m1(int n) {for(int i = 0; i<n; i++) System.out.println(n); } public void m2(int n) {for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n);} n = 6 # of executions 1 6 36 time

28. Three simple methods public void m0(int n) { System.out.println(n); } public void m0(int n) { System.out.println(n); } public void m1(int n) { for(int i = 0; i<n; i++) System.out.println(n); } public void m1(int n) { for(int i = 0; i<n; i++) System.out.println(n); } n time n n public void m2(int n) { for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n); } public void m2(int n) { for(int i = 0; i<n; i++) for(int j = 0; j<n; j++) System.out.println(n); } O(1) O(n)O(n) O(n2)O(n2)

29. Approximate measure public void m2(int n) { for(int i = 0; i<n; i++) for(int j = 0 ; j<n; j++) System.out.println(n); } public void m2a(int n) { for(int i = 0; i<n; i++) for(int j = i ; j<n; j++) System.out.println(n); } Better performance? NO Better performance? NO

30. Approximate measure 12610100 # exe c 143610010000 # exe c 1321555050 public void m2(int n) { for(int i = 0; i<n; i++) for(int j = 0 ; j<n; j++) System.out.println(n); } public void m2a(int n) { for(int i = 0; i<n; i++) for(int j = i ; j<n; j++) System.out.println(n); } n n 2 O(n 2 ) n 2 +n 2 O(n 2 )

31. Approximate measure // linear search (unsorted array) public int search(double n, double[] arr) { int i = 0; while (i < arr.length && arr[i] != n) i++; return i; }  From 1 to n iterations: Failed search: n Average for successful search: n/2  Performance O(n)

32. BUT Different measure // binary search (sorted array) public int search(double n, double[] arr) {int i, min = 0, max = arr.length-1; while (min <= max) { i = (min + max ) / 2; if (arr[i] == n) return i; if (arr[i] < n) min = i+1; else max = i-1; } return arr.length; }  From 1 to log n iterations: Failed search: log n Average for successful search: log n - 1  Performance O(log n)

33. O(log n) performance

34.  examples: binary search in sorted arrays binary search tree operations retrieve, update, insert, delete B-trees heaps insert, remove

35. Sorting algorithms  two nested repeat operations both O(n)  O(n 2 )  bubble, insertion, selection,… one O(n) and one O(log n)  O(n log n)  mergesort, quicksort, heapsort,… Shellsort: ??, O(n (log n) 2 ), O(n 1.25 )

36. Analyzing an algorithm  rule of thumb: depth of nested structure (or recursion)  number of factors in performance measure  some algorithms are difficult to analyze (e.g. hash table processing)

37. Comparing ‘polymomial’ performance n factors O(1) 0 O(log n) 1 O(n) 1 O(n log n) 2 O(n 2 ) 2 O(n 3 ) 3 2112248 81382464512 321564320102432768 12817 896163842097152 10241101024819210485761073741824

38. Combinatorial problems Example: list the possible orderings of S: {1,2,3}: 123, 132, 213, 231, 312, 321 |S| 2345n orderings 2624120n!

39. Combinatorial problems Example: generate truth table for P^Q: PQP^Q TTT TFF FTF FFF number of propositions2345n rows in table 4816322 n

40. Combinatorial problems  only known solutions have n levels of nesting for n data values  performance is exponential: O(e n ) includes n!, 2 n, 10 n,…  this is our kind of problem!

41. Comparing polymomial (P) and exponential (NP * ) performance n factors O(log n) 1 O(n) 1 O(n log n) 2 O(n 3 ) 3 O(e n ), e.g. 2 n n 212284 83824512256 32564320327684294967296 1287 89620971523.4028 x 10 38 1024101024819210737418241.7977 x 10 308 * N on-deterministic P olynomial

42. NP – Non-deterministic polynomial  if a solution is known, it can be verified in polynomial O(n k ) time e.g., SAT a solution (T,T,F,T,F,…,T,F,T,F,F,F) can be tested by substituting in the fitness function.

43. P and NP problems P = NP? (Are there polynomial algorithms for combinatorial problems?) YES  there are but we are too stupid to find them NO  we need to find other ways to solve these problems (in 50+ years of trying, no answer to the question)

44. NP-hard P and NP problems NP P P complete NP-complete

45. Representing problems Abstraction from the messy real world to the ordered simplicity of the model

46. Abstraction  Representing the problem for processing What information is relevant? How should it be operationalized?  Data  Relations  Processes What information is irrelevant?  The puzzles

47. My real world problem I have a list of 100 books to order online. Some are available in hardcover (H) or pocket (P) versions and some are available used(U)? How do I minimize my cost for the books?

48. Variables  set of n controllable parameters V = {x 1, x 2, x 3, …, x n } each variable x i has a set of possible values, domain D i e.g., V = { x 1, x 2,..., x 100 }, 100 books to order x 1 ∈ D 1 = {H,P,U} x 2 ∈ D 2 = {H,P,U}... x 100 ∈ D 100 = {H,P,U}

49. Problem Space  set of all possible combinations of variable values  dimension of space n: number of variables  size of space: |D 1 | x |D 2 | x … x |D n | e.g., book order size = 3 x 3 x... x 3 = 3 100 sample point {H, H, U,..., S}

50. Fitness  the objective function  problem outcome as a function of the variables: f(x 1, x 2, x 3, …, x n )  goal: optimize (maximize or minimize) f e.g., total cost of purchase f(x 1, x 2, x 3, …, x 100 ) minimize f

51. Constraints  rules C(V) that eliminate some points in the problem space from consideration e.g., some books not available in all versions

52. Abstraction - “Operationalizing”  representation of fitness and constraints for evaluation and search e.g. x i = v i = (type, cost, weight) i fitness f=cost of books plus cost of shipping All new books (H or S) are shipped in one order at $5 per kilo of total weight but the shipping cost is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book. How to handle constraints?

53. Abstraction - “Operationalizing” e.g. x i = vi =(type, cost, weight) i fitness f=cost of books plus cost of shipping All new books (H or S) are shipped in one order at $5 per kilo of total weight but the shipping rate is waived if the total cost of the new books is over $100. Used books are shipped in a separate order with shipping cost of $3 per book. How to handle constraints? cost = ∑ v i.cost// book cost + 3 x ∑ (v i.type == U)// ship used + [(∑ (v i.type<>U).v i.cost) < 100] // ship new [5 x ∑ (v i.type<>U).v i.weight]

54. Exhaustive search bestFitness = MaxVal (assume minimize) bestV for all x 1 ∈ D 1, x 2 ∈ D 2, x 3 ∈ D 3,…x n ∈ D n if (x 1, x 2, x 3,…x n satisfy constraints C(V)) fitness = f(x 1, x 2, x 3,…x n ) if (fitness < bestFitness) bestFitness = fitness bestV = {x 1, x 2, x 3,…x n } return bestFitness, bestV

55. Exhaustive search example bestFitness = MaxVal bestV for all x 1 ∈ D 1, x 2 ∈ D 2, x 3 ∈ D 3,…x 100 ∈ D 100 if (x 1, x 2, x 3,…x 100 versions all exist) cost = f(x 1, x 2, x 3,…x 100 ) if (cost < bestCost) bestCost = cost bestV = {x 1, x 2, x 3,…x 100 } return bestCost, bestV

56. Summary  Parameters, dimension, solution space  Objective - fitness or evaluation function  Constraints - impossible points in solution space  Finding point in space with optimal fitness No algorithm to calculate point Space too large to search --> optimization methods with tradeoffs

57. Puzzles are Us SEND +MORE MONEY Each letter represents a different digit No leading zero’s Sum is correct Find assignment of digits to letters