Backtracking Algorithm Level II, Term ii CSE – 243 Md. Monjur-ul-hasan Lecturer Dept of cse, cuet Email: mail@monjur-ul-hasan.info

An Example Find out all 3-bit binary numbers for which the sum of the 1's is greater than or equal to 2. The only way to solve this problem is to check all the possibilities: (000, 001, 010, ....,111) The 8 possibilities are called the search space of the problem. They can be organized into a tree. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

An Example (cont…) 0 0 _ 0 0 0 0 0 1 0 1 _ 0 1 0 0 1 1 1 _ _ 0 _ _ 1 0 _ 1 0 0 1 0 1 1 1 _ 1 1 0 1 1 1 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Backtracking: Definition Start Success! Success! Failure For some problems, the only way to solve is to check all possibilities. Backtracking is a systematic way to go through all the possible configurations of a search space. Problem space consists of states (nodes) and actions (paths that lead to new states). When in a node can only see paths to connected nodes If a node only leads to failure go back to its "parent“ node. Try other alternatives. If these all lead to failure then more backtracking may be necessary. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Goals of Backtracking Possible goals Find a path to success Find all paths to success Find the best path to success Not all problems are exactly alike, and finding one success node may not be the end of the search Start Success! Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Backtracking and Recursive Backtracking is easily implemented with recursion because: The run-time stack takes care of keeping track of the choices that got us to a given point. Upon failure we can get to the previous choice simply by returning a failure code from the recursive call. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Improving Backtracking: Search Pruning Search pruning will help us to reduce the search space and hence get a solution faster. The idea is to a void those paths that may not lead to a solutions as early as possible by finding contradictions so that we can backtrack immediately without the need to build a hopeless solution vector. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

A More Concrete Example: Suduku 9 by 9 matrix with some numbers filled in all numbers must be between 1 and 9 Goal: Each row, each column, and each mini matrix must contain the numbers between 1 and 9 once each no duplicates in rows, columns, or mini matrices 8 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Suduku with Brut force A brute force algorithm is a simple but general approach Try all combinations until you find one that works This approach isn’t clever, but computers are fast Then try and improve on the brute force resuts Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Sodoku (Cont…) Brute force Sudoku Soluton After placing a number in a cell is the remaining problem very similar to the original problem? 1 Brute force Sudoku Soluton if not open cells, solved scan cells from left to right, top to bottom for first open cell When an open cell is found start cycling through digits 1 to 9. When a digit is placed check that the set up is legal now solve the board Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solving Sudoku – Later Steps 1 2 4 8 9 uh oh! Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solving Sudoku – Dead End We have reached a dead end in our search With the current set up none of the nine digits work in the top right corner 1 2 4 8 9 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solving Suduku - Backing Up 1 2 4 8 9 When the search reaches a dead end in backs up to the previous cell it was trying to fill and goes onto to the next digit We would back up to the cell with a 9 and that turns out to be a dead end as well so we back up again so the algorithm needs to remember what digit to try next Now in the cell with the 8. We try and 9 and move forward again. 1 2 4 9 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Problem with Suduku Brute force algorithms are slow The don't employ a lot of logic For example we know a 6 can't go in the last 3 columns of the first row, but the brute force algorithm will plow ahead any way But, brute force algorithms are fairly easy to implement as a first pass solution backtracking is a form of a brute force algorithm Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solving Suduku-Search pruning After trying placing a digit in a cell we want to solve the new sudoku board Isn't that a smaller (or simpler version) of the same problem we started with?!?!?!? After placing a number in a cell the we need to remember the next number to try in case things don't work out. We need to know if things worked out (found a solution) or they didn't, and if they didn't try the next number If we try all numbers and none of them work in our cell we need to report back that things didn't work Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solving Suduku-Recursive Backtracking Problems such as Suduko can be solved using recursive backtracking recursive because later versions of the problem are just slightly simpler versions of the original backtracking because we may have to try different alternatives Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

General Back Tracking Algorithm void checknode (node v) { if (promising ( v )) if (aSolutionAt( v )) write the solution else //expand the node for ( each child u of v ) checknode ( u ) } Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

General Back Tracking Algorithm (Cont…) Checknode uses the functions: promising(v) which checks that the partial solution represented by v can lead to the required solution aSolutionAt(v) which checks whether the partial solution represented by node v solves the problem. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Eight Queen Problem: A Classic Puzzle Attempts to place 8 queens on a chessboard in such a way that no queen can attack any other. A queen can attack another queen if it exists in the same row, columm or diagonal as the queen. This problem can be solved by trying to place the first queen, then the second queen so that it cannot attack the first, and then the third so that it is not conflicting with previously placed queens. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Problem (N=8) Place N Queens on an N by N chessboard so that none of them can attack each other Number of possible placements? In 8 x 8 64 * 63 * 62 * 61 * 60 * 58 * 59 * 57 = 4,426,165,368 roughly four and a half billion possible set ups n choose k How many ways can you choose k things from a set of n items? In this case there are 64 squares and we want to choose 8 of them to put queens on Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Solution: N Queen Puzzle We know that for queens: each row will have exactly one queen each column will have exactly one queen each diagonal will have at most one queen The previous calculation includes set ups like following: Includes lots of set ups with multiple queens in the same column Number of set ups 8 * 8 * 8 * 8 * 8 * 8 * 8 * 8 = 16,777,216 We have reduced search space by two orders of magnitude by applying some logic Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Solution: Chess board Store The previous calculation help us to model the chessboard not as a 2-D array, but as a set of rows, columns and diagonals. To simplify the presentation, we will study for smaller chessboard, 4 by 4 First: we need to define an array to store the location of so far placed queens PositionInRow 1 3 2 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Solution: Chess board Store (Cont..) We need an array to keep track of the availability status of the column when we assign queens. Suppose that we have placed two queens F T Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Solution: Chess board Store (Cont..) We have 7 left diagonals, we want to keep track of available diagonals after the so far allocated queens T F Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Solution: Chess board Store (Cont..) We have 7 left diagonals, we want to keep track of available diagonals after the so far allocated queens T F Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Solution: Chess board Store (Cont..) Now Run a loop from 1 to 4 with counter i and place a Queen to safe position of the ith column. Now try to make a solution for N Queen. Try to find all possible solution. [4,3] [3,1] [2,4] [1,2] Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

N Queen Puzzle: Algorithm putQueen(int row) { for col=1 to N if (column[col]==available && leftDiagonal[row+col]==available && rightDiagonal[row-col+norm]== available) positionInRow[row]=col; column[col]=!available; leftDiagonal[row+col]=!available; rightDiagonal[row-col+norm]=!available; if (row< squares-1) putQueen(row+1); else PRINT solution found column[col]=available; leftDiagonal[row+col]=available; rightDiagonal[row-col+norm]= available; } Base Update Forward Recursi. Back Track Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Backtracking Algorithm for optimization problems Procedure checknode (node v ) { node u ; if ( value(v) is better than best ) best = value(v); if (promising (v) ) for (each child u of v) checknode (u ); } best is the best value so far and is initialized to a value that is equal or worse than any possible solution. value(v) is the value of the solution at the node. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Backtracking for optimization problems (Cont…) To deal with optimization we compute: best - value of best solution achieved so far value(v) - the value of the solution at node v Modify promising(v) Best is initialized to a value that is equal to a candidate solution or worse than any possible solution. Best is updated to value(v) if the solution at v is “better” By “better” we mean: larger in the case of maximization and smaller in the case of minimization Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Modifying promising A node is promising when it is feasible and can lead to a feasible solution and “there is a chance that a better solution than best can be achieved by expanding it” Otherwise it is nonpromising A bound on the best solution that can be achieved by expanding the node is computed and compared to best If the bound > best for maximization, (< best for minimization) the node is promising Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Modifying promising for Maximization Problems For a maximization problem the bound is an upper bound, the largest possible solution that can be achieved by expanding the node is less or equal to the upper bound If upper bound > best so far, a better solution may be found by expanding the node and the feasible node is promising Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Modifying promising for Minimization Problems For minimization the bound is a lower bound, the smallest possible solution that can be achieved by expanding the node is less or equal to the lower bound If lower bound < best a better solution may be found and the feasible node is promising Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Optimization Example:0-1 Knapsack Problem The 0-1 knapsack problem: N items which cost p1,p2,p3….pn and have weight w1, w2,w3….wn. (where the i-th item is worth pi dollars and weight wi pounds.) vi and wi are integers. A thief can carry at most C (integer) pounds. How to take as valuable a load as possible. An item cannot be divided into pieces. The fractional knapsack problem: The same setting, but the thief can take fractions of items. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Knapsack Problem: In Mathematical Term Input description: A set of items 1n, where item i has size wi and value pi, and a knapsack capacity C Problem description: Find a subset S that maximizes given that i.e. all the items fit in a knapsack of size C Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack - Example E.g.1: E.g.2: 50 50 50 Item: 1 2 3 4 5 6 7 20 $100 $120 + $220 30 Item 3 30 20 Item 2 $100 + 20 Item 1 10 10 $60 $60 $100 $120 W $160 $6/pound $5/pound $4/pound E.g.2: Item: 1 2 3 4 5 6 7 Benefit: 5 8 3 2 7 9 4 Weight: 7 8 4 10 4 6 4 Knapsack holds a maximum of 22 pounds Fill it to get the maximum benefit Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Brute force tree A B C D E B C D E Weight = 15.12 Value = $315 In Out B C D E B C D E Weight = 15.12 Value = $315 Weight = 8.98 Value = $235 Weight = 9.98 Value = $225 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack Problem: Brute force Algorithm Let’s first solve this problem with a straightforward algorithm Since there are n items, there are 2n possible combinations of items. We go through all combinations and find the one with the most total value and with total weight less or equal to W Running time will be O(2n) In the tree representation, we can think of the previous algorithm doing a DFS in the tree If we reach a point where a solution no longer is feasible, there is no need to continue exploring We can “backtrack” from this point and potentially cut off much of the tree and many of the solutions We can backtrack if we know the best possible solution in current subtree is worse than current best solution obtained so far. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack: Algorithm Inputs: Positive integers n and W; arrays w and p, each indexed from 1 to n, and each containing positive integers sorted in nonincreasing order according to the values of p[i]/w[i]. Outputs: an array bestset indexed from 1 to n, where the values of bestset[i] is "yes" if the ith item is included in the optimal set and is "no" otherwise; an integer maxprofit that is the maximum profit. void knapsack (index i, int profit, int weight) { if (weight <= W&& profit > maxprofit){ // This set is best maxprofit = profit; // so far. numbest = i; // Set numbest to bestset = include; // number of items } // considered. Set // bestset to this // solution. if (promising(i)){ include [i + 1] = "yes"; // Include w[i + 1]. knapsack(i + 1, profit + p[i + 1], weight + w[i + 1]); include [i + 1] = "no"; // Do not include knapsack (i + 1, profit, weight); // w[i + 1]. } Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack: Algorithm (Cont…) bool promising (index i) { index j, k; int totweight; float bound;   if (weight >= W) // Node is promising only return false; // if we should expand to else { // its children. There must j = i + 1; // be some capacity left for bound = profit; // the children. totweight = weight; Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack: Algorithm (Cont…) while (j <= n && totweight + w[j] < = W){ // Grab as many items as totweight = totweight + w[j]; // possible. bound = bound + p[j]; j++; } k = j; // Use k for consistency if (k <=n) // with formula in text. bound = bound + (W - totweight) * p[k]/w[k]; // Grab fraction of kth   return bound > maxprofit; // item. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

0-1 Knapsack: Implementation (Cont…) numbest = 0; maxprofit = 0; knapsack(0, 0, 0); print maxprofit; // Write the maximum profit. for (j = 1; j <= numbest; j++) // Show an optimal set of items. print bestset[i]; Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Subset finding problem We say that running time of an alg. is O(f(n)) If there exists c such that for large enough n: RunningTime(n) < c·f(n) Translation: From some point c·f(n) is an upper bound on the running time. Why only for “large enough n”? Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET

Other Problems Do not assume that because we analyze running time asymptotically we no longer care about the constants. If you have two algorithms with different asymptotic limits – easy to choose. (may be misleading if constants are really large). Usually we are happy when something runs in polynomial time and not exponential time. O(n2) is much much better than O(2n). Md. Monjur-ul-hasan. Lecturer, Dept. of CSE CUET