1. Alex Bolsoy, Jonathan Suggs, Casey Wenner
0/1 Knapsack Problem
Alex Bolsoy, Jonathan Suggs, Casey Wenner
Casey

2. Abstract
The intent of this project is to examine multiple non-trivial algorithms for the 0/1 knapsack problem. Our project involves testing the effectiveness of simulated annealing, dynamic programming and genetic algorithms. Results will be compared based on value of results and the time and memory consumption of each algorithm.
Casey

3. Introduction Optimization problem.
Given a set of items and a knapsack.
The knapsack has a weight limit.
Each item has a weight value and a dollar value.
Place items in the knapsack to maximize the dollar value.
However, the combined weight of the items must be less than or equal to the weight limit of the knapsack.
Casey

4. Formal Problem Statement
Given a set of n items from 1 to n, each with a weight wi and value vi, along with a maximum weight capacity W,
where xi represents number of instances of item i to place in knapsack.
= V
Source: Wikipedia.org
Johnny

5. Test Set Data Problem instances from Pisinger’s website
Uncorrelated instance
100 items - no correlation between the weight and value of items
995 knapsack capacity
Correlated instance
100 items - correlation between the weight and value of items
900 knapsack capacity
Alex

6. Context Combinatorial optimization problem.
Applications in resource allocation in a variety of industries.
Extensively studied problem, early works dating back to 1897.
Tobias Dantzig.
NP-hard problem.
No known algorithm both correct and polynomial in time.
David Pisinger.
Danish computer science researcher (University of Copenhagen).
Extensive research done on problem.
2004 paper titled “Where are the hard knapsack problems?”
- Alex

7. Experimental Procedure

8. Simulated Annealing
Used to approximate global maximum in fixed amount of time
Includes hill climbing, but sometimes accepts worse total value (V), allowing algorithm to explore other options, avoiding settling on local maxima.

9. Simulated Annealing While Iterating (T > Tmin):
Randomly select item to put in knapsack
while new knapsack weight > W
randomly remove items until knapsack weight <= W and compare new V with previous V
If new V is better, accept combination
Else worse V, accept or reject new combination based on probability, determined by temperature and differences in previous V and current V

10. Simulated Annealing Temperature determined by cooling schedule
Start at T0, end at Tmin
T approaches Tmin logarithmically each iteration
Tk = T0/(1 + ln(1+k))
p = e(currentV - previousV)/Tk)
If chance < p: accept worse value
T = temperature
T0 = initial temperature
Tmin = final temperature
K = iteration
p = acceptance probability
chance = (0 <= random float <=1)

11. Simulated Annealing Analysis
O(n log k)
Polynomial time
n - number of items - deepcopy() item list
k - number of iterations
Memory
2 arrays of length n, 3 lists
Solution Quality
Uncorrelated Data - optimized to ~ 95% of best V for max W and max n.
Correlated Data - optimized to ~ 99.7% of best V for max W and max n.

12. Simulated Annealing Results
Uncorrelated Data Correlated Data

13. Simulated Annealing Results
Uncorrelated Data Correlated Data

14. Simulated Annealing Results

15. Simulated Annealing Results
Uncorrelated Data

16. Genetic Create a population of viable solutions
Crossover between viable solutions
favor higher V solutions, but include some lower V solutions for diversity
Mutation
A relatively small chance to change random values in a solution
Prevents stagnation
Sort values by fitness value
Do it all again

18. Uncorrelated Data Set, Static Iteration count
List Length
value
time 10
2287 20 30 40 50 60 70 80 90
100

19. Uncorrelated Data Set, Static Iteration count Capacity
value
time
100
1.282
200
300
400
500
600
700
800
900
995

20. Correlated Data Set, Static Iteration count; Capacity
Value
Time
100
380
200
760
300
1330
400
1710
500
2280
600
2660
700
3230
800
3610
900
3990

21. Correlated Data Set, Static Iteration count; List Size
Value
Time 10
3800 20 30
3990 40 50 60 70 80 90
100

22. Uncorrelated Given Proportional Iteration count
List Size
value
Time 10
2287 20
4156 30
5500 40 50 60 70
9053.4 80 90
100

23. Uncorrelated Given Proportional Iteration count
Capacity
value
time
100
200
300
400
500
600
700
800
900
995

24. Genetic Analysis O(O(F) * (O(C) + O(M))
O(F) fitness method
n where n is the number of items in a list
O(C) cross over method
N log (n) where n is the number of items in the data set
O(M) mutation method
Constant
Simplified Big(O); n log(n) polynomial time
Memory, a list of; 2(P(N)) Where P is the population size, and n is the number of items
Solution quality; Varies significantly given time and type of data.

25. Dynamic Programming
Make a matrix with the values of the items to find the optimal solution.
Backtrack over the matrix to determine which items make up the optimal solution.
Bottom-up algorithm.

26. Dynamic Programming

27. Analysis O(n*W) Memory - multidimensional integer list of size n*W
Pseudo-polynomial time
Building the matrix
Where n = number of items and W = knapsack capacity
Memory - multidimensional integer list of size n*W
Solution Quality - finds the best solution every time
Difficulty - relatively easy to understand and implement in code
In computational complexity theory, a numeric algorithm runs in pseudo-polynomial time if its running time is a polynomial in the numeric value of the input (the largest integer present in the input) — but not necessarily in the length of the input (the number of bits required to represent it), which is the case for polynomial time algorithms. In general, the numeric value of the input is exponential in the input length, which is why a pseudo-polynomial time algorithm does not necessarily run in polynomial time with respect to the input length.

28. Uncorrelated Data Set Input Time Maximum 0.000248 10 0.011598 2287 20 10
2287 20
4156 30
5500 40
7758 50
8373 60
8709 70
9147 80 90
100
* Input = (Number of Items)
Input
Time
Maximum
100
2156
200
3544
300
4452
400
5252
500
5978
600
6826
700
7552
800
8150
900
8719
995
9147
* Input = (Knapsack Capacity)

29. Uncorrelated Data Set Input Input2 Count Time Maximum 1 7.21E-06 1001
7.21E-06
100 10 2
914
200 20 3
1915
300 30 4
2822
400 40 5
4513
500 50 6
5447
600 60 7
6624
700 70 8
7552
800 80 9
8150
900 90
8719
995 11
0.1102
9147
* Input = (Knapsack Capacity Number of Items)

30. Correlated Data Set Input Time Maximum 0.000692 10 0.014425 3800 20 10
3800 20
3990 30 40 50 60 70 80 90
100
* Input = (Number of Items)
Input
Time
Maximum
100
380
200
760
300
1330
400
1710
500
2280
600
2660
700
3230
800
3610
900
3990
* Input = (Knapsack Capacity)

31. Correlated Data Set Input Input2 Count Time Maximum 10 1 7.81E-05 10010 1
7.81E-05
100 20 2
380
200 30 3
760
300 40 4
1330
400 50 5
1710
500 60 6
2280
600 70 7
2660
700 80 8
3230
800 90 9
3610
900
3990
* Input = (Knapsack Capacity Number of Items)

32. Comparisons

33. Comparisons

34. Comparisons

35. Comparisons

36. Interpretation/Conclusions
Genetic
Slowest
Semi-difficult to implement
Not very good for Knapsack Problem
Simulated Annealing
Fastest algorithm - not always the very best answer
Difficult to implement
Best option for large datasets
Dynamic Programming
Most accurate algorithm - gets the highest answer every time
Runs out of memory on extremely large datasets
Relatively easy to implement
Best option for most datasets
Alex
As long as the capacity of the knapsack is less than the size of the population, the dynamic programming will outperform the genetic algorithm. However, once the capacity becomes greater than the size of the population, the dynamic programming number of operations and memory required will be a lot greater than the genetic algorithms ones.

37. Future Work Modified simulated annealing algorithm.
Try restarts, exponential cooling schedule
Modified dynamic programming algorithm.
Reduce size of matrix by only calculating necessary elements.
Modified genetic algorithm.
Develop a more dynamic termination method.
Optimize population size and mutation chance
Combinations of different algorithms.
Apply algorithms to TSP.
Johnny

38. Five Questions What type of problem is the knapsack problem?
Optimization problem
What is the computational complexity of the optimization form of the knapsack problem?
NP-hard
What is the Big-O of a dynamic programming algorithm for the knapsack problem?
O(n*W), where n is number of items and W is knapsack capacity
What is the Big-O of the genetic algorithm for the knapsack problem?
n log (n)
Which algorithm that we studied is least effective for the knapsack problem?
Genetic

39. Works Cited https://en.wikipedia.org/wiki/Knapsack_problem