Intelligent design primer Collecting CSI from people for search algorithms Existing users of technique Search algorithm primer Why use search algorithms in the first place? How search algorithms work The search algorithm dilemma Intelligent Agents to the rescue How we help solve the dilemma through superior pattern recognition Empirical evidence of human capability The Experiment Hypothesis Experiment design Results Questions Guide for the Perplexed OUTLINE OF CONFUSION

WHAT IS ID? IS IT JUST GOD OF THE GAPS? INTELLIGENT DESIGN PRIMER

The fundament question of Intelligent Design Theory: How do we know intelligent design when we see it? The fundament claim of Intelligent Design Theory: Only intelligent agents create information. FUNDAMENTALS

IRREDUCIBLE COMPLEXITY

EXPLANATORY FILTER

COMPLEX SPECIFIED INFORMATION Is this fractal complex specified information?

Creating irreducible complexity creates complex specified information WHAT DOES IT MEAN TO CREATE INFORMATION?

HOW IS ID DIFFERENT THAN ALL OF MODERN SCIENCE? ID

WHY IS ID FITTER THAN DARWINISM? ID focuses on the information creation instead of on the information. OR THISTHIS? Would you rather own…

Intelligent design primer Collecting CSI from people for search algorithms Existing users of technique Search algorithm primer Why use search algorithms in the first place? How search algorithms work The search algorithm dilemma Intelligent Agents to the rescue How we help solve the dilemma through superior pattern recognition Empirical evidence of human capability The Experiment Hypothesis Experiment design Results Questions Guide for the Perplexed OUTLINE OF CONFUSION

COLLECTING ACTIVE INFORMATION FOR SEARCH & OPTIMIZATION

COLLECTING CSI FROM PEOPLE According to Intelligent Design theory, particularly in “Search for a Search”, intelligent agents such as people are capable of improving search algorithm performance beyond mathematical bounds. Goal: Create a generalized interface for people to contribute to an algorithmic search and optimization process, thus demonstrating human supra-computational capability.

COLLECTING CSI FROM PEOPLE

COMMERCIAL CSI COLLECTION Mechanical Turk Marketplace of simple web based jobs for low skill work reCaptcha Uses captchas to correct OCR text translation Foldit Players fold genes along with algorithm, achieving results superior to gene folding algorithm alone Google Image Labeling Players compete to label images

COMMERCIAL CSI COLLECTION Mechanical Turk Marketplace of simple web based jobs for low skill work reCaptcha Uses captchas to correct OCR text translation Fold It Players fold genes along with algorithm, achieving results superior to gene folding algorithm alone Google Image Labeling Players compete to label images

Intelligent design primer Collecting CSI from people for search algorithms Existing users of technique Search algorithm primer Why use search algorithms in the first place? How search algorithms work The search algorithm dilemma Intelligent Agents to the rescue How we help solve the dilemma through superior pattern recognition Empirical evidence of human capability The Experiment Hypothesis Experiment design Results Questions Guide for the Perplexed OUTLINE OF CONFUSION

WHAT ROBOTS CAN DO SEARCH ALGORITHM PRIMER

WHEN ARE SEARCH ALGORITHMS USED? Many problems can be solved by straightforward algorithms in an amount of time polynomial proportional to the problem size. These problems are generally tractable for solving exactly with a computer, though a significant amount of computing power and space may be necessary. However, there is a much larger group of problems which, as far as we know, cannot be solved in polynomial time (NPC+). For these problems the best we can do is a best effort attempt to get as close to the optimal as possible within our computation time and space limits. There are numerous different heuristic and approximation algorithms that are used for NPC+ problems, and this is where search algorithms are used. Since we don’t know how to find the optimum solution, we have to search around in a problem space.

HOW COMPLEXITY CLASSES SCALE BLUE=LINEAR, GREEN=POLYNOMIAL, RED=EXPONENTIAL

SOME EXAMPLES OF NPC+ PROBLEMS Finding binding sites on proteins Delivery route planning Calculating cheap airline trips Stock market portfolio selection Packing your belongings for a move Making the Internet fast

HOW SEARCH ALGORITHMS WORK Search is a process of hill climbing, focusing on using information in previously found solutions to find even better solutions. One well known example is the Newton-Raphson method of finding square roots. The problem is an uneven search landscape will cause a search to become stuck on low lying peaks and crags. To get out of these traps, the search algorithm has to have an element of exploration. Exploration consists of sampling areas of the landscape, and hill climbing in promising sections.

HOW SEARCH ALGORITHMS WORK Search is a process of hill climbing, focusing on using information in previously found solutions to find even better solutions. One well known example is Newton’s method of finding square roots. The problem is in an uneven search landscape will cause a search to get stuck on low peaks and crags. To get out of these traps, the search algorithm has to have an element of exploration.Exploration consists of sampling areas of the landscape, and exploring promising sections.

FINDING GOOD PLACES TO EXPLORE How do we know where to go? x x x x

THE DILEMMA Unfortunately, selecting good areas to hillclimb is itself a very difficult problem to solve, and depending on how good of a guess is desired the selection algorithm will be NPC+. Consequently, using search effectively to solve an NPC+ problem ends up introducing a new problem of equal or greater complexity (as predicted by Dembski’s “Search for a Search” paper). Consider the following solution set from which a search algorithm needs to select a new space to explore.

EXAMPLE Which solution signifies a new area to investigate? 1   2   3   4   5   6   7   8   9  

Intelligent design primer Collecting CSI from people for search algorithms Existing users of technique Search algorithm primer Why use search algorithms in the first place? How search algorithms work The search algorithm dilemma Intelligent Agents to the rescue How we help solve the dilemma through superior pattern recognition Empirical evidence of human capability The Experiment Hypothesis Experiment design Results Questions Guide for the Perplexed OUTLINE OF CONFUSION

OUR SUPERIOR PATTERN RECOGNITION INTELLIGENT AGENTS TO THE RESCUE

WHY CAN WE IMPROVE ALGORITHMS? In Dr. Dembski’s “Search for a Search” he shows that search algorithms are incapable of finding a search target any better than a random search, without the insertion of external information. Furthermore, he shows that such information cannot come from another search algorithm. It can only come from a non-algorithmic source. Intelligent design theory posits that intelligent agents are capable of creating this information, and consequently capable of improving the capabilities of search and optimization algorithms.

CAN WE IMPROVE ALGORITHMS? In Dr. Dembski’s “Search for a Search” he shows that search algorithms are incapable of finding a search target any better than a random search, without the insertion of external information. Furthermore, he shows that such information cannot come from another search algorithm. It can only come from a non-algorithmic source. Intelligent design theory posits that intelligent agents are capable of creating this information, and consequently capable of improving the capabilities of search and optimization algorithms. ????

HUMAN VS ALGORITHM Shows human and algorithmic performance on an NP-Complete (Travelling Salesman Problem). Points and O(n)/O(n ln n) plots show human capability, O(n 2 ) and greater show algorithmic capability.

HUMAN VS ALGORITHM

HOW HUMANS HELP SOLVE THE DILEMMA If humans are capable of adding information to the search process, then we can assist the search algorithm in exploring the problem space more effectively than algorithmically possible. The reason why algorithms have trouble searching is because they don’t have a good, generic pattern detection ability. They can’t effectively detect patterns in the solutions that lead them to better solutions. However, we humans are known for our pattern detection, and can use our superior ability to help out the algorithm. Let’s take another look at the search process.

EXAMPLE Which solution signifies a new area to investigate? Must be both very unlike other good solutions, while being highly ranked. 1   2   3   4   5   6   7   8   9  

EXAMPLE Which solution signifies a new area to investigate? Must be both very unlike other good solutions, while being highly ranked. 1   2   3   4   5   6   7   8   9   This solution is most unlike the rest, while also being highly ranked.

Intelligent design primer Collecting CSI from people for search algorithms Existing users of technique Search algorithm primer Why use search algorithms in the first place? How search algorithms work The search algorithm dilemma Intelligent Agents to the rescue How we help solve the dilemma through superior pattern recognition Empirical evidence of human capability The Experiment Hypothesis Experiment design Results Questions Guide for the Perplexed OUTLINE OF CONFUSION

IN WHICH THINGS KIND OF WORK THE EXPERIMENT

HYPOTHESES Grand hypothesis: humans can improve any improvable search algorithm beyond mathematical limits Actual hypothesis: humans can improve a particular search algorithm in a particular domain Criteria for verification: human generated solution displaces best solutions found by computer in fewer samples of solutions

EXPERIMENT Problem: find primes that generated RSA key pair The fitness function has access to an original plain text and its cypher text. Metric: two objectives to be maximized 1) similarity between original plain text and cypher text generated by a given set of primes 2) similarity between original cypher text and its decryption generated by a given set of primes Algorithm: multi-objective genetic algorithm Human involvement: users of Amazon’s Mechanical Turk service will select a set of solutions for one iteration of GA optimization Method of comparison: best solution found in proportion to number of solutions checked by humans/algorithm.

SCREENSHOT

SCREENSHOT EXPLANATION Stars represent relative valuation of solution. 5 stars means one of best solutions found so far. Solution is really just a bit string (universal problem representation). However, to make patterns more discernable and more appealing to the eye, substrings are mapped to images. Checkbox selected by user to signify solution set for algorithm exploration.

AMAZON TURK RESULTS Optimal solution found by both algorithm and Amazon Turk user with values of 45 and 121 Objective #1 Objective #2 There exists an optimum solution with objective values of 64 and 236 Optimum

CONCLUSION Actual hypothesis not verified. Humans (may have) contributed to, but did not improve, the search process. Solution found did not displace solutions found by algorithm, since exact same solution was found by algorithm. Therefore, no human generated improvement observed. However, human finding same solution shows definite contribution. Experiment shows slight promise. However, Amazon Turk users are known to script their responses. So, results may be output of a script, not a human. Many things can be improved in algorithm, GUI, data collection and mathematical analysis.

IMPROVEMENTS TO EXPERIMENT Add Captcha to submission form so Turkers cannot script form submission. More descriptive user interface. Describe experiment? Turn into a game? Other suggestions? Better comparison between human and algorithm?

WHY IS THERE NO PROBLEM INFORMATION? This representation is all the search algorithm sees. It knows nothing about the nature of the problem. Consequently, to perform a fair comparison, the human user cannot be given any additional problem domain information. 1   2   3   4   5   6   7   8   9  

WHY COMPARE ON THE NUMBER OF SOLUTIONS EVALUATED? Both human users and algorithms are allowed to do whatever they want with the solutions that have been found so far. Consequently, the number of solutions evaluated is the upper bound on information used by both parties to discover new search areas.