Tutorial 3 Empirical evaluation in AI CS 236501 Introduction to AI Tutorial 3 Empirical evaluation in AI

Intro. to AI – Tutorial 3 – By Nela Gurevich Agenda Importance of empirical evaluation in AI Empirical evaluation guidelines Designing and running an experiment Experimental results presentation Discussing the experimental results 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Empirical evaluation in AI Empirical = Exploratory + Experimental We wish to explain the behavior of an algorithm We conduct experiments to prove our explanation Sometimes, the opposite occurs – we make conclusion about an algorithm behavior from the results of an experiment Empirical evaluation is an important part in AI research Many algorithms do not have a sound theoretical proof for their properties 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Empirical evaluation in AI We use experiments to prove our theories It is important to design experiments properly It is important to clearly present our experimental results Clear presentation and explanation of experimental results are a major part of any AI research and of this course Home Assignments! We will guide you in the home assignments 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Designing an experiment: first step What would you like to do? Examine the effect of a certain parameter on an algorithm performance Example: Beam search with beam width = k We would like to examine the effect of k on algorithm performance Compare the performance of two (or more) algorithms Example: DFS vs. BFS on Tic-Tac-Toe puzzle 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Parameter effect on algorithm performance Fix all algorithm parameters, except the one that is inspected Example: Effect of beam width (k) on Beam Search Fix the problem difficulty Run the experiment for different values of the inspected parameter Example: Run Beam Search with k = {1, 5, 10, 50, 100, 500, 1000} – small, medium and large values Important: choosing the parameter values range correctly K = {1, 5, 7} or {k = 1001, 1005, 1007} is a bad decision The range of the values may vary for different problems 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Comparing two (or more) algorithms Run two algorithms with fixed parameters For proper comparison, avoid all possible differences Example: DFS vs. BFS on Tic-Tac-Toe Run the algorithms on problems of the same difficulty, or even on the same problem Example: Comparing two heuristics Use the same search algorithm with the same parameters Use problems of the same difficulty (or same problems) 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Designing an experiment: Random Elements Sometimes random elements affect the performance of an algorithm, for example: Random initial problems In Hill-Climbing search, when two successors have the same heuristic value, one is chosen at random In such cases, the experiment should be repeated more than once, and results should be averaged Important: initialize the random seed in your program, to avoid repeating the same “random” experiment over and over srand( (unsigned)time( NULL ) ); // in C 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Intro. to AI – Tutorial 3 – By Nela Gurevich Running an experiment Trace the experiment For debugging purposes Record all data that seems important Saves time later (no need to re-run experiments to get the needed data) Example: DFS Record number of expanded nodes, execution time, whether solution was found, solution length. Use batch (script) files Easy to reproduce results Save time near computer 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results Use tables or graphs – visual methods, easy to understand First of all, decide what is the purpose of the table/graph you want to present 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results Example: I run Beam Search on n random problems. Beam Search is not complete: sometimes it finds a solution to a problem, and sometimes it does not. I would like to present a graph that shows the effect of the beam width (k) parameter in Beam Search on the percentage of problems for which solution is found. 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results – common guidelines Graphs/tables should be very easy to understand No additional mathematical calculations should be required to understand the graph/table Run Alg 1 2 3 4 5 Alg1 100 140 200 90 20 Alg2 30 40 210 95 60 X Table1: Solution length found by Alg1 and Alg2 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results – common guidelines Avoid combining large amount of data in one graph/table Run Results 1 2 3 4 5 Alg1 expanded nodes 100 140 200 90 20 Alg1 solution length 7 9 10 Alg1 – time 22.2 34.2 55.4 33.2 11.1 Alg2 expanded nodes 150 110 67 99 105 Alg2 solution length 6 Alg2 - time 12.4 41.2 31.3 22 23.5 X 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results: graphs Name the axis Scale the axis properly Use graphs for continuous values Use graphs for discrete values 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Presenting experimental results: tables Name the data columns properly Specify the measurement units used in the table Use short tables When long, detailed tables need to be attached, attach them as appendices Summarize with averages, when needed 87 110 Avg 60 95 210 40 30 Alg2 20 90 200 140 100 Alg1 5 4 3 2 1 Problem Alg Table1: Solution length found by Alg1 and Alg2 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Discussing the experimental results It is essential to discuss the obtained experimental results in words ! Verbal discussion with visual presentation of the results is the main part of the empirical evaluation of algorithms 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Discussing the experimental results: guidelines Support every conclusion you make with data that proves this conclusion Insert graphs/tables in appropriate places in the text for easy reference Use short and clear sentences Avoid using many adjectives Avid combining many conclusions in one sentence 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Discussing the experimental results: guidelines Not always the conclusion is that Algorithm1 is clearly better than Algorithm2 on a given problem Discuss advantages/disadvantages of each algorithm Compare the different performance elements of the algorithms Example: Alg1 expands twice as many nodes as Alg2, on average, Alg1 and Alg2 take the same time to run on average Not always the conclusion from an experiment is that Algorithm1 is better than Algorithm2 in general Discuss how the algorithms are affected by the problem on which they are tested 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Intro. to AI – Tutorial 3 – By Nela Gurevich Summary Designing and running an experiment should be done carefully Present experimental results with graphs and tables. It is important that the visual presentation of the results is clear to the reader Findings should be summarized with a verbal discussion of the experimental results, backed up with visual presentation of results 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich

Intro. to AI – Tutorial 3 – By Nela Gurevich Final Tip Print homework on both sides of the paper – save trees 14-Jan-19 Intro. to AI – Tutorial 3 – By Nela Gurevich