1. INTELLIGENT TEST SCHEDULING TE-MPE Technical Meeting 10.04.2012 Michael Galetzka

2. SYSTEM ARCHITECTURE LSA database HWC java GUI HWC Sequencer server Test tracking server (+ scheduler) Hardware (FPGAs --> circuits)

3.  Not all tests can be executed at any given moment  Test phases  System locks: database, PIC  Tests under execution  Constraints between tests from different systems  e.g. ‘Only one 600A circuit is allowed to be active per QPS controller’  Some test can only be executed together  e.g. main quadrupole magnets operating above 2kA WHY SCHEDULING? Test ATest BTest CTest D

4. ADVANTAGES OF AUTOMATED SCHEDULING  Reduced work for the users (e.g. LHC operators)  Possibility to do more work in parallel (no need for SoC any more)  All constraints are checked and enforced automatically => fewer failures  Test execution can be optimized to reduce the overall timespan  A complete test execution schedule provides additional information  Which test will be executed when?  How long will it take until everything is completed?

5. Test plan for S1 A SMALL EXAMPLE Test ATest B Two systems: S1 and S2 One constraint: Test A can only run on one system at a time Test plan for S2 Test A

6. RESULTING SCHEDULES System S1 System S2 t Test A Test B System S1 System S2 t Test A Test B longer schedule optimal schedule

7. THAT DOES NOT SEEM SO HARD...  Test scheduling is a special instance of the constraint satisfaction problem (like sudoku or map coloring)  The time to find the optimal solution increases exponentially with the number of tests No. TestsTime taken (s) 50 102 15352 20??? 10000impossible

8. HOW IS IT DONE AT THE MOMENT?  Basically four steps: 1. Get all tests that should be executed 2. Remove all tests that can not be executed because of restrictions (e.g. phase dependecies or system already in use) 3. Pick one test for each system (according to the constraints with other tests) 4. Remove all tests that violate one or more grouping constraints System S1 System S2 System S3 System S4 System S5 Test A Test B Test C Test A Test B Test X Test C Test E Test F

9. THE CURRENT SOLUTION

10. Test plan for S1 A SMALL EXAMPLE Test ATest B Two systems: S1 and S2 One constraint: Test A on one system must run together with Test B on another system Test plan for S2 Test A

11. RESULTING SCHEDULES System S1Test A System S2Test A Step 1: start System S1Test B System S2 Step 2 System S1 System S2 Step 3: result System S1Test B System S2Test A Optimal solution

12. SCHEDULER COMMENTS

13. THE OBJECTIVE: A GENERAL PURPOSE SCHEDULER  Should be able to deal with all constrained scheduling problems  AccTesting test scheduling  (Software system test and deployment plans)  (Managing tunnel access) ...  Must be (reasonably) fast  Should be easy to understand and maintain  Should create good results = schedules with a small overall timespan

14. HEURISTIC REPAIR  A prototype has been implemented for testing  Starts with an infeasible solution  Tries to repair it until it has a working result  Uses a heuristic to decide what to repair first and how to repair it

15. EXAMPLE: HEURISTIC REPAIR System S1 System S2 t System S3 System S4 g

16. EXAMPLE: HEURISTIC REPAIR System S1 System S2 t System S3 System S4 g

17. EXAMPLE: HEURISTIC REPAIR System S1 System S2 t System S3 System S4 g

18. EXAMPLE: HEURISTIC REPAIR System S1 System S2 t System S3 System S4 g

19. EXAMPLE: HEURISTIC REPAIR System S1 System S2 t System S3 System S4 g

20. EXAMPLE: OPTIMAL SOLUTION System S1 System S2 t System S3 System S4 g

21. RESULTS SO FAR

22. OTHER POSSIBLE APPROXIMATION ALGORITHMS  Tree Search  Ant Colony Optimization  Simulated Annealing  Genetic Algorithms  Neural Networks ...

23. EXAMPLE 1: TREE SEARCH  Uses a tree to represent the search space  Uses backtracking to navigate the search through the tree  Stops once a good solution has been found  Always finds a solution if one exists  hard to implement efficiently 1 2 35 4 7 89 6

24. EXAMPLE 2: ANT COLONY OPTIMIZATION  Uses virtual ants (independent agents) to find a solution  Each ant leaves a bit of pheromone on the trail used to find a solution  New ants are more likely to follow trails with more pheromone on them  Stochastic algorithm  Scales very well with the hardware  Not as easy to understand and maintain  Hard to test because it is not deterministic

25. SUMMARY  Constrained scheduling is a tricky problem  Current scheduling strategy can be improved  A general purpose scheduler can be used for many problems  A prototype of the Heuristic Repair has been implemented  The approximation algorithms are very diverse, there might be a better one than Heuristic Repair