1. Institute of Computer and Communication Network Engineering Multi-dimensional Robustness Optimization of Embedded Systems & Online Performance Verification Arne Hamann Steffen Stein Rolf Ernst

2. Institute of Computer and Communication Network Engineering Part I: Multi-dimensional Robustness Optimization of Embedded Systems Arne Hamann Rolf Ernst

3. 3 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Outline System property variations Sensitivity Analysis Stochastic Multi-dimensional Sensitivity Analysis Robustness Metrics –Hypervolume calculation –Minimum Guaranteed Robustness (MGR) –Maximum Possible Robustness (MPR) Experiments

4. 4 Arne Hamann, Steffen Stein, IDA, TU Braunschweig System Property Variations Why do system property variations occur? –Specification changes, late feature requests, product variants, software updates, bug-fixes Robustness to property variations –decreases design risk, and increases system maintainability and extensibility Property variations can have severe unintuitive effects on system performance Sensitivity analysis: achieve robustness without on-line parameter adaptation

5. 5 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Problem Formulation Find fixed parameter configuration that … … maximizes system robustness w.r.t. changes of several properties Robustness = the system can sustain property variations without severe performance degradation Not included: dynamic parameter adaptations (ongoing work submitted to EMSOFT 2007)

6. 6 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Stochastic Sensitivity Analysis (1) Problem of exact sensitivity analysis approaches: computational effort grows exponentially with number of considered dimensions Solution: scalable stochastic analysis able to quickly bound system sensitivity

7. 7 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Stochastic Sensitivity Analysis (2) Sensitivity analysis formulated as multi- objective optimization problem  Pareto-front of optimization task corresponds to sought-after sensitivity front Use multi-criteria evolutionary algorithms to approximate sensitivity front –E.g. SPEA2 (ETH Zurich): diversified sensitivity front approximation through Pareto-dominance based selection and density approximation

8. 8 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Creation of the Initial Population Creates a certain number of points representing a first approximation of sensitivity front Uses 1-dim sensitivity analysis –to bound the search space in each dimension (bounding hypercube) –to generate points representing the extrema of the sought-after sensitivity front Randomly place the rest of the initial points in bounding hypercube

9. 9 Arne Hamann, Steffen Stein, IDA, TU Braunschweig 6,5 10,85 26,2 10 Property 1 Property 2 Bounding Box Initial Population - Example

10. 10 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Bounding the Search Space (1) Idea: bound search space containing the sought-after sensitivity front –Bounding working Pareto-front F n evaluated Pareto-optimal working points –Bounding non-working Pareto-front F nw evaluated Pareto-optimal non-working points Bounding Pareto-fronts can be used to derive multi-dim. robustness metrics (later)

11. 11 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Bounding the Search Space (2) Space between bounding Pareto-fronts is called relevant region Variation operators use algorithm ensuring that generated offsprings (points) are situated in the relevant region –Below bounding non-working Pareto-front –Above bounding working Pareto-front  Efficiently focuses exploration effort

12. 12 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Bounding the Search Space (3) 6,5 10,85 26,2 10 Property 1 Property 2 Bounding Box

13. 13 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Front Convergence Mutate (1) Heuristic operator adapted to optimization problem Strategy: –Determine X closest points on opposite Pareto-front –Choose randomly one of these points –Place offspring point randomly on straight line connecting the parent point and the chosen random point Increases convergence speed of the bounding Pareto-fronts

14. 14 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Front Convergence Mutate (2) 6,5 10,85 26,2 10 Property 1 Property 2 Bounding Box

15. 15 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Front Convergence Mutate (3) 6,5 10,85 26,2 10 Property 1 Property 2 Bounding Box

16. 16 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Hypervolume Calculation Hypervolume as basis of the proposed robustness metrics Hypervolume is defined in a given hypercube and associated to a point set Two different notions of hypervolume –inner hypervolume : Volume of space Pareto-dominated by the given points inside the given hypercube –outer hypervolume : Volume of space Pareto-dominated by all points not Pareto- dominating any of the given points

17. 17 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Hypervolume Calculation (2) 2D-case –inner hypervolume: lower step function –outer hypervolume: upper step function Bounding Box [15,28]x[6,18] (15,18) (18,16) (20,12) (26,10) (28,6) ( )= 66 - ( )= 100 +

18. 18 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Robustness Metrics Given a set of properties … … use stochastic sensitivity analysis to derive upper and lower robustness bounds –Minimum Guaranteed Robustness (MGR) Defined as inner hypervolume of the bounding working Pareto-front F w –Maximum Possible Robustness (MPR) Defined as outer hypervolume of the bounding non-working Pareto-front F nw

19. 19 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Robustness Metrics (2) Obviously: MGR <= Real Robustness <= MPR 6,5 10,85 26,2 10 Property 1 Property 2 Bounding Box MGR MPR

20. 20 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Robustness Exploration Idea: Pareto-optimize MGR and MPR Advantages –Stochastic sensitivity analysis is scalable  Little computational effort necessary to reasonably bound robustness potential of given configuration –In-depth analysis can be performed once interesting configurations are identified (i.e. high MGR or high MPR)  Perfectly suited for robustness optimization

21. 21 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Example System Distributed embedded system 4 computational resources … …connected via CAN bus 3 constrained applications –Sens  Act –S in  S out –Cam  V out

22. 22 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Approximation Quality (1) Approximation after 100 evaluations (20 sec) MGR = 2447 MPR = 2937 Approximation after 200 evaluations (40 sec) MGR = 2580 MPR = 2813

23. 23 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Approximation Quality (2) Approximation after 300 evaluations (60 sec) MGR = 2632 MPR = 2777 Result using exact sensitivity analysis (85 sec) MGR = 2585 MPR = 2826

24. 24 Arne Hamann, Steffen Stein, IDA, TU Braunschweig 3D - Robustness Maximization Original configuration Optimized configuration

25. 25 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Integration of New Functionality Integration of a fourth application with lowest priorities What combinations WCET T 9 and WCCT C 6 are feasible? Is there optimization potential? Idea: initially assume WCET T 9 and WCCT C 6 equal zero T9T9 C6C6 Sens 2 Sink

26. 26 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Integration of New Functionality (2) WCCT C 6 WCET T 9 Areas below the curves represent feasible systems

27. Institute of Computer and Communication Network Engineering Part II: Online Performance Verification Steffen Stein Rolf Ernst

28. 28 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Outline Motivation Framework Architecture In Detail: Global Analysis Layer System Setup Approach to Analysis Control Experimental Results

29. 29 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Future Challenges In-Field updates Run-time Reconfigurations 90% of Innovation in Software Networked Systems Not Manageable at Design-Time! Engine Control SW Driver Assistance Multimedia Service

30. 30 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Approach Generally Speaking: –Make Systems clever enough to handle Integration Problem themselves Here: Timing Properties ToDo –Gather performance Data during runtime –Evaluate/ Optimise online –Feed Results back into running Systems Result: Evolving Systems

31. 31 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Architecture: Organic Computing Single Instance Multiple Instances Multiple collaborating Instances Layered approach System under Observation and Control (SuOC) observercontroller observescontrols reports selects observation model Goals/ Design Rules Source: Towards a generic observer/controller architecture for Organic Computing, U. Richter, M. Mnif, J. Branke, C. Müller-Schloer, H. Schmeck, INFORMATIK 2006 -- Informatik für Menschen

32. 32 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Analysis Engine ObserverController ObserverController Heterogeneous Networked Embedded System (SuOC) Analysis Engine Control Plane Local Layer Use resourcesGather dataAdjust settings Observer Controller Data Exchange Global Analysis Layer Global Controller Layer Global Observer Layer Self-Organisation Control Framework Self-Organisation

33. 33 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Distributed Setup ARM DSP uCPPC CAN Real System T2T2 T5T5 T1T1 T6T6 S2S2 S1S1 S4S4 S3S3 T8T8 T0T0 T7T7 T9T9 T3T3 T4T4 Global Model

34. 34 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Analysis Control T1T1 T3T3 T2T2 T4T4 T1T1 T6T6 Network Tunnel Distributed Analysis Control T1T1 T3T3 T2T2 T4T4 T1T1 T6T6 Analysis Control Do for all not up-to-date Resources Analyse end Until all Resources are up to date While (true) if Resource invalidated analyse Resource end

35. 35 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Performance of trivial Approach System size (# tasks) # analysis runs (resource level)

36. 36 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Problem T1T1 T4T4 T2T2 T5T5 S1S1 S2S2 T3T3 T6T6 T7T7 T8T8 S3S3 T9T9 Exponential increase in number of necesary Analysis runs Solution: Caching

37. 37 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Performance with caching System size (# tasks) # analysis runs (resource level)

38. 38 Arne Hamann, Steffen Stein, IDA, TU Braunschweig Conclusion Distributed Performance Analysis implemented Suitable as evaluator for online performance control / optimization Future Work: From System observations to analysable Model.