technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen1 Performance Modelling of Complex Hardware/Software Systems B.D. Theelen

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen2 Overview Introduction System-Level Design Software/Hardware Engineering Case Studies Modelling and Validation Performance Evaluation Accuracy Analysis Summary

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen3 Introduction Designing a complex system within limited time involves taking important decisions in an early phase of the design process, which may have a deep impact on the performance of the system Early assessment of the impact of design decisions involves using system-level design methods and tools Design Time Concepts Requirements Implementation System-Level Design Component-Level Design (Automatic) System Implementation

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen4 Three Phases FormulationFormulation –Informal identification and specification of system concepts and design issues FormalisationFormalisation –Formal specification of behaviour and architecture –Validation of adequacy of formal system specification –Formal specification of the properties to evaluate EvaluationEvaluation –Property analysis and design decisions

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen5 Software/Hardware Engineering (SHE) UMLFormulation using (stereotyped) UML diagrams –Performed case studies: documents with plain texts and various diagrams POOSLFormalisation based on system-level modelling language POOSL –Intended for complex real-time distributed hardware/software systems –Small set of very expressive language primitives –Formal semantics concurrencytimeprobabilismDefines unambiguously how to handle concurrency, time and probabilism Evaluation Markov chain analysis –Performance evaluation founded on Markov chain analysis Analytic computation or estimation by simulation model checking –Formal verification of correctness properties based on model checking Tools creationvalidation –SHESim: Interactive creation and validation of POOSL models analysis –Rotalumis: Fast execution of POOSL models for analysis

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen6 Case Study: Network Processor (Deep) packet processing in telecommunication systems Utilisation of shared busses Identify possible bottlenecks Comparison POOSL - SystemC

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen7 Case Study: Dataflow System Part of Alcatel’s Internet Router Prepares/processes packets for correct routing Arbitration of memory accesses from 3 components Evaluate alternative arbitration mechanisms

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen8 Case Study: Internet Router Is the specified flow-control mechanism suitable for various product variants?

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen9 Executable Model Modelling guidelinesModelling guidelines (validation) –Readability (aggregate data objects), postpone optimising for execution speed –Log considerations on the adequacy of abstractions Modelling patternsModelling patterns –Point abstractions, abstraction from scheduling mechanisms, stream-based modelling, scalability of model, model refinement

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen10 Validation Validation guidelinesValidation guidelines –Perform step-by-step inspections –Analyse easy-to-check properties –Investigate impact of rare events –Consider symmetries in system –Recall considerations on adequacy of abstractions

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen11 Property Formalisation Two options for expressing performance metrics external observers –Performance analysis based on external observers: temporal rewards –Reflexive –Reflexive performance analysis: POOSL Extend model with additional behaviour to evaluate performance metrics Guidelines for reflexive performance analysis –Define extensions in subclasses –Use method overriding capabilities of POOSL Formal Semantics Understandable POOSL Model Markov Decision Process + Extensions for Performance Evaluation + Reward Structure External Scheduler Discrete-Time Markov Chain + Reward Structure

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen12 Performance Evaluation computationAnalytical computation: Ergodic Theorem simulationEstimation by simulation: Central Limit Theorem –Example: packet-loss probability confidence intervals –Enables accuracy analysis with confidence intervals automatic termination –Enables automatic termination of simulation when results are accurate

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen13 Accuracy Analysis For Internet Router example: a 99% confidence interval [ , ] is obtained for the average latency –Estimation of average latency is seconds –We are 99% sure that the real average latency lies in [ , ] –An upper bound for the relative error is 1% –Takes seconds of simulated real-time –Takes 11.9 hours of simulation time Central Limit Theorem only applicable for simple averages –Latency, packet loss probability algebra of confidence intervalsComplex averages and variances: algebra of confidence intervals –Defines set operations on confidence intervals –Enables accuracy analysis for range of different averages and variances –Jitter, buffer occupancy, processor utilisation, throughput, burstiness POOSL library classes for analysing different long-run averages

technische universiteit eindhoven PROGRESS 11 December 2002www.ics.ele.tue.nl/~btheelen14 Summary Three large case studies in corporation with industrial partners SHE method distinguishes three phases –Formulation based on informal UML diagrams –Formalisation with formal modelling language POOSL Guidelines for modelling and validation Modelling patterns Guidelines for reflexive performance analysis –Performance evaluation based on Markov chain analysis POOSL model implicitly defines Markov chain Extensions for performance analysis define reward structure Analytical computation or estimation by simulation Techniques for performance analysis –Confidence intervals allow automatic termination of simulation –Algebra of confidence intervals for accuracy analysis of complex metrics Library classes for accuracy analysis