technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen1 Performance Modeling in the Large: A Case Study B.D. Theelen

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen2 Contents Introduction Parallel Object-Oriented Specification Language (POOSL) Case Study: Internet Router Modeling the Internet Router Exhaustive Performance Analysis Performance Evaluation by Simulation Accuracy Analysis Conclusions

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen3 System-Level Design Methods Focus on developing executable models, which – Describe a system in the earliest phases of the design process – Abstract from many implementation details – Are relatively cheap to create, validate, test, debug and simulate Allow the analysis of system properties before the system is actually being realized in terms of hardware and software Correctness properties – Correctness properties: absence of deadlock, … Performance properties – Performance properties: throughput, packet-loss, jitter, … Enable to take well-founded design decisions based on the obtained system property analysis results Require the use of a well-defined modeling language Parallel Object-Oriented Specification Language (POOSL)

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen4 POOSL System-level modeling language for complex real-time distributed hardware/software systems very expressiveSmall set of very expressive primitives mathematically definedFormal (mathematically defined) semantics –Probabilistic features for performance analysis Process part based on real-time extension of CCS –Process objects model the elementary time-related behavior –Channels enable synchronous message passing between process objects –Clusters allow for describing hierarchy Data part based on imperative object-oriented languages –Data objects model data structures and operations on themwww.ics.ele.tue.nl/~lvbokhov/poosl

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen5 Alcatel 7770 RCP (Routing Core Platform) Internet Router 640 Gb/s – 10 Tb/s OC-3 – OC kW $

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen6 Overview of the Internet Router Is the specified flow-control mechanism suitable for future product variants? Inputs Outputs Sources Input BuffersOutput Buffers Sinks Switch Core 11 NN 1 N N 1 N 1 1 N M 1 M 1

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen7 System Parameters N and M SourcesNxM Source Activities Input BuffersN Input Dispatchers NxN Queue Output Handlers Switch CoreNxNSwitch Activities Output BuffersN Queue Input Handlers NQueue Output Handlers SinksN Sink Activities 2N 2 + (M+ 4) NActivities Huge number of concurrent activities

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen8 Scalability of the Model drawingInstantiating a process object is performed by drawing the object Modeling each component with a separate process object is not favorable for large N and M parameterizedsimilarInitialize parameterized number of similar concurrent activities Initialize()() InitializeActivity(1)(). InitializeActivity(Identity: Integer)() par Activity(Identity)()/* Invokes the actual Activity */ and if (Identity < N) then InitializeActivity(Identity + 1)() fi rap. (Optimization: dynamic creation and termination of concurrent activities)

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen9 Executable Model SHESimCreate executable model with SHESim tool –Freely available on:

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen10 Validation adequateIs the model adequate for analyzing the performance metrics? SHESimValidate, test and debug model using SHESim tool

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen11 Formalization of Performance Properties The flow-control mechanism should contribute to –Minimizing packet-loss –Minimizing buffer capacity –Minimizing latency –Maximizing throughput Performance metrics to analyze –Probability of losing packets –Average and maximum buffer occupancy –Average and maximum latency –Throughput Use POOSL formalism to extend model for performance analysis

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen12 Performance Evaluation with POOSL Formal Semantics Markov Decision Process & Reward Structure Understandable POOSL Model & Extensions for Performance Analysis Out!Packet(p) 1 S1S1 S2S2 S3S3 S4S4 External Scheduler Discrete-Time Markov Chain & Reward Structure Markov Chain is implicitly defined by expressive modeling language and not by hand

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen13 Performance Evaluation with POOSL Analytically: Ergodic Theorem for Markov Chains –Example: time-average buffer occupation Simulation: Central Limit Theorem for Markov Chains –Dependable random variables –Example: packet-loss probability –Enables accuracy analysis with confidence intervals

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen14 Performance Evaluation by Simulation RotalumisFast simulation with Rotalumis tool –Freely available on: Simulation = generation of one trace through Markov chain How long should a simulation take to obtain accurate results? confidence intervalsAnalyze accuracy of simulation results with confidence intervals Automatic termination of simulation

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen15 Accuracy Analysis For N=512 and M=128, a 99% confidence interval [ , ] is obtained for the average latency –Estimation of average latency is seconds –We are 99% sure that the real average latency lies in [ , ] –An upper bound for the relative error is 1% –Takes seconds of simulated real-time –Takes 11.9 hours of simulation time long-run averagesPOOSL classes for accuracy analysis of long-run averages –Long-run sample average: packet-loss probability, latency –Long-run sample variance: jitter –Long-run time average: buffer occupancy, processor utilization –Long-run time variance: buffer occupancy Rare events (losing a packet) may change the status of a metric

technische universiteit eindhoven 20 October 2001www.ics.ele.tue.nl/~btheelen16 Conclusions compact abstractreal-life systemsadequatelyperformancePOOSL allows for a compact abstract description of real-life systems to adequately investigate their performance properties Performance analysis requires extending the original model POOSL model implicitly defines a Markov chain computationFormal semantics enables computation of performance validationsimulationPOOSL tools offer good validation and simulation possibilities Confidence and accuracy of simulation results should be known accuracy analysisPOOSL library classes for accuracy analysis Future research –Long-run rate average (throughput) and rate variance (burstiness) –Rare events monitors –Support for performance metric monitors