UAB Dynamic Monitoring and Tuning in Multicluster Environment Genaro Costa, Anna Morajko, Paola Caymes Scutari, Tomàs Margalef and Emilio Luque Universitat Autònoma de Barcelona Paradyn Week 2006 March 2006

2 Outline Introduction Multicluster Systems Applications on Wide Systems MATE New Requirements Design Conclusions

3 Introduction System performance  New problems require more computation power. Performance is a key issue.  New wide systems are built over the available resources and the user does not have total control of where the application will run.  It became more difficult to reach high performance and efficiency for these wide systems.

4 Introduction (II)  To reach performance goals, users need to find and solve bottlenecks.  Dynamic Monitoring and Tuning is a promising approach.  With dynamic systems’ properties, efficient resource use is hard to reach even for expert users.

5 Multicluster Systems  New systems are built using existing resources. Examples are NOW and HNOW linked with multistage network interconnections.  Intra cluster communications have different latencies than inter cluster communications.  Generally multiclusters built of clusters (homogenous or heterogeneous) interconnected by WAN.

6 Multicluster Systems (II) Each cluster can have its own scheduler and can be exposed either through a head node or by all nodes

7 Applications on Wide Systems Hierarchical Master/Worker Applications  Raise the possibility of performance bottlenecks Load imbalance problems Inefficient resource use Non-deterministic inter cluster bandwidth Worker Master Sub Master Sub Master explores data locality Common data are transmitted once Cluster A Cluster B

8 Applications on Wide Systems (II) Hierarchical Master/Worker Applications  Sub master is seen as a high processing node by the master.  Work distribution from master to sub master should be based on: Available bandwidth Computing power  These characteristics may have dynamic behavior.

9 MATE  Monitoring, Analysis and Tuning Environment Dynamic automatic tuning of parallel/distributed applications. Modifications Instrumentation User TuningMonitoring Tool Solution Problem / Performance analysis Performance data Application development Application Execution Source Events DynInst

10 Machine 3 Machine 2Machine 1 MATE (II) Analyzer AC instr. events modif. events DMLib Task 1 Task 2 Task 3 instr. AC Application Controller - AC Dynamic Monitoring Library - DMLib Analyzer

11 MATE (III)  Each tuning technique is implemented in MATE as a “tunlet”, a C/C++ library dynamically loaded to the Analyzer process. measure points – what events are needed performance model – how to determine bottlenecks and solutions tuning actions/points/synchronization - what to change, where, when Analyzer DTAPI Tunlet Performance model Measure points Tuning point, action, sync Tunlet Performance model Measure points Tuning point, action, sync

12 New Requirements Transparent process tracking  AC should follow application process to any cluster. Lower inter cluster instrumentation communication overhead  Inter cluster communications generally have high latency and lower bandwidth.

13 Transparent process tracking System Service  Machine or Cluster can have MATE enabled as daemon that detects startup of new processes. MATE Enabled Machine AC MATE Enabled Machine AC Task n startup detection MATE Enabled Machine DMLib AC Task n attach control receives Analyzer information Analyzer subscription DESIGN

14 new ‘Task’ Transparent process tracking Application plug-in  AC can be binary packaged with application binary. DMLib AC Task DMLib AC Task Remote Machine DMLib Remote Machine AC Task n detects Dyninst create control Analyzer subscription Job submission new ‘Task’ create DESIGN (II)

15 Lower communication overhead Smart event collection  Total application trace may generate much overhead. Event aggregation  Remote trace events should be aggregated to trace event abstractions, saving bandwidth. Inter Cluster Trace Event Routing DESIGN (III)

16 Analyzer Approaches Centralized  Requires tunlets modification to distinguish instrumentation data of local application processes. Hierarchical  Requires tunlets dismembering into local tunlets and global tunlets. Distributed  Requires that tunlets instances located on different Analyzer instances cooperate to tune an application.

17 Machine B3Machine B1 Machine B2 Machine A3 Machine A2 Machine A1 Lower communication overhead (II) Centralized Analyzer Approach Analyzer AC Task 1 Task 2 Task 3 AC Task 1 Task 4 Task 3 AC Task 2 Event Router Cluster BCluster A DESIGN (IV)

18 Machine A4 Global Analyzer Machine B2 Local Performance Model Analysis Hierarchical Analyzer Approach Abstract Events Machine B3Machine B1 Machine A3 Machine A2 Machine A1 Local Analyzer AC Task 1 Task 2 Task 3 AC Task 1 Task 4 Task 3 AC Cluster BCluster A Local Analyzer DESIGN (V)

19 Distributed Monitoring, Analysis and Tuning Environment Distributed Analyzer Approach Cluster ACluster B Machine B2 Machine B3Machine B1 Machine A3 Machine A2 Machine A1 Analyzer AC Task 1 Task 2 Task 3 AC Task 1 Task 4 Task 3 AC Cluster BCluster A Analyzer Tunlet instances cooperation DESIGN (VI)

20 Conclusions and future work  Conclusions Interference of instrumentation information on inter cluster communication should be minimal. Process tracking enables MATE for multicluster systems. Centralized Analyzer approach benefits tunlet developer but does not scale. Distributed Analyzer approach scales but requires different model based analysis.

21 Conclusions and future work (II)  Future Work Development of new tunlets for distributed and hierarchical Analyzer approach. Tuning based only of local instrumentation data. Semantics of aggregation for Instrumentation events. Patterns of distributed tunlets cooperation. Scenarios of distributed Analyzer cooperation in multiclusters.

22 Thank you…