Component Infrastructure of CQoS and Its Application in Scientific Computations Li Li 1, Boyana Norris 1, Lois Curfman McInnes 1, Kevin Huck 2, Joseph Kenny 3, Meng-Shiou Wu 4 1 Argonne National Laboratory, Argonne, IL. 2 University of Oregon 3 Sandia National Laboratories, California 4 Ames Laboratory CCA meeting Jan. 2009

2 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work

3 Overall Goals Automate the configuration and runtime adaptation of high- performance component applications, through the so called Computational Quality of Service (CQoS) infrastructure –Instrumentation of component interfaces –Performance data gathering –Performance analysis –Adaptive algorithm support Motivating application examples –Quantum Chemistry challenges: How, during runtime, can we make the best choices for reliability, accuracy, and performance of interoperable QC components? When several QC components provide the same functionality, what criteria should be employed to select one implementation for a particular application instance and computational environment? How do we incorporate the most appropriate externally developed components? (e.g., which algorithms to employ from numerical optimization components?)

4 Motivating Application Examples (cont.) Overall simulation times for nonlinear (time-dependent) PDE-based models often depend to a large extent on the robustness and efficiency of sparse linear solvers –Properties of linear system change during runtime –No single method is best because of the complexity of long-running applications Efficient parallel structured adaptive mesh refinement (SAMR) applications depend on load-balancing algorithms –Computational resources are dynamically concentrated to areas in need of a high accuracy –Application and computer state change at runtime –Dynamic resource allocation requires the workload partitioning algorithm be selected at runtime according to state change

5 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work

6 CQoS Analysis Infrastructure Performance monitoring, problem/solution characterization, and performance model building Performance Databases (historical & runtime) Interactive Analysis and Model Building Substitution Assertion Database Substitution Assertion Database Scientist can analyze data interactively Scientist can provide decisions on substitution and reparameterization Instrumented Component Application Cases Instrumented Component Application Cases CQoS Control Infrastructure Interpretation and execution of control laws to modify an application’s behavior Control System (parameter changes and component substitution) Control System (parameter changes and component substitution) CQoS-Enabled Component Application CQoS-Enabled Component Application Component A Component B Component C Component Substitution Set Component Substitution Set

7 Database Needs for the Scientific Application Adaptation Performance analysis of candidate solver/algorithm –Large number of performance runs –Store, manage, and search performance data Store and manage hardware, compiler, and application metadata –Information essential to algorithm selection, e.g., system configurations, problem properties, application states Optimal algorithm determination –Input data (or problem features) –Algorithmic parameters –Performance models (or hints)

8 Database Needs for Scientific Application Adaptation (cont.) Database use cases: –Store historical performance data and application meta- data –Facilitate offline performance analysis –Match the current application state against historical data through DB queries during runtime –Search for optimal algorithm w.r.t. current application state –Retrieve settings associated with the optimal algorithm so we can apply it immediately to the application during runtime

9 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work

10 CQoS Database Component Design Designed C++ and SIDL interfaces for CQoS database management Implemented prototype database management components –Description and software: S_database_components_version_ S_database_components_version_0.0.0 –Based on PerfDMF performance data format and PERI metadata formats –Comparator interface and corresponding component for searching and matching parameter sets

11 CQoS Database Component Design Adaptive Heuristic Adaptive Heuristic Perf. Comparator Perf. data: compare/match Perf. Database Perf. data: query/store … : component : component connection Fig.1. Connect database and comparator components to adaptive heuristics component. There can be multiple database and comparator components that deal with different data types. Metadata: query/store Metadata: compare/match Meta-Comparator Meta-Database

Use DB interfaces in 2D driven-cavity /* instantiate parameter 1 */ ierr = ComputeQuantity(matrix,"icmk","splits",&res,&flg); CHKERRQ(ierr); MatrixProperty param1("splits", "matrix_meta", res.i); /* instantiate parameter 2 */ ierr = ComputeQuantity(matrix,"structure","nnzeros",&res,&flg); CHKERRQ(ierr); MatrixProperty param2("nnzeros", "matrix_meta", res.i); /**** Store matrix property set into database. ***/ int myRank; ierr = MPI_Comm_rank(PETSC_COMM_WORLD, &myRank); CHKERRQ(ierr); if (myRank == 0){ int localID; int trialID; string conninfo("dbname = perfdb"); /* Generate a runtime database manager. It connects to a PostgreSQL database through DB interfaces. */ RunTimeRecord *R = RunTimeRecord::instance(); R->Connect2DB(conninfo); trialID = R->getTrialID(); localID = R->getCurEvtID(cflStr); /* instantiate a parameter set */ PropertySet aSet; /* add parameter 1 and 2 into the set */ aSet.addAParameter(&param1); aSet.addAParameter(&param2); /* store the parameter set into database, */ R->loadParameterSet(trialID, localID, aSet); }

13 CQoS Performance and Metadata Performance (general) –Historical performance data from different instances of the same application or related applications: Obtained through source instrumentation, e.g., TAU (U. Oregon) Binary instrumentation, e.g., HPCToolkit (Rice U.) Ideally, for each application execution, the metadata should provide enough information to be able to reproduce a particular application instance. Examples: –Input data (reduced representations) E.g., molecule characteristics,matrix properties –Algorithmic parameters E.g., convergence level, maximum number of iterations –System parameters Compilers, hardware –Domain-specific Provided by scientist/algorithm developer

14 Outline Motivation CCA and CQoS introduction Database component design Application examples Ongoing and future work

15 1 J. Steensland and J. Ray, "A Partitioner-Centric Model for SAMR Partitioning Trade-Off Optimization : Part I," International Journal of High Performance Computing Applications, 2005, 19(4): Example: CQoS in Quantum Chemistry Initial focus: parallel application configuration of QC applications so that these can run effectively on various high-performance machines –Eliminate guesswork or trial-and-error configuration Future work: more sophisticated analysis to configure algorithmic parameters for particular molecular targets, calculation approaches, and hardware environments

Interactions of the Quantum Chemistry Components With the Database and Comparator CQoS Components

CQoS/QC Component Wiring

18 CQoS Component Usage in Quantum Chemistry CQoS database usage –Application metadata Molecule characteristics: atom types, topology, moments of inertia Algorithm parameters: tunable parameters, convergence level –System parameters Compilers Machine info, e.g., number of nodes, threads per node, network –Historical performance data Execution times, etc. Obtained through source instrumentation, e.g., TAU Can guide configuration of related new simulations CQoS comparator components –Compare sets of parameters within the performance database –Quantum chemistry applications can match the current application state against historical data through database queries during runtime. –Use metadata to guide parameter selection and application configuration Match molecule similarity, basis set similarity, electronic correlation approach, etc.

19 Ongoing and Future Work (Incomplete List) Integration of ongoing efforts in –Performance tools: common interfaces and data representation (leverage PerfExplorer, TAU performance interfaces, PERI tools, and other efforts) Support training experiment design –To perform an empirical search for selecting the optimal solver components/parameters Incorporate more offline performance analysis capabilities (machine learning, statistical analysis, etc.) Apply to more problem domains, implementing extensions as necessary

20 Acknowledgements to Collaborators TAU Performance Tools group, University of Oregon Victor Eijkhout, the University of Texas at Austin CCA Forum members Funding: –Department of Energy (DOE) Mathematical, Information, and Computational Science (MICS) program –DOE Scientific Discovery through Advanced Computing (SciDAC) program –National Science Foundation

interface DB extends gov.cca.Port{ bool connect(); bool disconnect(); bool isClosed(); void setConnectionInfo(in string info); string getConnectioninfo(); int executeQuery(in string commd, out Outcome res); void storeParameter(in int trialID, in int iterNo, in Parameter aParam); // store a parameter into DB void storeParameterSet(in int trialID, in int iterNo, in ParameterSet aParamSet); // store a set of parameter into DB void getParameter(in int trialID, int iterNo, inout Parameter aParam); // retrieve parameter value void getParameterSet(in int trialID, int iterNo, inout ParameterSet aParamSet);// retrieve parameter set value int getMatchingTrialsBetween(in ParameterSet lower, in ParameterSet upper, out Outcome trialIDs); // retrieve trials from database, whose parameter set value is within [lower, upper] int getMatchingTrials(in ParameterSet lower, in vector e psilons, out Outcome trialIDs);// retrieve trials from database, whose parameter set value is within [lower-epsilons, lower+epsilons] } interface Comparator extends gov.cca.Port { /* Comparison operations between parameter sets */ void setLHS(in ParameterSet lefthand); void setRHS(in ParameterSet righthand); ParameterSet getLHS(); ParameterSet getRHS(); int getDimension(); Parameter getLHSParameterAt(in string paraName); Parameter getRHSParameterAt(in string paraName); void setToleranceAt(in string name, in double epsilon); double getToleranceAt(in string name); void setRelationAt(in string name, in int aRelation); int getRelationAt(in string name); bool doCompare(); } Main Database Component Interfaces

22 Database Component Usage – Example 1: 2D Driven Cavity Flow 1 1 T. S. Coffey, C.T. Kelley, and D.E. Keyes. Pseudo-transient continuation and differential algebraic equations. SIAM J. Sci. Comp, 25:553–569, Linear solver: GMRES(30), vary only fill level of ILU preconditioner Adaptive heuristic based on: –Matrix properties (which change during runtime) computed with Anamod (Eijkhout,

23 How are the Database Components Used? During runtime, the driver (e.g., linear solver proxy component) evaluates important matrix properties, and matches the properties to historical data in MetaDB through PropertyComparator interfaces. Linear solver performance data is retrieved and compared given the current matrix properties. This is accomplished by the PerfComparator component. The linear solver parameters resulting in the best performance, in this case fill level of ILU preconditioner, is returned back to the driver. The driver adapts accordingly to continue execution.