Scalable Systems Software Center Resource Management and Accounting Working Group Face-to-Face Meeting June 5-6, 2003

Resource Management and Accounting Working Group Working group scope Progress over last quarter Next steps Topics for group consideration

Working Group Scope The Resource Management Working Group is involved in the areas of resource management, scheduling and accounting. This working group will focus on the following software components: Queue Manager Scheduler Accounting and Allocation Manager Meta Scheduler Other critical resource management components are being developed in the Process Management and Monitoring Working Group: Process Manager Cluster Monitor

Proposed Component Architecture Queue Manager Allocation Manager Node Monitor Meta Scheduler Local Scheduler Node Manager Process Manager Security System Information Service Discovery Service Color Key Working Group Resource Management and Accounting Execution Management and Monitoring Node Configuration and Infrastructure Infrastructure Services Event Manager

Resource Management Prototype Demonstration Queue Manager Allocation Manager Node Monitor Local Scheduler Process Manager Discovery Service Color Key Working Group Resource Management and Accounting Execution Management and Monitoring Node Configuration and Infrastructure Job Submission Client 1 Submit-Job 3 Query-Node 6 Exec-Process 4 Create-Reservation 2 Query-Job 5 Run-Job 8 Delete-Job 0 Service-Lookup 7 Query-Job 9 Withdraw-Allocation This demo runs a simple end-to-end test with a job being submitted running past it’s wallclock limit

General Progress SSS front-end created for QBank Ready for re-release of v1.0 Initial SSS Resource Management Suite –OpenPBS-SSS sss_xml front-end –Maui Scheduler –QBank sss_xml front-end Created Job Object Specification version 2.0 –Takes into account all stages of a job’s lifecycle –Support for job steps, preferences, request choices, charging, meta- scheduling, dynamic jobs, multi-task jobs, awareness policy –Distinguishes between requested, utilized and dedicated properties

General Progress Completed version 2.0 of the SSSRMAP resource management interface specification –Includes specifications for authentication and encryption –Has been implemented by Gold Accounting and Allocation Manager (as a proof of design) Beginning to see adoption of SSSRMAP specification –Commitment from SLURM (LLNL) to write interface to SSSRMAP –Commitment from Cluxterworx (Linux Networx) to write interface to SSSRMAP –Interest from bproc-based scheduler (Clemson University) to interface to queue manager via SSSRMAP –CLUBMask resource manager (Penn State) to interface with scheduler via SSSRMAP –Interest from CERN for a data manager interface to scheduler using SSSRMAP

Scheduler Progress Implemented XML client-server interface (40% of clients now using SSSRMAP) New interfaces to support generic resource loads (paging space, I/o, processor load, etc) for resource limit enforcement and tracking Documentation on resource limit enforcement and tracking Added support for multi-task group jobs Support for dynamic reservations (growing and shrinking to support MPI dynamic jobs)

Scheduler Progress Security -- support for a user specified keyfile containing the security token Performance -- continued efforts in memory- footprint reduction Fault tolerance – implemented a fallback server Ease of use -- Initial web-GUI developed (communicates directly with Maui server)

Queue Manager Progress Updated service directory and event manager interfaces Implemented caching of service directory lookups and prioritizing the wire protocol types returned for fault tolerance and performance. Beginning implementation of SSSRMAP v2 wire protocol and xml specification.

Accounting and Allocation Manager Progress Gold –Added support for 95% of functionality from QBank –Allocation design enhancements allocations shareable by users, projects and machines (also supports exclusions) Special wildcard types (ANY, NONE, MEMBER, DEFINED) Enhanced support for activation and expiration times (& active state) –Support added for Deposits Use of deposit shares for non-interactive deposit defaults –Support added for Hierarchical accounts (projects) Affected withdrawals, deposits, reservations, balance checks, etc. Support for recursive trickle up withdrawals and trickle down deposits

Accounting and Allocation Manager Progress Gold –Support added for Refunds –Implemented Guaranteed Quotes –Implemented Transfers –Support added for debit vs. credit allocations –Support for operations (aggregate functions) on returned query fields (sort, sum, max, unique, count, group by, etc) –Negation of options –Association metadata added to aid in GUI object navigation –Enhanced support for transaction logging, journaling, undo, redo –Implemented more flexible charging algorithm

Accounting and Allocation Manager Progress Gold –Implemented SSSRMAP version 2.0 –Implemented SSS Job Object version 2.0 –Infrastructure added for Role-Based Access Control –Support added for method overriding and method scope resolution –Progress on open source front (Gold and sss_xml front-ends) obtained approval from PNNL IP to apply a BSD open source license Sent letter to Fred requesting DOE approval to assert copyright –Created Accounting and Allocation Manager Binding document describing use of SSSRMAP protocol –Beginning effort to develop Web-based GUI (JSP) –Implemented SSSRMAP v2 authentication –Almost completed implementing SSSRMAP v2 encryption

Meta-Scheduler Progress Added basic data scheduling! (tested with Globus) Created interface for data-cache scheduling Fault tolerance improvements –job queue is persistent –Will recover from network failure, system failure, loss of checkpoint files Major documentation in all areas

Future Work Implement v2 SSS Resource Management and Accounting interface specification (all components) Implement v2 Job Object Specification Implement default SSSRMAP v2 security authentication and encryption for all components Release v1.0 Initial SSS Resource Management Suite and improve download and documentation webpages Release Portability enhancements (AIX, Tru64, possibly Cray) Create per-component interface specification documents (binding to SSSRMAP) Draft Design Specification documents

Future Work Local Scheduler Test interaction with checkpoint/restart mechanisms when interfaces ready Continued work on resource limit enforcement and tracking quality of service support for completion time guarantees Implement SSSRMAP v2.0 Security integration (authentication and encryption) Support for maleable jobs (pre-execution) Abstracting resource manager interfaces to accept multiple sources of input data and control Enable simulation to live submission translator

Future Work Queue manager Implement persistence via database (replacing flat files) Add Epilogue/Prologue support and job submission verification script Interface with Node Monitor Full PBS qsub compatibility Implement full input/output handling (need to define PM interfaces, if any) Implement SSSRMAP v2.0 (including security)

Future Work Accounting and Allocation manager Implement SSSRMAP v2 encryption (and test authentication) Implement Role-Based Access Control (fine- grained command authorization) Integration with Directory Service Open source gold (BSD license) Progress on Web-based JSP GUI

Future Work Meta Scheduler Continued effort in allocation management, credential management, data management, Enablement of grid level prioritization and fairness policies

Issues requiring inter-group discussion