Computing Experience… The ORNL Cluster Computing Experience… Stephen L. Scott Oak Ridge National Laboratory Computer Science and Mathematics Division Network and Cluster Computing Group December 6, 2004 RAMS Workshop Oak Ridge, TN scottsl@ornl.gov www.csm.ornl.gov/~sscott

OSCAR

Open Source Cluster Application Resources What is OSCAR? Step 8 Done! Step 1 Start… Step 2 Step 3 Step 4 Step 7 OSCAR Framework (cluster installation configuration and management) Remote installation facility Small set of “core” components Modular package & test facility Package repositories Use “best known methods” Leverage existing technology where possible Wizard based cluster software installation Operating system Cluster environment Administration Operation Automatically configures cluster components Increases consistency among cluster builds Reduces time to build / install a cluster Reduces need for expertise Step 6 Step 5

OSCAR Components Administration/Configuration HPC Services/Tools SIS, C3, OPIUM, Kernel-Picker, NTPconfig cluster services (dhcp, nfs, ...) Security: Pfilter, OpenSSH HPC Services/Tools Parallel Libs: MPICH, LAM/MPI, PVM Torque, Maui, OpenPBS HDF5 Ganglia, Clumon, … [monitoring systems] Other 3rd party OSCAR Packages Core Infrastructure/Management System Installation Suite (SIS), Cluster Command & Control (C3), Env-Switcher, OSCAR DAtabase (ODA), OSCAR Package Downloader (OPD)

Open Source Community Development Effort Open Cluster Group (OCG) Informal group formed to make cluster computing more practical for HPC research and development Membership is open, direct by steering committee OCG working groups OSCAR (core group) Thin-OSCAR (Diskless Beowulf) HA-OSCAR (High Availability) SSS-OSCAR (Scalable Systems Software) SSI-OSCAR (Single System Image) BIO-OSCAR (Bioinformatics cluster system)

OSCAR Core Partners Dell Indiana University IBM NCSA Intel Bald Guy Software RevolutionLinux Indiana University NCSA Oak Ridge National Laboratory Université de Sherbrooke Louisiana Tech Univ. November 2004

eXtreme TORC powered by OSCAR 65 Pentium IV Machines Peak Performance: 129.7 GFLOPS RAM memory: 50.152 GB Disk Capacity: 2.68 TB Dual interconnects - Gigabit & Fast Ethernet eXtreme TORC

HA-OSCAR: RAS Management for HPC cluster The first known field-grade open source HA Beowulf cluster release Self-configuration Multi-head Beowulf system HA and HPC clustering techniques to enable critical HPC infrastructure Active/Hot Standby Self-healing with 3-5 sec automatic failover time

Scalable Systems Software

Scalable Systems Software ORNL ANL LBNL PNNL SNL LANL Ames IBM Cray Intel SGI NCSA PSC SDSC Participating Organizations Problem Resource Management Accounting & user mgmt System Build & Configure Job management Monitoring Computer centers use incompatible, ad hoc set of systems tools Present tools are not designed to scale to multi-Teraflop systems Goals Collectively (with industry) define standard interfaces between systems components for interoperability Create scalable, standardized management tools for efficiently running our large computing centers * Project Coordinator: Al Geist (ORNL) * Collaboration among DOE labs, NSF supercomputing centers and Industry. * This shows lots of stuff but key points are: - participating orgs - working groups - summary of problem/goal/impact (mentioned already) Impact Reduced facility mgmt costs. More effective use of machines by scientific applications. To learn more visit www.scidac.org/ScalableSystems

Scalable System Software SSS-OSCAR Scalable System Software Leverage OSCAR framework to package and distribute the Scalable System Software (SSS) suite, sss-oscar. sss-oscar – A release of OSCAR containing all SSS software in single downloadable bundle. SSS project developing standard interface for scalable tools Improve interoperability Improve long-term usability & manageability Reduce costs for supercomputing centers Map out functional areas Schedulers, Job Mangers System Monitors Accounting & User management Checkpoint/Restart Build & Configuration systems Standardize the system interfaces Open forum of universities, labs, industry reps Define component interfaces in XML Develop communication infrastructure TJN: (11/5/2004) Condensed lots of stuff into just a few slides, draft#1

OSCAR-ized SSS Components Bamboo – Queue/Job Manager BLCR – Berkeley Checkpoint/Restart Gold – Accounting & Allocation Management System LAM/MPI (w/ BLCR) – Checkpoint/Restart enabled MPI MAUI-SSS – Job Scheduler SSSLib – SSS Communication library Includes: SD, EM, PM, BCM, NSM, NWI Warehouse – Distributed System Monitor MPD2 – MPI Process Manager TJN: (5/15/04) Notes: - SSSLib – also contains the Build & Configuration components (node state mgr, etc.) However, the Service Directory, Event Mgr, and underlying communication infrastructure are the primary aspects of this components.

Cluster Power Tools

C3 Power Tools Command-line interface for cluster system administration and parallel user tools. Parallel execution cexec Execute across a single cluster or multiple clusters at same time Scatter/gather operations cpush/cget Distribute or fetch files for all node(s)/cluster(s) Used throughout OSCAR and as underlying mechanism for tools like OPIUM’s useradd enhancements.

C3 Building Blocks cpushimage - “push” image across cluster System administration cpushimage - “push” image across cluster cshutdown - Remote shutdown to reboot or halt cluster User & system tools cpush - push single file -to- directory crm - delete single file -to- directory cget - retrieve files from each node ckill - kill a process on each node cexec - execute arbitrary command on each node cexecs – serial mode, useful for debugging clist – list each cluster available and it’s type cname – returns a node name from a given node position cnum – returns a node position from a given node name none

C3 Power Tools Example to run hostname on all nodes of default cluster: $ cexec hostname Example to push an RPM to /tmp on the first 3 nodes $ cpush :1-3 helloworld-1.0.i386.rpm /tmp Example to get a file from node1 and nodes 3-6 $ cget :1,3-6 /tmp/results.dat /tmp * Can leave off the destination with cget and will use the same location as source.

Motivation for Success!

RAMS – Summer 2004

Preparation for Success! Personality & Attitude Adventurous Self starter Self learner Dedication Willing to work long hours Able to manage time Willing to fail Work experience Responsible Mature personal and professional behavior Academic Minimum of Sophomore standing CS major Above average GPA Extremely high faculty recommendations Good communication skills Two or more programming languages Data structures Software engineering