1. The New Lonestar Dan Stanzione Deputy Director, TACC dan@tacc.utexas.edu 3/24/2011

2. What we’re after today. For the new user, an explanation of what supercomputing and Lonestar are all about For the experienced user, a quick rundown of the architecture and features of Lonestar For everyone, information on the allocation process

3. A note on using WebEx for Remote Users We will be monitoring the chat window throughout the session: Please use chat to ask a question, and someone will interrupt me. We have more than 40 people on the phone; would be best if you mute your line until you have a question!

4. TACC Mission To enable discoveries that advance science and society through the application of advanced computing technologies.

5. TACC Vision Provide the most powerful, capable computing technologies and techniques that enable people— researchers, educators, developers, engineers, businessmen, etc.—to advance science and society. Provide leadership in the advanced computing community in technology R&D, support, education, and expertise to ensure maximum impact of current and future technologies in diverse applications. Enable transformational science and societal achievements that change, influence, and improve our understanding of the world, and the world itself.

6. Enormous scale for science Lonestar, at 302 peak TeraFlops, is around the 25 th fastest supercomputer in the world by peak speed. It is the 3 rd largest system available to US University users (TACC’s Ranger is #2). Over the next few years, Lonestar will deliver more than 8 quintillion calculations to the science and engineering community (8 million trillion computations) –Your laptop could do this in 44,000 years.

7. From a Proud History of Lonestar Systems Lonestar-1: 1997, Cray T3E, 88 processors, 50 GigaFlops Lonestar-2: 2004 Cray-Dell cluster, first Terascale x86 cluster to the open science community, 6 TeraFlops. Lonestar-3: 2006-2010 Dell, 5,848 Intel Woodcrest processors, 62 TeraFlops. (1,200x the original Lonestar). Lonestar-4 began production last Tuesday.

8. Lonestar: The Stats (We’ll talk about what all this means later) 302 TeraFlops (Trillion floating point operations per second). 22,656 Intel Westmere Processor cores @3.33 Ghz in 1,888 Dell Blades. (12 cores per node) 2GB RAM per core, 24GB per node, 44.3 TB total. 40Gbps QDR Infiniband interconnect in a fully non-blocking fat tree (Mellanox). 1.2Petabytes of disk in parallel filesystem (DataDirect, Lustre)

9. Lonestar

10. Comparing Lonestar and Ranger Ranger’s peak speed is 579TF, Lonestar’s is 302TF. This measure is a theoretical performance peak for the *entire* system. No real code will deliver this (due to limits on memory bandwidth, code concurrency, etc). If you are *not* using the whole system at once, Lonestar will be faster than Ranger for small jobs

11. Specialized Subsystems Lonestar is a comprehensive system for science and engineering High Performance Computing High Throughput Computing Visualization subsystem Large memory subsystem GPU/Accelerated Computing Subsystem

12. Lonestar: The Partnership Lonestar is a powerful instrument for science, but what makes it unique is how the Lonestar project was formed. –NSF funded 1/3 rd of initial system. –UT-Austin funded another 1/3 rd. –Texas A&M, Texas Tech, and several UT research centers funded the final third. –UT-System then provided funding to expand another 30%! NSF joins with 17 institutions to build a system with capabilities no one could afford!

13. What is High Performance Computing? Lonestar is a highly parallel, distributed memory, Linux cluster. –Each node is a self-contained Linux computer. –Nodes are connected by a very high speed local interconnect. To use an HPC system (and go faster than your computer), your code needs to take advantage of parallelism and distributed memory. –Your Windows software will not work. –Your program that isn’t parallel will not go faster. –Your program that isn’t parallel will run out of memory.

14. Parallel Programming The most common method for building distributed memory, parallel code, is to use the Message Passing Interface, or MPI. Many widely used scientific codes, both open source and commercial, already use MPI. If you are developing your own code, we can help you learn MPI. –MPI works best with the Fortran and C/C++ language families.

15. Distributed Memory Programming

16. Parallel Programming (2) The *other* dominant parallel programming model is OpenMP. –OpenMP supports shared memory parallelism –This will work on Lonestar, but limited to one node (normally 12 processors). OpenMP/MPI hybrid programs are increasingly common, and will work well on Lonestar.

17. How do I use Lonestar? Get an account Get an allocation Use Secure Shell to access a login node (ssh is built in on Mac, Linux, many free clients available for Windows, i.e. Putty; ssh lonestar.tacc.utexas.edu). A login node is a single computer that talks to the public network. You can compile, edit, and transfer files on the login node. You can not compute there!! – You must use the batch queuing system to start a job on compute nodes

18. A Note on UNIX/Linux The normal mode of accessing Lonestar assumes you understand command-line access to *nix systems. We provide online and in-person training for those who are unfamiliar with Unix. –You can be productive without being an expert, but a few basic skills are required. Increasingly, users can access these systems through web portals (science gateways) or web service interfaces. –Most of these are designed for a specific discipline/project, not the general user (i.e. CIPRES)

19. Allocations In order to use Lonestar, or any TACC system you must get an allocation. An allocation determines how many resources you can consume. Allocations are typically measured in Service Units, or “SU”s. An SU is equivalent to 1 hour on 1 processor core on the system on which you are allocated.

20. Allocations(2) Each time you run a job, your allocation is docked by the number of SU’s you use. The “cost” of a given job is determined by the resources you keep someone else from using. –The minimum unit allocated is a node. Each LS node has 12 processor-cores. A job that takes 4 hours on 12 processors costs 48 SU’s A job that takes 4 hours on 8 processors also costs 48. The specialized nodes (i.e. nodes with GPUs, nodes with extra memory) have a higher SU charge rate. We will provide around 200 million SU’s per year on Lonestar to researchers like you.

21. Allocations(3) There are 3 types of allocations: Startup – up to 50,000 SU’s –Does not require a funded project. Instructional – up to 100,000 SU’s to support a course using HPC resources Research – 100,000 – millions (committee review for over 500k).

22. Allocations(4) This may seem complex, but if you are an active researcher at an eligible institution, we will in all likelihood give you an allocation. As your use becomes very large, you may not get as much as you want, but you will receive some. –Demonstrating you are making productive use of the time you are allocated will likely lead to more allocation!

23. How Do I get an Allocation Allocations at Texas A&M Allocations at Texas Tech Allocations at UT-Austin Allocations at other UT System institutions Allocations at any other college, university, or research institute. Corporate Allocations What if I need more?

24. Allocations at Texas A&M Apply to your local HPC Committee first: –https://sc.tamu.edu/forms/https://sc.tamu.edu/forms/ Once they approve your application, they will transmit your account request to us, and send you further instructions.

25. Texas Tech, UT-Austin, UT System Apply for a new account/allocation online at the TACC User Portal –Go to TACC New User Info for directions: http://www.tacc.utexas.edu/user-services/new-user- info/#allocation http://www.tacc.utexas.edu/user-services/new-user- info/#allocation Summary: –First, create a portal account at: https://portal.tacc.utexas.edu (click “new user”) https://portal.tacc.utexas.edu –If you are a PI, use this account to create an allocation (click “create new project” under “allocations”). –Once your project is approved, add users to your projects.

26. TeraGrid If you are not at one of the Lonestar partner institutions (or if you are, and want more allocation), you must apply through the TeraGrid. –Fill out a request on POPS: https://pops- submit.teragrid.org/https://pops- submit.teragrid.org/ –Find more info at the TG User Portal: https://portal.teragrid.org/ https://portal.teragrid.org/

27. Corporate or additional allocations Corporate users can join TACC’s STAR industrial affiliates program to gain access. –Membership will include access to training, consulting, and a startup allocation. –Additional allocations are available for a fee. Fees are used to expand the system to accommodate the additional demand. –Contact Melyssa Fratkin, mfratkin@tacc.utexas.edu for more details mfratkin@tacc.utexas.edu Individual researchers can also purchase additional allocation.

28. Allocations on other TACC Systems Allocations through TeraGrid are also available for TACC’s Ranger and Longhorn systems. 5% of Ranger cycles (20M SU’s/year) are set aside for Texas Higher Ed users. This is a large number of SUs. UT Austin users are eligible for additional allocations on Ranger, Longhorn, and Corral.

29. Lonestar: More Architecture Details Network layout, and why you care. Storage architecture, filesystems, and why you care. The specialized subsystems.

30. Lonestar Network Architecture In a parallel computer, the quality of the network has a *huge* impact on the overall performance. One of the primary differences between HPC and “cloud” offerings. In an MPI program, a good network can yield 500-1,000% performance improvement over commodity ethernet. –Due to both bandwidth and latency.

31. Lonestar Network Architecture Lonestar has a non-blocking fat tree with a ~2 microsecond latency. Our tree topology allows any pair of nodes to communicate even if 75% of core switches fail. Every blade chassis connected thru minimum latency path to 15 other chassis (3,072 cores)

32. Lonestar Filesystems You will see three primary filesystems on Lonestar (and all TACC systems): –/home ; An NFS filesystem with each user’s home directory. This is permanent storage for the life of your account, receives back ups, but is relatively slow, and has a small quota. –/scratch; Is very large and very fast, not backed up, for temporary files (quotas are very high, but unused files will be automatically deleted). –/work; medium large and medium fast; not backed up; for large datasets you need to use over and over again. Local scratch is also available on /tmp on each node.

33. Where do I keep my Data? Keep your code and configuration files in your home filesystem. Write your large scale data to /scratch while your are computing. If you need to preserve your data on /scratch, migrate it to the tape archive, Ranch. –rcp ${ARCHIVER}:$ARCHIVE/myfile $SCRATCH If you have a large dataset you need to broadly share, contact us about a Corral allocation.

34. Massive Computing Requires Massive Data Storage: Corral 1.2 petabytes of DataDirect Networks online disk storage Mulitple access mechanisms –MySQL & Postgres SQL databases –Lustre parallel filesystem –iRODS –Web-based access Can easily expand to petabytes Designed for hosting data collections!

35. ... and the Ranch Data Archival System 10 petabyte Sun StorageTek Silo –10,000 tapes –Used for long-term storage –Access provided to users of other TACC resources –Expanding to 100PB now

36. Specialized Subsystems We strive to make Lonestar a “one-stop shop” for your advanced computing needs. The “traditional model” of supercomputing has been to do large simulations on the supercomputer, then bringing your data back to your workstation for plotting, archiving, etc. We strongly believe in doing more of your science, whether simulation- or data- intensive, on the HPC system (moving data is expensive!).

37. Shared Memory System We are currently incorporating a set of nodes with 1TB of RAM into Lonestar. –True shared memory; no need to use MPI. –Will run serial programs unmodified with access to full memory. –Excellent platforms for large scale OpenMP codes. Each node is a quad-socket with Intel Nehalem EX processors (24 cores total), with 64 16GB memory DIMMS. –Processor, memory slower than “normal” compute nodes.

38. GPGPU Compute System GPU = Graphics Processing Unit. GPGPU = General Purpose GPU, or computing with graphics cards. The “original” GPGPUs were limited to single precision math and no error-correcting RAM; these problems are remedied in newer versions. We are incorporating nodes with nVidia M2070 GPU “Fermi” cards. Full support for CUDA programming model (including PGI Fortran CUDA compiler).

39. Remote Visualization. Graphics capable GPUs will also be available on a subset of nodes. Software support for hardware- assisted remote visualization (Visit, ParaView, TACC’s EnVision). Consulting support for remote visualization TACC’s Longhorn remote visualization portal will be extended to Lonestar.

40. High Throughput Computing We also support user’s who need to run very large numbers of serial (non-parallel) jobs. Use our “launcher” tools; –automated launching of many jobs using the same code for parameter studies, large scale data processing. –“module help launcher” HTC runs use the standard compute nodes.

41. What Software is Available on Lonestar? TACC strives to keep a comprehensive software stack on our systems. Everything you need to compile, build, optimize, analyze and debug scientific code. Many of the most widely used scientific applications. But… there are many thousands of scientific applications, and we can’t support them all.

42. Software(2) If the code you need is not currently installed, you can: –Install it yourself in your home directory. –If other user groups may need it, ask us to create a module for it (we support these on a best effort basis). –Ask us to help you configure it in your home directory.

43. Software(3) We use the “modules” command to set up particular packages in your environment –E.g. “module load intel” to make the Intel compiler your default. –Avoids conflicting paths or complex environment variables. –Type “module avail” to list the modules available for your current configuration. You can find the complete, current software list for Lonestar (auto-generated daily) at: http://www.tacc.utexas.edu/software_modules.php #_ http://www.tacc.utexas.edu/software_modules.php #_

44. The User Environment For comprehensive notes, take some TACC training courses! A couple of important notes: –Linux command line interface –“Modules” to select software packages. –Ibrun for launch of MPI jobs –My_affinity tools to optimize tasks on multicore nodes –SGE (Grid Engine) for batch submission…

45. User Environment(2) Compute jobs *can not* be run on the login nodes (no faster than a normal workstation!) Grid Engine is the resource management system to access the compute nodes. –Request number/type of nodes you need. –Specify how long you need to run. –Specify where your output files go. –Jobs typically described with a job script:

46. Sample Job Script #!/bin/bash #$ -N myMPI # Job Name #$ -j y # Combine stderr and stdout #$ -o $JOB_NAME.o$JOB_ID # Name of the output file #$ -pe 12way 24 # Requests 12 tasks/node, 24 cores total #$ -q normal # Queue name "normal" #$ -l h_rt=01:30:00 # Run time (hh:mm:ss) - 1.5 hours ibrun./a.out # Run the MPI executable named "a.out"

47. How do I learn More? The User Guide –http://services.tacc.utexas.edu/index.php/lonestar- user-guidehttp://services.tacc.utexas.edu/index.php/lonestar- user-guide Training Classes –April 11-12: Introduction to Parallel Programming on TACC Systems –Check full listings at: http://www.tacc.utexas.edu/user-services/training/ http://www.tacc.utexas.edu/user-services/training/ Online Training –http://www.cac.cornell.edu/ranger/http://www.cac.cornell.edu/ranger/ Advanced User Support

48. How do I get help? The User Portals –https://portal.tacc.utexas.edu/gridsphere/gridspherehttps://portal.tacc.utexas.edu/gridsphere/gridsphere –https://portal.teragrid.org/https://portal.teragrid.org/ –For most routine requests, use the portal to submit a ticket; you will be contacted quickly (tickets monitored 24x7). Password help, general inquiries, software installs, requests for consulting, allocation questions etc. Other inquiries –Contact Chris Hempel, User Services Director –Hempel@tacc.utexas.eduHempel@tacc.utexas.edu

49. Q&A Thanks for attending, and thanks for working with TACC for your scientific computing needs. TACC: Powering Discoveries that Change the World dan@tacc.utexas.edu