1. State of HCC 2012 Dr. David R. Swanson Director, Holland Computing Center

2. Nature Communications, July 17, 2012 Nebraska Supercomputing Symposium 2012

3. HCC CPU Hour Usage 2012 Nebraska Supercomputing Symposium 2012 Zeng (Quant Chem) 4.5M Starace (AMO Phys) 2.7M Rowe (Climate) 2.0M NanoScience 6.4M Comp Bio 3.0M Comp Sci 1.7M Physics 0.7M Mech E 0.4M

4. High Performance Computing http://t2.unl.edu/status/hcc-status Xiao Zeng, Chemistry, UNL (prior slide) DFT and Car Parrinello MD HPC – tightly coupled codes Requires expensive low-latency local network (infiniband) Requires high-performance storage (Panasas, Lustre) Requires highly reliable hardware Nebraska Supercomputing Symposium 2012

5. Eureka! A Higgs! (or at least something currently indistinguishable) "I think we have it. We have discovered a particle that is consistent with a Higgs boson." –CERN Director-General Rolf Heuer Nebraska Supercomputing Symposium 2012

6. US CMS Tier2 Computing Nebraska Supercomputing Symposium 2012

7. Compact Muon Solenoid (CMS) 5.5 mi Large Hadron Collider Nebraska Supercomputing Symposium 2012

8. CMS Grid Computing Model Nebraska Supercomputing Symposium 2012

9. Eureka! A Higgs! (or at least something currently indistinguishable) Ca. 50 PB of CMS data in entirety Over 1 PB currently at HCC’s “Tier2”, 3500 cores Collaboration at many scales –HCC and Physics Department –Over 2700 scientists worldwide –International Grid Computing Infrastructure –Data grid as well –UNL closely linked to KU, KSU physicists via a jointly hosted “Tier3” Nebraska Supercomputing Symposium 2012

10. Data Intensive HTC Huge database Requires expensive high-bandwidth wide area network (dwdm fiber) Requires high-capacity storage (HDFS, dCache) HTC – loosely coupled codes Requires hardware Nebraska Supercomputing Symposium 2012

11. Outline HCC Overview New User report HCC-Go Moving Forward (after break) –Next purchase –It’s the Data, stupid… –Other Issues Nebraska Supercomputing Symposium 2012

12. Outline New User report HCC-Go Moving Forward (next section) –Next purchase (motivation) –New Communities –PIVOT –It’s the Data, stupid… Nebraska Supercomputing Symposium 2012

13. HOLLAND COMPUTING CENTER OVERVIEW Nebraska Supercomputing Symposium 2012

14. HCC @ NU Holland Computing Center has a University-wide mission to –Facilitate and perform computational and data intensive research –Engage and train NU researchers, students, and other state communities –This includes you! –HCC would be delighted to collaborate Nebraska Supercomputing Symposium 2012

15. Computational Science – 3 rd Pillar Experiment Theory Computation/Data Nebraska Supercomputing Symposium 2012

16. Lincoln Resources 10 staff Red Sandhills 5,000 compute cores 3 PetaBytes storage in HDFS Nebraska Supercomputing Symposium 2012

17. Sandhills “Condominium Cluster” 44 nodes X 32-core, 128 GB, IB Lustre (175 TB) Priority Access –$HW + $50/month –4 groups currently SLURM Nebraska Supercomputing Symposium 2012

18. Omaha Resources 3 Staff Firefly Tusker 10,000 compute cores 500 TB storage New offices soon: 158J PKI Nebraska Supercomputing Symposium 2012

19. Tusker 106*64= 6784 cores 256 GB/node 2 nodes w/ 512 GB 360 TB Lustre –100 TB more en route QDR IB 43 TFlop Nebraska Supercomputing Symposium 2012

20. Tusker ¼ footprint of Firefly ¼ the power 2X the TFLOPS 2X the storage Fully utilized Maui/Torque Nebraska Supercomputing Symposium 2012

21. In between … HCC (UNL) to Internet2: 10 gbps HCC (Schorr) to HCC (PKI): 20 gbps Allows us to do some interesting things –“overflow” jobs to/from Red –DYNES project –Xrootd mechanism Nebraska Supercomputing Symposium 2012

22. HCC Staff HPC Applications Specialists –Dr. Adam Caprez –Dr. Ashu Guru –Dr. Jun Wang –Dr. Nicholas Palermo System Administrators –Dr. Carl Lundstedt –Garhan Attebury –Tom Harvill –John Thiltges –Josh Samuelson –Dr. Brad Hurst Nebraska Supercomputing Symposium 2012

23. HCC Staff Other Staff –Dr. Brian Bockelman –Joyce Young GRAs –Derek Weitzel –Chen He –Kartik Vedalaveni –Zhe Zhang Undergraduates –Carson Crawford –Kirk Miller –Avi Knecht –Phil Brown –Slav Ketsman –Nicholas Nachtigal –Charles Cihacek Nebraska Supercomputing Symposium 2012

24. HCC Campus Grid Holland Computing Center resources are combined into an HTC campus grid –10,000 cores, 500 TB in Omaha –5,000 cores, 3 PB in Lincoln –All tied together via a single submission protocol using OSG software stack –Straightforward to expand to OSG sites across the country, as well as to EC2 (cloud) –HPC jobs get priority; HTC ensures high utilization Nebraska Supercomputing Symposium 2012

25. HCC Model for a Campus Grid Me, my friends and everyone else Grid Campus Local 25 Nebraska Supercomputing Symposium 2012

26. HCC & Open Science Grid National, distributed computing partnership for data- intensive research –Opportunistic computing –Over 100,000 cores –Supports the LHC experiments, other science –Funded for 5 more years –Over 100 sites in the Americas –Ongoing support for 2.5 (+3) FTE at HCC Nebraska Supercomputing Symposium 2012

27. It Works! Nebraska Supercomputing Symposium 2012

28. HCC Networking Monitoring Nebraska Supercomputing Symposium 2012

29. OSG Resources Nebraska Supercomputing Symposium 2012

30. Working philosophy Use what we buy –These pieces of infrastructure are linked, but improve asynchronously –Depreciation is immediate –Leasing is still more expensive (for now) –Buying at fixed intervals mitigates risk, increases ROI –Space, Power and Cooling have a longer life span Share what we aren’t using –Share opportunistically – retain local ownership –Consume opportunistically – there is more to gain! –Collaborators, not just consumers –Greater good vs. squandered opportunity Nebraska Supercomputing Symposium 2012

31. Working philosophy A Data deluge is upon us Support is essential –If you only build it, they still may not come –Build incrementally and buy time for user training –Support can grow more gradually than hardware Links to national and regional infrastructure are critical –Open Source Community –GPN access to Internet2 –Access to OSG, XSEDE resources –Collaborations with fellow OSG experts –LHC Nebraska Supercomputing Symposium 2012

32. HCC New Users FY UNL- City UNL- East UNOUNMC Outside NU system 2011424 (74)33 (10)75 (19)30 (17)112 (26) 2012519 (95)50 (17)105 (30)35 (5)130 (18) Nebraska Supercomputing Symposium 2012

33. New User Communities Theatre, Fine Arts/Digital Media, Architecture Psychology, Finance UNMC Puerto Rico PIVOT collaborators Nebraska Supercomputing Symposium 2012

34. HCC NEW USER REPORT: HEATH ROEHR Nebraska Supercomputing Symposium 2012

35. HCC-GO : DR. ASHU GURU Nebraska Supercomputing Symposium 2012

36. MOVING FORWARD Nebraska Supercomputing Symposium 2012

37. NEW PURCHASE Nebraska Supercomputing Symposium 2012

38. $2M for … More computing –need ca. 100 TF to hit Top500 for Jun 2013 –Likely use all of funds to hit that amount More storage –Near-line archive (9 PB) –HDFS Specialty hardware –GPGPU/Viz –Mic hardware Nebraska Supercomputing Symposium 2012

39. More computing How much RAM/core? Currently almost always oversubscribed Large scale jobs almost impossible (> 2000 core) Safest investment – will use right away Firefly due to be retired soon – EOL Nebraska Supercomputing Symposium 2012

40. More computing Nebraska Supercomputing Symposium 2012

41. More Computing Nebraska Supercomputing Symposium 2012

42. More storage Most rapidly growing demand Growing contention, can’t just queue up Largest unmet need (?) Nebraska Supercomputing Symposium 2012

43. Storage for $2M $2M HDFS cluster –250 nodes –4000 cores (Intel) –9.0 PB (RAW) –128 GB / node Nebraska Supercomputing Symposium 2012

44. Other options GPGPUs most Green option for computing Highest upside for raw power (Top500) Mic even compatible with x86 codes SMP uniquely meets some needs, easiest to use/program Bluegene, Tape silo, … Nebraska Supercomputing Symposium 2012

45. HCC personnel timeline Nebraska Supercomputing Symposium 2012

46. HCC networking timeline Nebraska Supercomputing Symposium 2012

47. HCC cpu timeline 900X Nebraska Supercomputing Symposium 2012

48. HCC storage timeline 30,000X Nebraska Supercomputing Symposium 2012

49. Composite Timeline Data increase/ CPU Cores = 33 Data increase/ WAN bandwidth = 150 It takes a month to move 3 PB at 10 Gb/sec Power < 100X increase, largely constant last 3 years Nebraska Supercomputing Symposium 2012

50. Storage at HCC Affordable, Reliable, High Performance, High Capacity –Pick 2 –So multiple options /home /work /shared Currently, no /archive Nebraska Supercomputing Symposium 2012

51. /home Reliable Low performance –No W from workers ZFS Rsync’ed pair, one in Omaha, one in Lincoln Backed up incrementally, requires severe quotas Nebraska Supercomputing Symposium 2012

52. /work High performance High(er) capacity Not permanent storage Lenient quotas More robust, more reliable “scratch space” Subject to purge as needed Nebraska Supercomputing Symposium 2012

53. /share Purchased by given group Exported to both Lincoln and Omaha machines Usually for capacity, striped for some reliability Nebraska Supercomputing Symposium 2012

54. Storage Strategy Maintain /home for precious files –Could be global Maintain /work for runtime needs –Remain local to cluster Create /share for near-line archive –3-5 year time frame (or less) –Use for accumulating intermediate data, then purge –Global access Nebraska Supercomputing Symposium 2012

55. Storage strategy Permanent archival has 3 options –1) library –2) Amazon glacier Currently $120/TB/year –3) tape system Nebraska Supercomputing Symposium 2012

56. HCC Data Visualizations Fish! HadoopViz OSG Google Earth Web-based monitoring –http://t2.unl.edu/status/hcc-status/http://t2.unl.edu/status/hcc-status/ –http://hcc.unl.edu/gratia/index.phphttp://hcc.unl.edu/gratia/index.php Nebraska Supercomputing Symposium 2012

57. Other discussion topics Maui vs. SLURM Queue length policy Education approaches –This (!) –Tutuorials (next!) –Afternoon workshops –Semester courses –Individual presentations/meetings –Online materials Nebraska Supercomputing Symposium 2012

58. ©2007 The Board of Regents of the University of Nebraska NU Administration (UNL, NRI) NSF, DOE, EPSCoR, OSG Holland Foundation CMS: Ken Bloom, Aaron Dominguez HCC: Drs. Brian Bockelman, Adam Caprez, Ashu Guru, Brad Hurst, Carl Lundstedt, Nick Palmero, Jun Wang. Garhan Attebury, Tom Harvill, Josh Samuelson, John Thiltges Chen He, Derek Weitzel