LARGE SCALE DEPLOYMENT OF DAP AND DTS Rob Kooper Jay Alemeda Volodymyr Kindratenko

The need for scaling How can we scale? How can DAP architecture scale? How can DTS architecture scale? What options do we have to scale? Amazon solution for scaling XSEDE solution for scaling Cloud solution for scaling

Finite Resources CPUMemoryDiskNetwork

Scalability A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.

Scaling Up And Out Scale UP (vertically) Adding resources to a single system “Speed” Performance Moor’s Law Scale OUT (horizontally) Cloud Adding nodes to the system Nodes can be commodity hardware (vs HPC) Increase software complexity Increase management complexity

Elasticity Need ability to grow/shrink on demand Based on workload add or remove resources Keep requirements small If many people use one service bring up more of those Don’t bring up services that people don’t use

Software Server Architecture Software Server Image Magic Image Magic Open Office ffmepg 3D Studio … … Polyglot Unknown Format Data Unknown Format Data Useable Data Useable Data

Medici 2.0 Architecture Frontend Webapp Load Balancer MongoD B HTTP HTML JSON HTTP HTML JSON External Services Frontend Webapp Frontend Webapp Event Bus (rabbitMQ) Extractor (Java) Extractor (Python) Elastic search Filesystem …… MongoD B Elastic search Elastic search

How to grow? More servers at ISDA Funding is in Brown Dog Not sustainable Commercial Clouds Amazon, … XSEDE NSF funded HPC computation NCSA Cloud infrastructure

AWS Web Application Reference Architecture

AWS Batch Processing Reference Architecture

Pricing Small machine (1CPU, 2GB) Linux $0.026 per Hour Windows $0.036 per Hour Server is approx. $10,000 and can hold 20 VMs Average lifespan 5 years (~ $500 per VM) Equals around 2 years of Amazon time But cheaper if we only need it 8 hours per day! And 7 hours/day in case of windows.

XSEDE Resources Jay Alameda National Center for Supercomputing Applications 23 July 2014

What is XSEDE Integrating service for wide variety of High Performance Computing (HPC) and Visualization and Data Analysis (RDAV) resources – Front line support – Uniform documentation – Extended collaborative support – Training, education and outreach services – Allocations

Variety of HPC and RDAV resources Dynamic list at erview erview – Overview, and expiration dates for each resource – Traditional clusters – Visualization and data analysis resources – Storage resources – High throughput resources – Testbeds – Services

Potentially Interesting Resources for Browndog Testbed resource “FutureGrid” – Production through 9/30/2014 – Partitioned into HPC Infrastructure as a Service (IaaS) – Nimbus – Openstack – Eucalyptus Dedicated – Layer Platform as a Service (PaaS) (eg, MapReduce, Hadoop) on top of these partitions

Potentially Interesting Resources for Browndog - 2 Service resource “Quarry” – Web service hosting environment – Resource end date not specified – Available for XRAC allocations with web-service component Storage: either NSF home directories, or lustre based storage. – OpenVZ provides virtual hosting of RPM based linux distributions – Persistent virtual machine

New XSEDE Resource: Comet Long-tail science system hosted at San Diego Supercomputer Center Builds on experience with SDSC Gordon (flash memory, persistent storage nodes), and SDSC Trestles (long-tail science) – 99% of jobs in 2012 used < 2048 cores – These jobs consumed half of the total core hours across NSF resources.

Comet Partially designed to pick up FutureGrid use (virtual clusters) Gateway hosting nodes and virtual machine repository Optimized for jobs within a rack Continues access to flash memory (Gordon) Capacity computing: computing for the 99% of XSEDE jobs

Comet virtualization Leverage experience and expertise from FutureGrid Virtual machine jobs scheduled like batch jobs Flexible software environments for new communities and applications Virtual machine repository Virtual HPC cluster (multi-(whole)-node), miminum latency and overhead penalty

XSEDE and BrownDog Premise: BrownDog will become an integral part of a researcher’s workflow Question: Should BrownDog evolve into an XSEDE resource provider, to provide data services for XSEDE?

National Center for Supercomputing Applications University of Illinois at Urbana-Champaign ISL Resources Volodymyr Kindratenko Innovative Systems Laboratory

Hadoop

OpenStack Cloud

Virtual Lab for Advanced Design

High memory node Dell PowerEdge R920 CPU Intel Xeon E7-4860v2 2.6 GHz (4) RAM3 TB Storage 2x 300 GB 10,000 RPM SAS 6 Gbps HDD 4x 800 GB SAS Read-Intensive MLC 12 Gbps SSD 6x 1 TB 7,200 RPM Near-Line SAS 6 Gbps HDD Interconnect 6x 1 Gbps Ethernet 2x 10 Gbps Ethernet CPU0 CPU1 CPU2 CPU3 RAM PHC QPI PCIe, DMA

Other systems GPU Server 8 NVIDIA C2050 GPUs Intel Xeon Phi Server 2 Xeon Phi 7120 (Knights Corner) application accelerators HPC cluster 8 nodes