Research Duke CSG – Spring 2015 – Happy Valley Charley Kneifel

Where we were Duke Shared Cluster Resource (DSCR) In some ways – Research Computing was the Cluster Monolithic Cluster – Intel Chips, Common Image, Variety of capabilities Sun Grid Engine Scheduler Solid Scientific Support Staff Problematic Financial Model Irregular capital investments Stressed since 2008 economic downturn New financial model seen as betrayal of original agreement Eroded trust with faculty Breakaway/Splintered Cluster Faculty/Researchers spin up their own clusters instead of participating

Change Occurs New Provost New Vice Provost for Research Active Faculty Dissatisfaction New Director of Research Computing Support from CIO and all of the above to make a change

Where we are Duke Compute Cluster (DCC – renamed from DSCR) Monolithic Cluster – Intel Chips, Common Image, Variety of capabilities Starting on the road to flexibility Mixture of 1/10GB Networking SLURM Job Scheduler Solid Scientific Support Staff Reinvigorated Financial Model Cover 80% of the need on campus No annual fees for housing nodes Recurring investments in storage, computing, networking, special capabilities (GPU) CapEx to OpEx Breakaway/Splintered Cluster Faculty/Researchers spin up their own clusters instead of participating (but listening) Broadening Services Outreach and educational activities to researchers – mending fences/building bridges

Where we are going Flexible Duke Compute Cluster Multiple OS (accommodate individual researcher/group needs) Virtualized Secure Additional Computing Servers/Services Specialized Services (GPU) Large Memory Machines Flexible Storage Long Term Scratch High Performance (All SSD)

Where we are going (continued) Flexible Networking 10G Host Connections or Cisco FI 40G+ Interswitch/FI Connections 20G+ Storage Connections SDN Services SLURM Scheduler Solid Scientific Support Staff Faculty/Researchers want to come back to the middle

Challenges History, History, History Wall between health system and university Researchers in basic sciences in School of Medicine vs. Clinical Data Firewalls/Inspection Research Computing at Duke Research thought of this as the Cluster (DSCR) not the full suite of services

How Are We Going to Get There? Allocations/Vouchers from the middle Early Engagement with Researchers – Advisory Committee Matching Grants Software Defined Network Services Cooperation with the Colleges/Departments Startup Packages Support for custom packages/images/OS Support for Protected Network Researchers Secure, Perimeter-ized, Process driven network environment Hypervisor based delivery of HPC to Protected Network Outreach/Training Docker Days/1-5 Day Courses/ Meetings with Faculty

How Are We Going to Get There (cont)? DevOps Automation Work Flow Support Hadoop on Demand and similar services GUI for researcher to link things together Carrots such as subsidized storage, GPUs, large memory servers Cut-n-Pastable documents suitable for grant submissions Flexibility Variety of Supported OSes (including Windows) Variety of Patch Levels / Libraries Removal/Aging out of Old Hardware Carrots for replacement