Networking updates: CHECO Marla Meehl UCAR/FRGP/BiSON/WRN/RMCMOA/RMRISC Manager NETS/FRGP/BiSON 2016-04-27

Acronym soup University Corporation for Atmospheric Research (UCAR; http://www.ucar.edu) National Center for Atmospheric Research (NCAR) Front Range GigaPoP (FRGP) Bi-State Optical Network (BiSON) Western Regional Network (WRN) Rocky Mountain Cyberinfrastructure Mentoring and Outreach Alliance(RMCMOA) Rocky Mountain Regional Internet Security Collaboration (RMRISC) Questions and discussions throughout as needed – and as for acronym soup ingredients, please ask if I miss one or check decoder page at http://bit.ly/acronym-soup

Current FRGP voting participants - 11 Colorado School of Mines Colorado State University - Fort Collins National Oceanic and Atmospheric Administration - Boulder State of Colorado University Corporation for Atmospheric Research (UCAR/NCAR) University of Colorado - Boulder University of Colorado - Colorado Springs University of Colorado - Denver University of Denver University of Northern Colorado University of Wyoming

FRGP participants – non-voting - 20 Auraria Higher Education Center Colorado College Colorado Community College System City of Boulder Colorado Department of Higher Education Colorado Mountain College Colorado State University - Pueblo Colorado Telehealth Network Community College of Denver Educational Access Gateway Learning Environment Network Johnson & Wales University Fort Lewis College I2-USDA JeffcoSD Metropolitan State University of Denver University Information Systems (CU system) NEON NREL State of Wyoming University NAVSTAR Consortium (UNAVCO)

Front Range GigaPoP (FRGP) http://www.frgp.net Continued and expanded LANDER support and partnership PerfSonar in the FRGP continues to expand and be a valuable tool 31 participants CU-CS, CC, CSU-P, JeffcoSD - fully active on BiSON and FRGP at 10G USGS - in discussions on fiber and TIC options SHEPC - in discussions on fiber build City of Golden - in discussions on fiber connectivity Pikes Peak Library District (PPLD) - pending approval

FRGP Cont’d 910 15th Level3 fiber leases Adding NOAA TIC colo - May 2016 CDOT fiber spliced to expand BiSON ring to Golden Level3 910 5 year colo agreement including cross connects signed Level3 fiber leases All paths renewed for 10 years (Boulder/Denver, DREAM, Boulder/Longmont, Center Green) Continued steady growth in all traffic (peering, commodity, I2, caching, etc.) Hiring two Network Engineer IIs

FRGP Cont’d Peering Member port maximum and 95th percentiles Google 2x10G running full so may increase in the fall Comcast - little progress on 10G and peering TR/CPS – still largest single FRGP usage Akamai - continues high utilization Upgrading servers Netflix - continues high utilization Coresite/Any2 – upgraded to 2x10G Direct peering with Syringa Open to other peering opportunities Member port maximum and 95th percentiles Adding aggregate port size comparison

Statistics 2015 FRGP commodity traffic 2007-2015 FRGP Peering Traffic 2010-2015 FRGP "Other" traffic 2010-2015 FRGP total traffic including caching 2014-2015 FRGP total traffic (including caching) 2014 - 2015

Bi-State Optical Network (BiSON) Total BiSON Participants = 9: CC (.167), CSM (1), CSU-FC (1), CSU-P (.167), CU-B (1), CU-CS (.167), FRGP (.5), I2-USDA (.5), JeffcoSD (1), NREL (.5), NOAA (1), UCAR (1), UW (1) Expanded BiSON to: Jefferson County School District – Full ring complete CSM – Full ring complete CU-UIS – in design discussions ADVA 10G Research Wave for CSU-P - pending installation UCAR NWSC 100G Wave on order CDOT Higher Ed agreement in review US 36, US 93, I25 North, I25 CSU Ag, I25 South/US 50

Internet2 Marla co-chairing I2 Diversity Initiative – a session planned at I2 Global Summit Survey out to CIOs Working on expanding efffort - webinars, workshops, blogs Awards for Global Summit announced - CSM awarded Non-Member NET+ Program Webinar recording Slides

Internet2 Tempe Future Strategies Meeting This was a meeting of the Internet2 connector/network member principals to engage in a discussion about the strategic direction of Internet2 Network Services Level setting about what the community faces in the next two years What the community will need through 2023 Marla and Jeff attended The following themes were identified from RON papers Improve Network Security and Operations Member Engagement and Membership Expansion Enhance research support Expand the network and Improve general broadband access Deliver shared/collaborative services

WRN WTC and WPC calls as needed 100G I2 ports in LA, Seattle, and Chicago Investigating further commodity sharing/aggregation Considering full 10G to Level3 Make sure TSIC and Level3 can back each other up Pacific Wave expanded to include WRN Pete attended meeting to discuss expansion

The Quilt Quilt continues to grow Next CIS RFP 2016 The Quilt now at 46 members 36 participants 10 affiliates Next CIS RFP 2016 Authorized Quilt Providers (AQPs) from the 2016 CIS RFP: CenturyLink Cogent Level 3 NTT America TATA Communications Telia Carrier (formerly called TeliaSonera International Carrier) Verizon

Quilt items Quilt strategic relationships Convened multiple meetings in Washington DC among Quilt leaders and FCC On-going relationship building with NSF FCC, NTIA, and NSF all had reps attend the last Quilt member meeting ESnet and NSF colocated meetings with Quilt in Austin in September 2015 - very successful Quilt security working group Quilt submitted regional security proposal to NSF Best practices

EAGLE-Net Partnered with Zayo to operate and manage network http://www.zayo.com/news/zayo-to-oversee-and- support-network-for-colorados-eagle-net-alliance-2/ Signed network operator agreement with Zayo (11/30/15) FRGP working in partnership on fiber and service opportunities Decommissioned Deproduction/Openmedia Foundation Transition ECBOCES to EN Zayo fiber opportunities E-rate

Rocky Mountain Cyberinfrastructure Mentoring and Outreach Alliance (RMCMOA) NSF CC*IIE Grant PIs/Organizations: Colorado State University (Pat Burns), UCAR (Marla Meehl), University of Colorado (Thomas Hauser) , University of Utah (Joe Breen), IRON (Michael Guryan – Senior Personnel)

RMCMOA Cont’d Third RMCMOA workshop held in conjunction with Westnet Meeting, 1/5/16 Agenda with presentations Well attended Well received per the survey Final RMCMOA Workshop 8/9 - 8/11, CSU in conjunction with RMACC - technical engineering focus Submitted SC ‘15 women funding supplement – WINS (more on that later) “ACI’s plan remains to have another CC* solicitation in 2016. I regret not being able to discuss an accurate estimate of a release timeframe, and I think the community will be asked to have just a little more patience with the process.”

CC*DNI Related Proposals Submitted - 3/24/15 Boise State - CI Engineer - AWARDED Boise State - Network Infrastructure - AWARDED CSM - CI Engineer - AWARDED CSU-P - Network Infrastructure - AWARDED NMSU - CI Engineer - AWARDED Operating Innovative Networks (OIN) - ESnet, I2, IU - AWARDED UH Mauna Loa - AWARDED UNM - Regional - AWARDED UW - CI Engineer - AWARDED CU-Boulder/Wyoming/NCAR DIBBS - NOT AWARDED

Women in IT Networking at SC (WINS) Submitted 3 year follow-on proposal to NSF Unsolicited Fund at least 5 women each year for 3 years Encourage awardees to attend in following years Attempt to acquire SC or vendor funding for returning awardees Build a pipeline that becomes self-sustaining Invitations to submit coming out soon Please consider having women in your organization apply

Rocky Mountain Regional Internet Security Collaboration (RMRISC) NSF CICI Regional Security Proposal PIs/Organizations: Colorado State University (Scott Baily/Christos Papadopoulos), UCAR(Marla Meehl/Scot Colburn/Paul Dial/John Hernandez), University of Colorado (Thomas Hauser/Erin Shelton) , University of Utah (Joe Breen), University of Wyoming (Dane Skow/Matt Kelly) Partners: CSM, ESnet, I2, The Quilt, UETN

Rocky Mountain Regional Internet Security Collaboration (RMRISC) During the two-year award term, the project team will conduct at least one regional workshop per year focused on securing network infrastructure. The project team will lead or partner with The Quilt, the Energy Science Network (ESnet), and Internet2 to present to the region quarterly focused security webinars. The project team will provide resources for research and education sites by developing online materials covering security best-practices, tools and other resources. On request, the project team will conduct up to eight security reviews and assessments during the project. The project team will also work closely with researchers at CSU to provide a real world testbed for deploying newly developed data analytics tools to detect, alert, and divert attacks.

Cheyenne (NWSC-2) HPC System Details Cheyenne will be built by Silicon Graphics International Corp. (SGI) in conjunction with centralized file system and data storage components provided by DataDirect Networks (DDN) 5.34 petaflops Cluster 4,032 2-socket nodes with 2.3 GHz Intel® processors Total of 145,152 cores (36 cores/node) 864 nodes have 128 GB mem and 3,168 have 64 GB of memory 313 TB total memory 100 Gbps EDR (Enhanced Data Rate) Infiniband interconnect Linux operating system Workload Manager Intel Parallel Studio XE compiler suite Cluster Management Fabric Manager

GLADE (NWSC-2) PFS Details Key features of the new data storage system:   The NCAR Globally Accessible Data Environment Over 20 petabytes of usable file system space Can be expanded to over 40 petabytes by adding drives 200 gigabytes per second  aggregate I/O bandwidth 3,360 x 8 TB NL SAS drives 48 x 800 GB mixed-use SSD drives for metadata 24 NSD (Network Shared Disk) servers Red Hat Enterprise Linux operating system IBM GPFS file system  

GLADE (NWSC-2) PFS Details The new data storage system will be integrated with NCAR’s existing GLADE file system It will provide an initial total capacity of over 36 petabytes (20 PB new and 16 PB existing GLADE space), expandable to over 56 petabytes with the addition of extra drives The new disk system will transfer data at the rate of 200 gigabytes per second, more than twice as fast as the current file system’s rate of 90 gigabytes per second

High Bandwidth Low Latency HPC and I/O Networks Current NWSC HPC Architecture Geyser, Caldera, Pronghorn DAV clusters GLADE Central disk resource 16.4 PB, 90 GB/s Yellowstone 1.50 PFLOPS peak High Bandwidth Low Latency HPC and I/O Networks FDR InfiniBand and 10Gb Ethernet NCAR HPSS Archive 160 PB capacity 51 PB current usage ~12 PB/yr growth Our current (familiar) environment is a 1.5 petaflop system (Yellowstone), with the 16 petabyte GLADE file system, backed up by the HPSS archive. Science Gateways RDA, ESG Data Transfer Services 10Gb Ethernet Remote Vis Partner Sites XSEDE Sites

High Bandwidth Low Latency HPC and I/O Networks NWSC HPC Architecture on January 2017 Cheyenne (NWSC-2) ~5.34 PFLOPS peak Geyser, Caldera, Pronghorn DAV clusters GLADE Central disk resource 36 PB, 200 GB/s Yellowstone 1.50 PFLOPS peak High Bandwidth Low Latency HPC and I/O Networks EDR / FDR InfiniBand and 40Gb Ethernet NCAR HPSS Archive 240 PB capacity 66 PB expected usage ~42 PB/yr growth By January 2017, we plan to put into production the NWSC-2 systems:    a. The new ~5 Petaflop NWSC-2 supercomputer    b. A ~20 petabyte augmentation of GLADE - with a 200 GB/sec aggregate        I/O bandwidth to the NWSC-2 supercomputer (over 2x what's currently        available with Yellowstone) One year overlap with Yellowstone Other Changes: * Added EDR InfiniBand network * Upgraded 10Gb ethernet network to 40Gb ethernet * Increased storage capacity, increased storage bandwidth Science Gateways RDA, ESG Data Transfer Services 10Gb Ethernet Remote Vis Partner Sites XSEDE Sites

Multi-Purpose Unified LAN Ethernet (MULE) Switch Juniper QFX10008 Switch - Multi-Purpose Unified LAN Ethernet (MULE) Switch Investigated other vendors and chose Juniper because: Largely Juniper shop Supports 288 100GE ports in the 8-slot QFX10008 The QFX100016 supports 16 slots, but that's more than we needed Happy with buffer capacity, which should give over 50 mS of line-rate buffer on all ports The QSFP28 optic should be more standard, dense, and flexible than the menagerie of optics offered by other vendors.

Multi-Purpose Unified LAN Ethernet (MULE) Switch Juniper QFX10008 Switch - Multi-Purpose Unified LAN Ethernet (MULE) Switch MULE serves as a bridge between EDR and FDR Infiniband networks QFX10008 modular switch with 8 slot capacity can host 240 100GE or 288 40GE ports Modules support QSFP and QSFP28 optics Modules: 30 x 100GE (QSFP28) or 36 x 40GE (QSFP) / 12 x 100GE or 60 x 10G (SFP+) Will use three 36x40GE modules and QSFP 40GE optics with four CWDM 10G lanes

Miscellaneous UCAR Presidential search still in-progress Implemented ARUBA wireless AP system Investigating clearpass for cert assignment NETS plans to use Aruba's ClearPass (TM) in several roles: a Certificate Authority that issues X.509 certificates to authenticate devices using EAP/TLS. This is best documented in the Clearpass-Guest-User-Guide-6.5.0, in the "Onboard" section. a RADIUS server that allows 802.1X authentication to Aruba Instant wireless Access Points via EAP/TLS. Configuration for this is described in the ClearPass Policy Manager 6.5 User Guide. CPPM, as ClearPass Policy Manager is often called, can be an OSCP (Online Certificate Status Protocol) server, so certificates can be revoked with relative ease.

Discussion Any follow up questions or feedback? Other items?