Monsoon, NEXCS and access to MASS NCAS-CMS UM training course 13th December 2017 AJ Watling Monsoon Collaboration Tech Lead HPC & Collaboration Technology & Information Services

Contents Met Office Scientific Collaboration Overview Monsoon 2 and NEXCS External access to MASS Isambard Met Office Science Repository Service (MOSRS) Further info / Questions

The Team Scott Irvine – IT Manager: HPC & Collaboration Katie Kerr Monsoon / MOSRS Service Owner Katie Kerr Monsoon & External MASS Access Service Manager Roger Milton Scientific Collaboration Technical Lead AJ Watling Monsoon Technical Lead

What do we do? What’s available & Best Practice Help Scientists use IT to do Science collaboratively What’s available & Best Practice Liaison between Met Office’s Science & IT teams, and the Scientific community Develop / Extend Functionality

XCS – Top500.org - June 2017 - #11 (Nov 2017 - #15)

HPC resource for collaboration Year System Service Compute Nodes TFLOPS 2010 IBM Power 6 MONSooN 30 15 2012 IBM Power 7 96 68 2015 Cray XC-40 116 95 2017 Monsoon2 NEXCS 556 278 (/5786) 667 334

XCS – Security Model XCS spans three trustzones, which can be adjusted in HPC capacity. Research (XCS-R, available now) >70% of XCS compute resource. Collaboration (XCS-C, available now) Monsoon2 (8.4%, JWCRP: Joint Met Office / NERC Projects) NEXCS (4.2%, NERC only projects) Operational (XCS-O, possibly ~2018) Operational Suite only, ~10% resources Potentially run more than one parallel suite on XCE, XCF, or XCS-O

MONSooN - Met Office / NERC Superco(o)mputer Nodes Joint Weather and Climate Research Programme (JWCRP) objectives: Enable closer collaboration between Met Office and NERC scientists Develop activities to address gaps in national portfolio of weather and climate research Promote effective pull through of research to improved forecasts MONSooN: Running since October 2009 “To provide a shared supercomputing service in an environment within which collaborative projects between NERC and Met Office can be performed. The joint service is being provided to enhance collaboration between NERC and Met Office researchers and improve their ability to pull-through environmental science more rapidly.” Met Office is recognised as being experts in HPC and running our Unified Model. External collaborators have often had difficulty in working closely with Met Office, particularly with Model development; a lot of effort goes into getting a model to work in a new environment; on a different architecture. Much scientific staff time ‘wasted’ on this rather than looking at the science and/or developing better code. Some collaborators are able to come into the Met Office and work on our systems. That can’t work for everyone. MONSooN allows NERC and Met Office Collaborators to come together in a common environment; code, HPC and post-processing facilities. Each project has a different requirement of these aspects.

Monsoon 2 NERC & Met Office collaboration area on XCS Available from 14th March 2017 8.4% of XCS, 556 nodes from 5786 Need to be collaborating with Met Office scientists Applications are assessed for Technical & Scientific ‘Fit’ Around 30 projects, 200+ registered users

Monsoon - not just a Supercomputer Allocation of Supercomputer – Cray XC-40 Post-Processor – RedHat Linux, like Met Office’s Scientific Desktop Data transfers - Janet and JASMIN (each 1Gbps) Read/Write access to MASS data Facilities for code development Collaboration wiki server http://collab.metoffice.gov.uk Mere HPC size isn’t critical, though it is relevant – other facilities much larger (Archer, Prace); co-location has benefits in itself Same architecture as current Met Office Supercomputers – this *is* useful Same architecture as Met Office Supercomputers, but separate zone. 280TB of fast local storage (/home, /projects) 100 TB of local storage - medium term archive for NERC Post-Processing server (64 bit Red Hat Linux)

NEXCS – NERC on XCS New NERC-only area on XCS 4.2% of XCS, 278 nodes from 5786 First available April 2017 No need to be collaborating with Met Office scientists Enrolment and resource allocation – through Grenville Lister, NCAS-CMS

NEXCS – mostly just a Supercomputer No postprocessor No access to MASS Data transfer to JASMIN (or similar) using scp/sftp/rsync Would like GridFTP with certificates to automate transfer to JASMIN Other methods have been explored…

Monsoon2 and NEXCS – utilisation

Authentication/Accounts Monsoon Evolution Authentication/Accounts Too Many Accounts/Passwords! Collaboration Twiki, PUMA, MOSRS, Monsoon, etc… For Monsoon, currently two-factor authentication via RSA fobs Other options are available – Google Authenticate, Yubikeys Want to follow Met Office Strategy (developing area)

Monsoon Evolution Collaboration Tools Current Web server Introduced 2010 1500+ Registered Users 650+GB Data Twiki Static Web – allows MO FTP upload Newsgroup Facility Met Office moving to cloud-based Office 365, including Yammer Skype For Business Step change in functionality There will be gaps… (e.g. static web) which will need to be filled for Scientific Collaboration

MASS - Overview Main Met Office resilient tape-based archiving system 2 automated tape-libraries - duplexed copies HPSS library-management system Bespoke user-interface ‘MOOSE’ Numbers correct as of 19th January 2015

MASS - current stats ~180TB archived, ~36TB retrieved per day ~1,000,000 archive and 100,000 retrieve commands per week ~90% of retrieves served from 6PB disk cache ~22-23 day cache longevity 26,000 8 TB tapes, 13-14 TB compressed data per tape 230 PB tape capacity, 146 PB tape used (mostly duplexed) Numbers correct at of 28th March 2017

MASS access MASS JASMIN ECMWF HPC Other Met Office CDN Monsoon HPC (XCS-C) Met Office CDN HPC (XCE,F,S-R) Other MASS access Clients within the Met Office and MONSooN environments also have read and write access to MASS JASMIN had read only access to MASS Use the power of JASMIN when post-processing environments in MONSooN or within the Met Office are not appropriate/powerful enough Other data can be imported to MASS via manual methods by Met Office staff, not using MOOSE

Monsoon, ECMWF, and JASMIN MASS access allows you to… Have access based on a MASS project basis List contents of data-sets Retrieve files Filter files (‘atomic access’) Get information about data-sets: ownership quality-assessments, comments etc.

Monsoon, ECMWF, and JASMIN MASS access allows you to… Monsoon and ECMWF MASS access allows you to: Store and/or overwrite files JASMIN MASS access does not allow you to: Store or overwrite files Only Met Office dataset owners can: Move, rename or delete files Change metadata about files or data-sets

MASS Access from JASMIN JASMIN MASS access MASS Access from JASMIN

Access to MASS from JASMIN You will need... JASMIN Account - available here https://accounts.jasmin.ac.uk/ One machine (mass-cli1) has a MOOSE client installed, not accessible by default. Request access from JASMIN Team here https://accounts.jasmin.ac.uk/services/additional_services/mass/

Access to MASS from JASMIN Also… Met Office MASS account for accessing from JASMIN – requires: A sponsor for each user – who should be a Met Office Senior Scientist A list of MASS projects or datasets for the user to access Permission of the dataset owner to make it available externally

Isambard - overview Cray CS400 in IT Hall 2 Provides multiple advanced architectures within the same system Enables evaluation and comparison across a diverse range of hardware platforms See http://gw4.ac.uk/isambard/

MOSRS – Met Office Science Repository Service What is it? Subversion code repositories UM 10.0 – 11.0 JULES, NEMO, UKESM, LFRic, NAME, and many more Rosie suite repository & database Trac environments inc. documentation & commentary for UM development documentation for each project Project list https://code.metoffice.gov.uk/trac/home/wiki/ProjectList

MOSRS - a common shared environment Makes collaboration easier Improved pull-through of science changes Developers work from common code base Share issue tracking, development Improves access to new releases More up-to-date models at collaboration sites

MOSRS – current usage 1st June 2017 - 1,500th user registered 31st October 2017 - UM 10.9 - Commit #45,850

Further Info / Any questions? Monsoon - http://collab.metoffice.gov.uk/ Isambard - http://gw4.ac.uk/isambard/ MOSRS - https://code.metoffice.gov.uk/ MASS from JASMIN - http://help.ceda.ac.uk/category/227-mass monsoon@metoffice.gov.uk +44 (0) 1392 88 6256