1. Monsoon, NEXCS and access to MASS
NCAS-CMS UM training course
11th April 2018
Roger Milton
Scientific Collaboration Tech Lead
HPC & Operational Meteorology Team
Technology & Information Services
Met Office

2. Contents
Met Office Scientific Collaboration Overview Monsoon 2 and NEXCS External access to MASS Isambard Met Office Science Repository Service (MOSRS) Further info / Questions

3. The Team Scott Irvine IT Manager: HPC & Operational Meteorology
(Monsoon / MOSRS Service Owner)
Katie Kerr
Monsoon & External MASS Access Service Manager
Roger Milton
Scientific Collaboration Technical Lead
AJ Watling
Monsoon Technical Lead

4. What do we do? What’s available & Best Practice
Help Scientists use IT to do Science collaboratively
What’s available & Best Practice
Liaison between Met Office’s Science & IT teams, and the Scientific community
Develop / Extend Functionality

8. XCS – Top500.org - June 2017 - #11 (Nov 2017 - #15)

9. HPC resource for collaboration
Year
System
Service
Compute Nodes
TFLOPS
2010
IBM Power 6
MONSooN 30 15
2012
IBM Power 7 96 68
2015
Cray XC-40
116 95
2017
Monsoon2
NEXCS
556
278 (/5786)
667
334

10. XCS – Security Model
XCS spans three trustzones, which can be adjusted in HPC capacity.
Research (XCS-R, available)
>70% of XCS compute resource.
Collaboration (XCS-C, available)
Monsoon2 (8.4%, JWCRP: Joint Met Office / NERC Projects)
NEXCS (4.2%, NERC only projects)
Operational (XCS-O, possibly ~2018)
Operational Suite only, ~10% resources
Potentially run more than one parallel suite on XCE, XCF, or XCS-O

11. MONSooN - Met Office / NERC Superco(o)mputer Nodes
Joint Weather and Climate Research Programme (JWCRP) objectives:
Enable closer collaboration between Met Office and NERC scientists
Develop activities to address gaps in national portfolio of weather and climate research
Promote effective pull through of research to improved forecasts
MONSooN: Running since October 2009
“To provide a shared supercomputing service in an environment within which collaborative projects between NERC and Met Office can be performed. The joint service is being provided to enhance collaboration between NERC and Met Office researchers and improve their ability to pull-through environmental science more rapidly.”
Met Office is recognised as being experts in HPC and running our Unified Model. External collaborators have often had difficulty in working closely with Met Office, particularly with Model development; a lot of effort goes into getting a model to work in a new environment; on a different architecture. Much scientific staff time ‘wasted’ on this rather than looking at the science and/or developing better code.
Some collaborators are able to come into the Met Office and work on our systems. That can’t work for everyone.
MONSooN allows NERC and Met Office Collaborators to come together in a common environment; code, HPC and post-processing facilities. Each project has a different requirement of these aspects.

12. Monsoon 2 NERC & Met Office collaboration area on XCS
Available from 14th March 2017
8.4% of XCS, 556 nodes from 5786
Need to be collaborating with Met Office scientists
Around 30 projects, almost 500 registered users
Applications are assessed for Technical & Scientific ‘Fit’

13. Monsoon2 - not just a Supercomputer
Allocation of Supercomputer – Cray XC-40
Post-Processor – RedHat Linux, like Met Office’s Scientific Desktop
Data transfers - Janet and JASMIN (2x1Gbps, moving to 10Gbps)
Read/Write access to MASS data
Also:
Facilities for code development
Collaboration wiki server
Mere HPC size isn’t critical, though it is relevant – other facilities much larger (Archer, Prace); co-location has benefits in itself
Same architecture as current Met Office Supercomputers – this *is* useful
Same architecture as Met Office Supercomputers, but separate zone.
280TB of fast local storage (/home, /projects)
100 TB of local storage - medium term archive for NERC
Post-Processing server (64 bit Red Hat Linux)

14. NEXCS – NERC on XCS NERC-only area on XCS Available since April 2017
4.2% of XCS, 278 nodes from 5786
No need to be collaborating with Met Office scientists
3 resource pools, around 35 registered users
Enrolment and resource allocation – through Grenville Lister, NCAS-CMS

15. NEXCS – mostly just a Supercomputer
No post-processor access
No access to MASS
Data transfer to JASMIN (or similar) using scp/sftp/rsync
Would like GridFTP with certificates to automate transfer to JASMIN
Other methods have been explored… network refresh in progress

16. Monsoon2 and NEXCS – utilisation

17. Authentication/Accounts
Too Many Accounts/Passwords!
PUMA, Monsoon/NEXCS, MOSRS, Collaboration Twiki, etc.
Monsoon/NEXCS – currently two-factor authentication using RSA fobs
Other options are available – Google Authenticate, Yubikeys
Want to follow Met Office Strategy (evolving)

18. Documentation sharing
Current Web server
Introduced 2010
1500+ Registered Users
650+GB Data
Twiki (last release in 2015)
Static Web – allows MO FTP upload
Newsgroup Facility (deprecated)
Met Office moving to Office 365
Yammer – external users can post
SharePoint – docs and calendar (soon?)
Skype For Business (maybe)
Anticipated gaps… (e.g. static web) which will need to be filled for Scientific Collaboration

19. MASS - Overview Main Met Office resilient tape-based archiving system
2 automated tape-libraries - duplexed copies
Only one copy (simplex) of less valuable data (saves tapes)
HPSS library-management system
Bespoke user-interface ‘MOOSE’
Numbers correct as of 19th January 2015

20. MASS – stats - April 2018 ~230TB archived, ~105TB retrieved per day
~1,700,000 archive and 200,000 retrieve commands per week
~60% of retrieves served from 6PB disk cache
~14 day cache longevity
42,000 8 TB tapes, 9-13 TB compressed data per tape
~450 PB tape capacity, 232 PB tape used
Mostly duplexed
Since 2018, only 65% of new data duplexed as cost saving measure
Numbers correct at of 28th March 2017

21. MASS access MASS Read/Write Read Only JASMIN ECMWF HPC Other
Monsoon HPC
(XCS-C)
Met Office CDN
HPC (XCE,F,S-R)
Other
MASS access
Read/Write
Read Only
Clients within the Met Office and MONSooN environments also have read and write access to MASS
JASMIN had read only access to MASS
Use the power of JASMIN when post-processing environments in MONSooN or within the Met Office are not appropriate/powerful enough
Other data can be imported to MASS via manual methods by Met Office staff, not using MOOSE

22. Monsoon, ECMWF, and JASMIN MASS access allows you to…
Have access based on a MASS project basis
List contents of data-sets
Retrieve files
Filter files (‘atomic access’)
Get information about data-sets:
Ownership, quality-assessments, comments etc.

23. Monsoon, ECMWF, and JASMIN MASS access allows you to…
Monsoon and ECMWF MASS access allows you to:
Store and/or overwrite files
JASMIN MASS access does not allow you to:
Store or overwrite files
Only Met Office dataset owners can:
Move, rename or delete files
Change metadata about files or data-sets

24. MASS Access from JASMIN
JASMIN MASS access
MASS Access from JASMIN

25. Access to MASS from JASMIN
You will need...
JASMIN Account - available here
One machine (mass-cli1) has a MOOSE client installed, not accessible by default. Request access from JASMIN Team here

26. Access to MASS from JASMIN
You will also need…
Met Office MASS account for accessing from JASMIN – requires:
A sponsor for each user – who should be a Met Office Senior Scientist
A list of MASS projects or datasets for the user to access
Permission of the dataset owner to make it available externally

27. Isambard - overview Cray CS400 in IT Hall 2
Provides multiple advanced architectures within the same system
Enables evaluation and comparison across a diverse range of hardware platforms
See

30. MOSRS – Met Office Science Repository Service
What is it?
Subversion code repositories
UM 10.0 – 11.1 (due June 2018)
JULES, NEMO, UKESM, LFRic, NAME, and many more
Rosie suite repository & database
Trac environments
inc. documentation & commentary for UM development
documentation for each project
Project list

31. MOSRS - a common shared environment
Makes collaboration easier
Improved pull-through of science changes
Developers work from common code base
Share issue tracking, development
Improves access to new releases
More up-to-date models at collaboration sites

32. MOSRS – current usage 1st June 2017 - 1,500th user registered
28th February 2018 – UM 11.0 – Commit #50,973

34. Further Info / Any questions?
Monsoon -
Isambard -
MOSRS -
MASS from JASMIN -
+44 (0)