1. Monsoon, NEXCS and access to MASS
NCAS-CMS UM training course
13th December 2017
AJ Watling
Monsoon Collaboration Tech Lead
HPC & Collaboration
Technology & Information Services

2. Contents
Met Office Scientific Collaboration Overview Monsoon 2 and NEXCS External access to MASS Isambard Met Office Science Repository Service (MOSRS) Further info / Questions

3. The Team Scott Irvine – IT Manager: HPC & Collaboration Katie Kerr
Monsoon / MOSRS Service Owner
Katie Kerr
Monsoon & External MASS Access Service Manager
Roger Milton
Scientific Collaboration Technical Lead
AJ Watling
Monsoon Technical Lead

4. What do we do? What’s available & Best Practice
Help Scientists use IT to do Science collaboratively
What’s available & Best Practice
Liaison between Met Office’s Science & IT teams, and the Scientific community
Develop / Extend Functionality

6. XCS – Top500.org - June 2017 - #11 (Nov 2017 - #15)

7. HPC resource for collaboration
Year
System
Service
Compute Nodes
TFLOPS
2010
IBM Power 6
MONSooN 30 15
2012
IBM Power 7 96 68
2015
Cray XC-40
116 95
2017
Monsoon2
NEXCS
556
278 (/5786)
667
334

8. XCS – Security Model
XCS spans three trustzones, which can be adjusted in HPC capacity.
Research (XCS-R, available now)
>70% of XCS compute resource.
Collaboration (XCS-C, available now)
Monsoon2 (8.4%, JWCRP: Joint Met Office / NERC Projects)
NEXCS (4.2%, NERC only projects)
Operational (XCS-O, possibly ~2018)
Operational Suite only, ~10% resources
Potentially run more than one parallel suite on XCE, XCF, or XCS-O

9. MONSooN - Met Office / NERC Superco(o)mputer Nodes
Joint Weather and Climate Research Programme (JWCRP) objectives:
Enable closer collaboration between Met Office and NERC scientists
Develop activities to address gaps in national portfolio of weather and climate research
Promote effective pull through of research to improved forecasts
MONSooN: Running since October 2009
“To provide a shared supercomputing service in an environment within which collaborative projects between NERC and Met Office can be performed. The joint service is being provided to enhance collaboration between NERC and Met Office researchers and improve their ability to pull-through environmental science more rapidly.”
Met Office is recognised as being experts in HPC and running our Unified Model. External collaborators have often had difficulty in working closely with Met Office, particularly with Model development; a lot of effort goes into getting a model to work in a new environment; on a different architecture. Much scientific staff time ‘wasted’ on this rather than looking at the science and/or developing better code.
Some collaborators are able to come into the Met Office and work on our systems. That can’t work for everyone.
MONSooN allows NERC and Met Office Collaborators to come together in a common environment; code, HPC and post-processing facilities. Each project has a different requirement of these aspects.

10. Monsoon 2 NERC & Met Office collaboration area on XCS
Available from 14th March 2017
8.4% of XCS, 556 nodes from 5786
Need to be collaborating with Met Office scientists
Applications are assessed for Technical & Scientific ‘Fit’
Around 30 projects, 200+ registered users

11. Monsoon - not just a Supercomputer
Allocation of Supercomputer – Cray XC-40
Post-Processor – RedHat Linux, like Met Office’s Scientific Desktop
Data transfers - Janet and JASMIN (each 1Gbps)
Read/Write access to MASS data
Facilities for code development
Collaboration wiki server
Mere HPC size isn’t critical, though it is relevant – other facilities much larger (Archer, Prace); co-location has benefits in itself
Same architecture as current Met Office Supercomputers – this *is* useful
Same architecture as Met Office Supercomputers, but separate zone.
280TB of fast local storage (/home, /projects)
100 TB of local storage - medium term archive for NERC
Post-Processing server (64 bit Red Hat Linux)

12. NEXCS – NERC on XCS New NERC-only area on XCS
4.2% of XCS, 278 nodes from 5786
First available April 2017
No need to be collaborating with Met Office scientists
Enrolment and resource allocation – through Grenville Lister, NCAS-CMS

13. NEXCS – mostly just a Supercomputer
No postprocessor
No access to MASS
Data transfer to JASMIN (or similar) using scp/sftp/rsync
Would like GridFTP with certificates to automate transfer to JASMIN
Other methods have been explored…

14. Monsoon2 and NEXCS – utilisation

15. Authentication/Accounts
Monsoon Evolution
Authentication/Accounts
Too Many Accounts/Passwords!
Collaboration Twiki, PUMA, MOSRS, Monsoon, etc…
For Monsoon, currently two-factor authentication via RSA fobs
Other options are available – Google Authenticate, Yubikeys
Want to follow Met Office Strategy (developing area)

16. Monsoon Evolution Collaboration Tools Current Web server
Introduced 2010
1500+ Registered Users
650+GB Data
Twiki
Static Web – allows MO FTP upload
Newsgroup Facility
Met Office moving to cloud-based Office 365, including
Yammer
Skype For Business
Step change in functionality
There will be gaps… (e.g. static web) which will need to be filled for Scientific Collaboration

17. MASS - Overview Main Met Office resilient tape-based archiving system
2 automated tape-libraries - duplexed copies
HPSS library-management system
Bespoke user-interface ‘MOOSE’
Numbers correct as of 19th January 2015

18. MASS - current stats ~180TB archived, ~36TB retrieved per day
~1,000,000 archive and 100,000 retrieve commands per week
~90% of retrieves served from 6PB disk cache
~22-23 day cache longevity
26,000 8 TB tapes, TB compressed data per tape
230 PB tape capacity, 146 PB tape used (mostly duplexed)
Numbers correct at of 28th March 2017

19. MASS access MASS JASMIN ECMWF HPC Other Met Office CDN Monsoon HPC
(XCS-C)
Met Office CDN
HPC (XCE,F,S-R)
Other
MASS access
Clients within the Met Office and MONSooN environments also have read and write access to MASS
JASMIN had read only access to MASS
Use the power of JASMIN when post-processing environments in MONSooN or within the Met Office are not appropriate/powerful enough
Other data can be imported to MASS via manual methods by Met Office staff, not using MOOSE

20. Monsoon, ECMWF, and JASMIN MASS access allows you to…
Have access based on a MASS project basis
List contents of data-sets
Retrieve files
Filter files (‘atomic access’)
Get information about data-sets:
ownership
quality-assessments, comments etc.

21. Monsoon, ECMWF, and JASMIN MASS access allows you to…
Monsoon and ECMWF MASS access allows you to:
Store and/or overwrite files
JASMIN MASS access does not allow you to:
Store or overwrite files
Only Met Office dataset owners can:
Move, rename or delete files
Change metadata about files or data-sets

22. MASS Access from JASMIN
JASMIN MASS access
MASS Access from JASMIN

23. Access to MASS from JASMIN
You will need...
JASMIN Account - available here
One machine (mass-cli1) has a MOOSE client installed, not accessible by default. Request access from JASMIN Team here

24. Access to MASS from JASMIN
Also…
Met Office MASS account for accessing from JASMIN – requires:
A sponsor for each user – who should be a Met Office Senior Scientist
A list of MASS projects or datasets for the user to access
Permission of the dataset owner to make it available externally

25. Isambard - overview Cray CS400 in IT Hall 2
Provides multiple advanced architectures within the same system
Enables evaluation and comparison across a diverse range of hardware platforms
See

28. MOSRS – Met Office Science Repository Service
What is it?
Subversion code repositories
UM 10.0 – 11.0
JULES, NEMO, UKESM, LFRic, NAME, and many more
Rosie suite repository & database
Trac environments
inc. documentation & commentary for UM development
documentation for each project
Project list

29. MOSRS - a common shared environment
Makes collaboration easier
Improved pull-through of science changes
Developers work from common code base
Share issue tracking, development
Improves access to new releases
More up-to-date models at collaboration sites

30. MOSRS – current usage 1st June 2017 - 1,500th user registered
31st October UM Commit #45,850

32. Further Info / Any questions?
Monsoon -
Isambard -
MOSRS -
MASS from JASMIN -
+44 (0)