1. http://www.ngs.ac.ukhttp://www.nesc.ac.uk/training Grid Computing and The National Grid Service

2. 2 Overview More about Grids - building on Monday’s ideas The National Grid Service –Introduction –Practical introduction to (some of) its services

3. 3 This morning Toolkits – Taverna … Community-specific orchestration of grid services Basic services: AA, job submission, info, … Middleware: “higher level services” Application toolkits Application Core middleware of Grids Computation authentication, authorisation tomorrow - Data

4. Enabling Grids for E-sciencE INFSO-RI-508833 4 INTERNET User communities negotiate with sites to agree access to resources Grid middleware runs on each shared resource to provide –Data services –Computation services –Single sign-on Distributed services (both people and middleware) enable the grid Typical current grid At each site that provides computation: Local resource management system (= batch queue) Condor PBS Torque …

5. 5 Basic Grid Services Data services – diverse types –Controlled sharing, computation close to data (big datasets) Compute services –Opening resources for users to run what they want! –Users commit to acceptable use policy –MUST have authorisation and authentication Control access to resources: compute and data –But in way that scales! Know who did what Authentication and Authorisation –grid resources share mechanisms for these

6. 6 More on the ecosystem theme…. Grids emerging from many developments Production grids (like EGEE and NGS) adopting the most reliable technologies In compute services especially see alternative ways to run jobs –Common characteristics: Program executes in batch mode –Submit / query status / retrieve output when done “File staging”: transfer of input and output files to/from where job runs Unless have scheduled resources, can’t guarantee times to job completion –Current emphases Long jobs Running many similar jobs (parameter sweeps, pattern searching,) –User communities benefit from scale and diversity of resources… and sharing of resources with collaborators

7. http://www.ngs.ac.ukhttp://www.nesc.ac.uk/training The National Grid Service Mike Mineter and Katie Weeks

8. 8 Policy for re-use This presentation can be re-used for academic purposes. However if you do so then please let training- support@nesc.ac.uk know. We need to gather statistics of re-use: no. of events, number of people trained. Thank you!!training- support@nesc.ac.uk

9. 9 Overview Introduction to the NGS –What we are, history, hardware, how to access it Services on the NGS –Compute, data –Community support Bioinformatics examples

10. 10 What is the NGS? The NGS is the UK’s grid for academics Providing compute and data resources on the grid Providing help and support to those using the grid (any grid) The NGS is led and coordinated by the STFC in collaboration with the University of Manchester, the University of Oxford, the University of Edinburgh and the White Rose Grid at the University of Leeds

11. 11 A Brief History 2001 - UK e-Science Grid –GridPP and others start 2003 - Initial grid service ITT –4 independent clusters to investigate provision of a grid service April 2004 - NGS pre-production service –EGEE, GridPP-2 August 2004 – GOSC (Grid Operations Support Centre) proposed –Coordinating NGS and providing central services September 2004 - NGS production service / GOSC April 2006 – NGS/GOSC phase 1 review May 2006 - NGS phase-2 approved –More integrated programme –EGEE-2 started in April October 2006 – NGS phase-2

12. 12 The Mission of the NGS The Mission of the National Grid Service To provide coherent electronic access for UK researchers to all computational and data based resources and facilities required to carry out their research, independent of resource or researcher location

13. http://www.ngs.ac.ukhttp://www.nesc.ac.uk/training

14. 14 U of A HPCxHPCx UofD NGS Core Nodes: Host core services, coordinate integration, deployment and support +free to access resources for all VOs. Monitored interfaces + services NGS Partner Sites: Integrated with NGS, some services/resources available for all VOs Monitored interfaces + services NGS Affiliated Sites: Integrated with NGS, support for some VO’s Monitored interfaces (+security etc.) General principle here: establish core and grow it: compute, data and operational services RAL Oxford Leeds Man. HECTORHECTOR U of B U of C Commercial Provider ? PSRE NGS Vision

15. 15 Core Sites

16. 16 Partner Sites

17. 17 Affiliates Edinburgh became partner in January 2008

18. 18 All

19. 19 Membership options Two levels of membership (for sharing resources): 1.Affiliates –run compatible stack, integrate support arrangements –adopt NGS security policies –all access to affiliate’s resources is up to the affiliate except allowing NGS to insert probes for monitoring purposes 2.Partners also –make “significant resources” available to NGS users –enforce NGS acceptable use policies –provide accounting information –define commitments through formal Service Level Descriptions –influence NGS direction through representation on NGS Technical Board

20. 20 NGS software Computation services based on Globus Toolkit –Use compute nodes for sequential or parallel jobs, from batch queues –Alternative ways to run jobs exist… Data & storage services: –Storage Resource Broker: Primarily for file storage and access Virtual filesystem with replicated files –NGS Oracle service –“OGSA-DAI”: Data Access and Integration Primarily for grid-enabling data not on the SRB or Oracle files, relational, XML Authorisation, Authentication –Built on “X.509 certificates”

21. 21 NGS Helpdesk All NGS and grid related queries come through here: –Certificates –Requests for applications to be installed –Problems accessing NGS –Technical queries etc. Users can submit queries –By emailing support@grid-support.ac.uksupport@grid-support.ac.uk –Via the form on the website www.ngs.ac.ukwww.ngs.ac.uk –By calling 01235 446822

22. 22 Website First port of call for information Online documentation covering all areas Newly organised Self-guided learning online Familiarise yourself with documentation Contact the helpdesk if you find anything missing/incomplete NGS web site: http://www.ngs.ac.ukhttp://www.ngs.ac.uk Wiki service: http://wiki.ngs.ac.ukhttp://wiki.ngs.ac.uk

23. 23 Training Contact the training team at NeSC –Training- support@nesc.ac.ukTraining- support@nesc.ac.uk Or look online –http://www.nesc.ac.uk /training/events/http://www.nesc.ac.uk /training/events/ Note the Digital Library

24. 24 User Forums

25. 25 Bioinformatics examples…..

26. 26 http://www.ngs.ac.uk/sites/ral/applications/bioinformatics/

27. 27 What do you need? All requests for software to be installed on the NGS are welcomed. We make every effort to install software that is requested. Licensing can be a difficulty though – open source software requests are more likely to be accepted.

28. 28 Simulation of a drug permeating through a membrane Name: Dr Brian Cheney Institution: University of Southampton Research: Membrane Permeation Drs Brian Cheney and Jonathan Essex research membrane permeation of small molecules at the University of Southampton. They’re interested in learning what physical and chemical features make a molecule a good or bad permeant, and in developing ways to quantify and estimate a molecule’s permeability.

29. 29 Integrative Biology Project. “we can get 20ms of animation in 20 minutes using 32 CPUs on the NGS. And the benefits of services such as the Storage Resource Broker are immense - it’s fantastic to be able to share data with colleagues all over the world so easily.” Name: Thushka Maharaj Institution: University of Oxford Research: The effects of defibrillation on the heart

30. 30 Simulating DNA Mechanics Overtwisted DNA circles Simulating DNA stretching Work on the NGS done by Sarah Harris and Jon Mitchell, University of Leeds

31. 31 Prediction of dose in radiotherapy patients Neutron Scattering Systems Biology Climate modelling

32. 32 Econometric analysis substrate complex product complex ADP ATP Molecular Dynamics

33. 33 Oracle Working with researchers at Southampton University Developing a database to store chemical properties data Complex interactions between properties Data stored as Resource Description Frameworks within Oracle Massive, complicated queries can be run on data

34. 34 BBSRC and SRB Dedicated SRB resource to BBSRC Over 500 users Over 10 sites archiving data Used to archive, query and retrieve research material Java GUI developed by STFC

35. 35 Growth within BBSRC

36. 36 Working with…. John Innes Institute (Norwich) Institute for Grasslands and Environmental Research (Aberystwyth) NERC Environmental Bioinformatics Centre (Oxford)

37. 37 What can the NGS do for me? Provide computational power Provide data storage Support your collaborations Provide support and advice in all areas of grid computing (Try to) respond to what you need…. If you tell us what that is!