1. http://www.ngs.ac.ukhttp://www.grid-support.ac.uk http://www.eu-egee.org/http://www.pparc.ac.uk/http://www.nesc.ac.uk/ NGS in the future: emerging middleware Mike Mineter mjm@nesc.ac.uk

2. 2 Goal of talk The NGS is running a production service Different middleware may be deployed in the future. The talk seeks to outline some of the possibilities

3. 3 NGS middleware evolution ETF NGS Other software sources Software with proven capability & realistic deployment experience ‘Gold’ services Prototypes & specifications Feedback & future requirements EGEE… Deployment/testing/advice Operations Engineering Task Force UK, Campus and other grids

4. 4 Outline Middleware currently being prepared for deployment –Resource broker Under assessment: –gLite middleware from EGEE –OMII –GT4

5. 5 Resource broker (This is NOT the SRB!!!) Current NGS middleware : Toolkits inviting development of higher level services On the current NGS we have –GRAM to submit jobs –Information service to tell us what queues are busy The RB takes the work out of deciding where to run a job

6. Major components ReplicaCatalogue Logging & Book-keeping ResourceBrokerStorageElementComputingElement InformationService Job Status DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Output “sandbox” Publish SE & CE info “User interface”

7. 7 Resource broker Job Description Language file: describes resources needed by a job Commands analogous to GT2: –edg-job-submit –edg-job-status –edg-job-get-output

8. 8 Example edg-job-submit myjob.jdl –Myjob.jdl JobType = “Normal”; Executable = "$(CMS)/exe/sum.exe"; InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"}; OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000; Rank = other.GlueCEStateFreeCPUs;

9. 9 More about the RB Developed by the European DataGrid project, EDG then “hardened” by LCG, and now one of the sources for the EGEE middleware Uses components of Condor –matchmaker and Condor-G Try the GENIUS portal on GILDA –GILDA is a dissemination grid running the LCG-2 middleware –Demo site: https://grid-demo.ct.infn.it/ And look at http://lcg.web.cern.ch/LCG/ http://www.hep.ph.ic.ac.uk/e- science/projects/demo/index.html http://lcg.web.cern.ch/LCG/ http://www.hep.ph.ic.ac.uk/e- science/projects/demo/index.html

10. 10 Resource broker - summary The resource broker receives a job description in JDL It chooses a batch queue for job submission, using the information services Its an example of the higher services that can be deployed for the NGS, built upon the current toolkits

11. 11 Outline Middleware currently being prepared for deployment –Resource broker –(NGS portal – yesterday!) Under assessment: –gLite middleware from EGEE –OMII –GT4

12. Enabling Grids for E-sciencE INFSO-RI-508833 12 EGEE – towards e-infrastructure EGEE is building a large-scale production grid service to: Underpin research, technology and public service Link with and build on national, regional and international initiatives Foster international cooperation both in the creation and the use of the e- infrastructure Network infrastructure & Resource centres Operations, Support and training Collaboration Pan-European Grid

13. Enabling Grids for E-sciencE INFSO-RI-508833 13 Project Goals A four year programme: Build, deploy and operate a consistent, robust and secure grid that attracts new computing resources Improve and maintain the middleware in order to deliver a reliable service to users Attract new users from science and industry and ensure training and support for them

14. Enabling Grids for E-sciencE INFSO-RI-508833 14 In the first 2 years EGEE will Establish production quality sustained Grid services –3000 users from at least 5 disciplines –integrate 50 sites into a common infrastructure –offer 5 Petabytes (10 15 ) storage Demonstrate a viable general process to bring other scientific communities on board Propose a second phase in mid 2005 to take over EGEE in early 2006 Pilot New

15. Enabling Grids for E-sciencE INFSO-RI-508833 15 EGEE Organisation 70 leading institutions in 27 countries, federated in regional Grids ~32 M Euros EU funding for first 2 years starting April 2004 (matching funds from partners) Leveraging national and regional grid activities Promoting scientific partnership outside EU

16. Enabling Grids for E-sciencE INFSO-RI-508833 16 Activities Definition Network Activities –NA1: Project Management –NA2: Dissemination and Outreach –NA3: User Training and Induction –NA4: Application Identification and Support –NA5: Policy and International Cooperation Service Activities –SA1: Grid Support, Operation and Management –SA2: Network Resource Provision Joint Research Activities –JRA1: Middleware Reengineering + Integration –JRA2: Quality Assurance –JRA3: Security –JRA4: Network Services Development Emphasis in EGEE is on operating a production grid and supporting the end-users

17. Enabling Grids for E-sciencE INFSO-RI-508833 17 Co-existence with deployed infrastructure –Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid services Site autonomy –Reduce dependence on ‘global, central’ services Open source license gLite: Guiding Principles Service oriented approach –Allow for multiple interoperable implementations Lightweight (existing) services –Easily and quickly deployable –Use existing services where possible  Condor, EDG, Globus, LCG, … Portable –Being built on Scientific Linux and Windows Security –Sites and Applications Performance/Scalability & Resilience/Fault Tolerance –Comparable to deployed infrastructure EDGVDT... LCG...AliEn

18. Enabling Grids for E-sciencE INFSO-RI-508833 18 gLite Services for Release 1 Grid Access Service API Access Services Job Provenance Job Management Services Computing Element Workload Management Package Manager Metadata Catalog Data Services Storage Element Data Management File & Replica Catalog Authorization Security Services Authentication Auditing Information & Monitoring Information & Monitoring Services Application Monitoring Site Proxy Accounting JRA3 UK CERN IT/CZ Focus on key services

19. 19 The slides that follow were selected and (in a few cases) modified by Mike Mineter (NeSC) from those presented in January 2005 at an OMII training day Steven Newhouse, Peter Henderson Stephen Crouch & Karen Ng Goal of this presentation: to raise awareness of the OMII and its OMII_1 release MM Open Middleware Infrastructure Institute

20. 20 Open Middleware Infrastructure Institute OMII goal: to be the source of open source grid software Institute of the University of Southampton Utilise existing software and standards Production focused software development Integrate, test & document ‘a product’ Focus on the user experience Easy to install & use Utilise existing software and standards Provide a solid web service base for others to build on

21. 21 Where does our software come from? Open Source Community Tomcat, Axis, etc., Software Repository Accept software contributions Software deployed, tested & graded to provide feedback Managed Programme Fill gaps to build a solid enabling infrastructure Projects to bring research software to production quality

22. 22 Managed Programme GridSAM (Job Submission & Monitoring service) BPEL (Workflow service) Grimoires (Registry service based on UDDI) FIRMS (Reliable messaging) FINS (Notification) GeodiseLab (Matlab toolbox) WSRF::Lite integration OGSA-DAI (Database service) WSeSS (Using SSH to tunnel requests to resources)

23. 23 OMII_1 release: A basic File-Compute Grid Enables a generic computational task Move input data from the client to the service provider Process the data using an application on the service provider Retrieve the output data from the service provider

24. 24 OMII Server Infrastructure WS-Security PBAC AXIS Happy Axis TOMCAT Static Webpage Acct Mgmt Servlet Resource Mgmt Servlet Account Allocation Data Job TestService ExampleService

25. 25 Try out the OMII_1 client ! Register at www.omii.ac.uk & loginwww.omii.ac.uk Goto the downloads page Download the client distribution SuSE 9.0 Client may work on other Linuxs but no exhaustive testing Windows XP (SP 1 & 2) Distribution requires JDK 1.4.2_04 Does not work with ‘just’ a JRE Will not work with JDK 1.4.2_05/06 & JDK 1.5.0 No testing with earlier JDKs.

26. 26 Outline Middleware currently being prepared for deployment –Resource broker –(NGS portal – yesterday!) Under assessment (by ETF for NGS): –gLite middleware from EGEE –OMII –GT4

27. 27 A Service-Oriented Infrastructure Service-oriented applications –Wrap applications as services –Compose applications into workflows Service-oriented infrastructure –Provision physical resources to support application workloads Appln Service Users Workflows Composition Invocation Provisioning Carl Kesselman at Globus Week, NeSC, 4th April – 8th April 2005

28. 28 Summary Middleware currently being prepared for deployment –Resource broker –(NGS portal – yesterday!) The context for the next generation of middleware: service orientation, based on WS and the emerging standards Under assessment by the Engineering Task Force: –gLite middleware from EGEE –OMII –GT4