Tony Doyle GridPP Collaboration Meeting 14 Feb 2002

Tony Doyle - University of Glasgow GridPP Status Report Starting Point Project Management Structures Experiments Board (EB) CERN and the LHC Computing Grid (LCG) Technical Board (TB) Project Management Board High Level Deliverables Staff Effort and Recruitment DataGrid “Inter Grids” 2002 Challenges UK Testbed = “The Grid” Summary Agenda Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow GridPP Management Who’s who?.. c.f. Steve’s web list of contacts Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Collaboration Board Chair Steve Lloyd Birmingham Peter Watkins Liverpool Themis Bowcock Bristol Greg Heath Manchester Robin Marshall Brunel Peter Hobson Oxford Susan Cooper CERN Les Robertson Queen Mary Tony Carter Cambridge Janet Carter Royal Holloway John Strong Durham Mike Whalley RAL Ken Peach Edinburgh Steve Playfer Sheffield Neil Spooner Glasgow David Saxon Sussex David Wark IC Peter Dornan Swansea Chris Alton Lancaster Peter Ratoff UCL Peter Clarke Project Leader (in attendance) Tony Doyle e-Science Director (in attendance) Neil Geddes Members are 'Group Leaders' of all GridPP Institutes Chair elected by Board Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Experiments Board ATLAS Roger Jones, Andy Parker CMS Dave Britton, Dave Newbold LHCb Nick Brook (Chair), Glenn Patrick ALICE Ian Bloodworth BaBar Roger Barlow (Deputy Chair) D0 Iain Bertram CDF Todd Huffman Others Dan Tovey Theory Alan Irving CERN Jamie Shiers TB Cross Members: Pete Clarke, tbd Members chosen by Experiments Chair and Deputy elected by Board Tony Doyle - University of Glasgow

Recruitment Timetable 8th October - EB input deadline Inputs: 1. TECHNICAL CASE Complete 2. ASSESMENT OF HOW MUCH EFFORT IS REQUIRED Complete 3. WHO WILL CONTRIBUTE TO/LEAD THESE DEVELOPMENTS Complete 4. WHAT IS REMAINING TO BE DONE - I.E. A POSTS REQUIREMENT Complete 5. A JOB SPECIFICATION OF THE OUTSTANDING POST(S) INCLUDING DELIVERABLES AND MILESTONES Complete 19th October - TAG assessment document Next step - technical deployment issues 26th October - PMB definition of tasks Next step - deployment/resource issues first review of CERN posts 2nd November - Tender document 5-6th November - Collaboration Meeting 30th November- Deadline for submission by Institutes 7th December - PRSC mtg. 14th December- Oversight Committee mtg. Ambitious timescales. Experiments Input? “good” to “very good” Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Experiments Board Nick Brook New phase for EB – moving from focussing on post “creation” to the need for exploitation and utilisation of GridPP resources Remit: To ensure that the GridPP project is ‘experimentally-driven’ ‘Long term’ planning – assessment of needs and reqts from GridPP Dissemination of experiences and information between UK collaborations To facilitate close working relationship between expts. Overall liaison between the expts & GridPP Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow EB Meetings Nick Brook Next meeting Late March, after LCG startup workshop Areas of discussion: status of recruitment status & experience of Grid work long term plans (changes to original submissions) reporting procedures to GridPP testbed experiences management & "booking" of  UK Grid resources quarterly reporting to TB Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow CERN Status Tony Cass The LCG Project is up and running PEB, SC2 and POB have all met. Launch Workshop, 11th-15th March Hardware for 2002 Prototype is available Scheduling ongoing. RTAGs launched for Persistency, Managing Software, Maths Libraries and Grid Requirements. Initial report from Persistency and Managing Software RTAGs at launch workshop. RTAG reports will determine assignments for GridPP staff in relevant areas. Fabric Management Activity Plan being prepared Similarly, this will determine assignments for GridPP staff working on Fabric Management. Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow CERN concerns Tony Cass Ensuring compatibility between EU and US testbeds Understanding impact if first physics moves to 2007 On Production system On LCG Phase I Scheduling of the Prototype Understand needs and types of challenge - pushing the envelope or mass production for LHCC milestones Fitting these in resources available Tight schedule for new tape drives for ALICE And we need to support running experiments in a reduced budget Scheduling to allow test of Grid tools across multiple sites. Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Technical Board Pete Clarke Composition to date (reflects dynamic nature): Chair Pete Clarke Deputy Robin Middleton WP1 Dave Colling WP2 Gavin McCance WP3 Steve Fisher WP4 Lex Holt/Paul Anderson WP5 John Gordon WP6 Andrew McNab WP7 Richard Hughes-Jones WP8 Frank Harris Middleware Alex Martin Experiments Steve Burke Security Dave Kelsey EB Nick Brook Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow TB Meetings Pete Clarke December adoption of DataGRID reporting mechanisms for WP1-7 (3-monthly reports) Jan 9th EDG: Post quarterly submission review of delivered resources EDG: Review of UK deliverable responsibilities Gantt chart : testbed functionality Testbed rollout planning: first discussion Application builders workshop planning adoption of TB-SUPPORT@JISCMAIL.AC.UK Feb 8th Detailed discussion of Testbed rollout planning => SEE TOMORROW First discussion of non-LHC exp support GridPP DEVELOPMENT Grid Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Resource Monitoring Dave Britton Next three-monthly reports in March (~18th).. Input = 3-monthly report A work-segment is a structure something like: {Quarter, Person, Institute, Experiment, Task, StaffMonths} for example: {2002Q1, P.Mealor, UCL, NONE, T7.2, 3.00} Extract information such as: 1) Total delivered effort as a function of time. 2) Effort as a function of task or workgroup. 3) Effort as a function of Institute or Experiment (as a function of time). 4) Histograms of any of these quanitites (e.g. effort vs workgroup or Institute or experiment). Technically - Input into Excel (visual basic macros) Need to integrate into project management tools Need to match to high level goals Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow TB Documentation Pete Clarke EDG resource management document produced and circulated WP1-7 mechanisms now in place and being used usefully Applications posts monitoring mechanisms now being reviewed Testbed rollout planning and support document in preparation iterated within TB next step is circulation to EB and WPs affected (WP6) for comment will then be circulated to all for comment to be outlined by PC tomorrow [Note: TB-WWW pages being fixed now] Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow TB Meetings Pete Clarke Application Builders meeting 31st Jan @ RAL aim was “how to use middleware” talks where appropriate (e.g. WP1) in addition overview of what other middleware does All future meetings will be as requested by GridPP (i.e. the audience) Will probably only cover one theme Next planned is half day (or more) on use of Job submission system We will attempt to make the format more “training oriented” (perhaps use RAL or IC training rooms) Will circulate request for interest (6 weeks from now) Use other meetings, where appropriate e.g. RAL (SRB) and NeSC e-Science Institute (Globus).. But try to minimise.. Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow

Project Management Board Project Leader Tony Doyle Appointed by CB Deputy Project Leader John Gordon Appointed by Project Leader Chair of Collaboration Board Steve Lloyd Ex officio Chair of Technical Board Peter Clarke Appointed by Project Leader Chair of Experiments Board Nick Brook Ex officio Deputy Chair of EB Roger Barlow Ex officio Resource Manager Dave Britton Appointed by Project Leader DataGrid UK Project Leader Robin Middleton Ex officio CERN Liaison Tony Cass Appointed by CERN e-Science Director Neil Geddes Ex officio As proposal plus Resource Manager + DataGrid UK Project Leader + Part time Secretary at Glasgow Three year posts (except EB 1+2) Meets weekly via VC and monthly face to face Tony Doyle - University of Glasgow

High Level Deliverables To be re-defined to extend to “The Grid” i.e. testbed Aspiration: all that you expect from a central service? Probably not.. Tomorrow’s Discussion.. Tony Doyle - University of Glasgow

High Level Deliverables Requires input from WP contacts Tony Doyle - University of Glasgow

High Level Deliverables Pre-defined at high level, but requires further iteration Tony Doyle - University of Glasgow

Projected Staff Effort [SY] Area GridPP @CERN CS WP1 Workload Management 0.5 [IC] 2.0 [IC] WP2 Data Management 1.5 [Ggo] 1.0 [Oxf] WP12 WP3 Monitoring Services 5.0++ [RAL, QMW] 1.0 [HW] Security ++ [RAL] 1.0 [Oxf] WP4 Fabric Management 1.5 [Edin., L’pool] WP5 Mass Storage 3.5++ [RAL, L’pool] WP6 Integration Testbed 5.0++ [RAL/M’cr/IC/Bristol] WP7 Network Services 2.0 [UCL/M’cr] 1.0 [UCL] WP8 Applications 17.0 ATLAS/LHCb (Gaudi/Athena) 6.5 [Oxf, Cam, RHUL, B’ham, RAL] CMS 3.0 [IC, Bristol, Brunel] CDF/D0 (SAM) 4.0 [IC, Ggo, Oxf, Lanc] BaBar 2.5 [IC, M’cr, Bristol] UKQCD 1.0 [Edin.] Total 36.0++ 10.0 ->20.0? 6.0 Resource Management Tony Doyle - University of Glasgow

DataGrid Recruitment Status Robin Middleton Dec 01 Tools being developed Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow DataGrid-UK Status Robin Middleton Recruitment EU funded posts filled & in post (1 additional from 1/2/02) All but 2 of 15 “unfunded” posts active Reporting UK templates defined, with TB meetings before & after quarterly submissions 81% of effort contracted delivered (but not uniform across WPs) for WP1-7 (WP8 hard to quantify) UK Progress (in 2001) Architectures, Testbed-1, network monitoring, certificates & security, R-GMA, LCFG, MDS deployment, Spitfire… … Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow DataGrid-UK Effort Robin Middleton Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow DataGrid-UK Status Robin Middleton EU Review All UK deliverables passed EDG internal review and now submitted to EU Review at CERN on 1st March 3 external reviewers ~40 deliverables (inc. quarterly reports) Summary presentations only Can expect detailed feedback of areas to address 4th Project Conference – Paris 4th-8th March Plans for 2002 – TB2 Absorb initial feedback from EU review Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow DataGrid-UK News Robin Middleton Licensing is still an issue ATF reconstituted with more Applications involvement New project web site at original address Based at CERN Uses EDMS for document handling Testbed (see elsewhere at meeting) Minor releases every 2 months V2 in September Transatlantic testbeds coming closer to reality Scope being clarified Interoperability a major issue  architectures Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow InterGrid etc Robin Middleton Strictly HENP-InterGrid HICB meeting on 17th Feb Sunday (prior to GGF) HIJTB meet monthly by phone Projects BaBar testbed GRIT – Grid Reference International Testbed ATLAS & CMS DataTAG started DataTAG : WP2=QoS, etc; WP4=m/w interoperability Collaborates with iVDGL CrossGrid, GridStart Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow GRIT Ruth Pordes Tony Doyle - University of Glasgow

“The Grid” is an evolving concept enabler for transient ‘virtual organisations’ anatomy: a software infrastructure that enables flexible, secure, co-ordinated resource sharing among dynamic collections of individuals, institutions and resources Foster, Kesselman & Tuecke (2001) evolution of and integration with web services physiology: everything is a Grid service i.e. a service that conforms to a set of conventions for management and exchanging messages Foster, Kesselman, Nick & Tuecke (2002) http://www.globus.org/research/papers/ogsa.pdf Global Grid Forum: define a standard Grid architecture big business and big science starting to work together GGF4 (Toronto) next week Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow GGF Architecture 1) scope the problem - what are we trying to produce 2) decide on the require functionality 3) modularize the functionality 4) decide on the relationships among the modules 5) identify the data that must flow among the modules 6) decide what state has to be maintained, and where 7) design the protocols for communicating among and with the modules 8) APIs The challenge in the Grid forum is - picked a large and amorphous problem (wide area distributed computing) - for which there is not agreement (yet) as to what all of the functionality should be - and even where we think that we understand the functionality requirements (e.g. data and resource management) there is not yet consensus on the modules ANSWERS?? DataGrid++ is “The Architecture” we have adopted - look here first Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow e-Science and Grid We are asked to map onto this plane… Recognisable words, focussing on “middleware” DataGrid++=middleware (where ++=SAM, SLAC-based developments) Experiments=Application Area Middleware=Application Area?? Draft 2.0 status Response should be built up from technical definitions Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow 2002 Challenges Complete rollout of TB-1 and plan future upgrades Reconvened ATF to work closely with applications Make TB-2 a success Deploy and exploit Tier-1/A Applications to make good use of testbeds Solve interoperability issues We are part of many larger collaborations/structures/groupings - we need to collaborate/discuss engage here, and Focus on implementation in the UK… this will tell us what works (and what doesn’t) at any given point. Tony Doyle - University of Glasgow

Status 13/2/02: Requires upgrades to functionality (extract hosts from info. services automatically) and maintainability (integrate into testbed workplan) Technical Management: See tomorrow’s discussion Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Status 13/2/02: Expect that Experiments will deliver production monitoring systems e.g. AliEn for ALICE (integrate into testbed workplan) Tony Doyle - University of Glasgow

Guidelines and Policies Disk/CPU resources allocated to each “group” Grid is based on distributed resources - a “group” is an experiment An institute is typically involved in many experiments Institutes define priorities on computing resources via OPEN policy statements e.g. Edinburgh/Glasgow RAL ++ ScotGrid - Priority 1: ATLAS, LHCb, Computing Science Data Mining BaBar (2/3-1/2-1/3) LHC (1/3-1/2-2/3) tomorrow’s - Priority 2: BaBar, CDF, UKQCD e-Science discussions BioInformatics All jobs submitted via Globus authentication - Certificates identified by user and experiment Need to implement “priority” Minimum amount of data to deliver at a time for a job? Where to store files? Which data access/storing activities have the highest priority? Sharing of the resources among groups? Users belong to multiple groups? How many jobs per group are allowed? What processing activities are allowed at each site? To which sites should data access and processing activities be sent? How should the resources of a local cluster of PCs be shared among groups? REQUIRES Discussion at the Collaboration Meeting… issues will arise which require ALL Institutes to publish a policy Longer term goals… Grid Credits issued? Its all in the small print. * Implemented by site administrators, with OPEN policies defined at each site based on e.g. case to funding authority What’s new? Ability to monitor/allocate unused resources We will be judged by how well we work as a Virtual Organisation Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Summary Most effort so far directed towards DataGrid=middleware Significant UK contributions Startup difficulties – largely overcome Recruitment, late filling of posts – funded & unfunded Lost ground being recovered.. All WP with UK involvement active & contributing Testbed-1 : so far modest, predictable teething problems (many !) Next major challenges focus on INTEGRATION US experiments programme - planning for today/tomorrow.. LCG programme - planning for 2007... Testbed (focus for tomorrow’s discussions) - requires input from system managers as well as software developers. Positioned reasonably well ~6 months into GridPP.. with many challenges ahead... Tony Doyle - University of Glasgow

Tony Doyle - University of Glasgow Closing thought… 14/2/02 With many hidden words, meanings, nuances… “communication is the answer to the problem” (apparently) Grids are everywhere.. Tony Doyle - University of Glasgow

3rd Collab Meeting Agenda Feb 14th Opening Session (Chair - Andy Parker): 11:00-11:30 Welcome and Background - Steve Lloyd 11:30-12:00 GridPP Project Status - Tony Doyle 12:00-12:30 LHC Computing Grid Project Status - Tony Cass Middleware Status Reports and Development (Chair - Tony Doyle): 13:30-13:50 Information Services (FTREE and MDS) - Alex Martin 13:50-14:10 Information Services (RGMA) - Manfred Oevers 14:10-14:30 Mass Storage - Jens Jensen 14:30-14:50 Fabric management - Michael George 14:50-15:10 Grid Data Management - Will Bell 15:10-15:30 Networking - Paul Mealor Experiment Developments (Chair - John Gordon): 16:00-17:00 ATLAS, CMS, LHCb and theory - Nick Brook 17:00-17:30 WP8 status - Frank Harris 17:30-18:25 BaBar, CDF, D0, SAM and others - Roger Barlow 18:25-18:30 Logistics - Andy Parker 18:30-19:30 Project Management Board Meeting Feb 15th Cambridge Perspective and TestBed Introduction (Chair - Robin Middleton): 9:00-9:30 Cambridge e-Science Centre I - Andy Parker 9:30-10:00 Cambridge e-Science Centre II - Mark Hayes 10:00-10:30 TestBed Technical Development - Pete Clarke Grid Deployment I (Chair - Pete Clarke): 11:00-11:30 Testbed Tools and Release in UK - Andrew McNab 11:30-11:40 Security issues - Linda Cornwall 11:40-11:55 Testbed status at RAL - Steve Traylen 11:55-12:10 Testbed status at M'cr - Andrew McNab 12:10-12:25 Testbed status at Bristol - Marc Kelly 12:25-12:40 Testbed status at IC - Dave Colling Grid Deployment II (Chair - Steve Lloyd): 13:30-13:40 Testbed status at Edinburgh/Glasgow - David Martin 13:40-13:50 Testbed status at Liveropool - Girish Patel 13:50-14:00 Testbed status at Oxford - Pete Gronbech 14:00-14:10 Testbed status at UCL - Paul Mealor 14:10-14:20 Testbed status at Cambridge - Mark Hayes 14:20-14:30 Testbed status at Brunel - Peter van Santen 14:30-14:40 Testbed status at Lancaster - Roger Jones 14:40-15:00 Discussion 15:00-15:30 TestBed UK Organization - John Gordon WELCOME! Tony Doyle - University of Glasgow