GridPP Collaboration Meeting 14 Feb 2002 Tony Doyle

Tony Doyle - University of Glasgow GridPP Status Report  Starting Point  Project Management Structures  Experiments Board (EB)  CERN and the LHC Computing Grid (LCG)  Technical Board (TB)  Project Management Board  High Level Deliverables  Staff Effort and Recruitment  DataGrid  “Inter Grids”  2002 Challenges  UK Testbed = “The Grid”  Summary  Agenda

Tony Doyle - University of Glasgow GridPP Management Who’s who?.. c.f. Steve’s web list of contacts

Tony Doyle - University of Glasgow Collaboration Board ChairSteve Lloyd BirminghamPeter WatkinsLiverpoolThemis Bowcock BristolGreg Heath ManchesterRobin Marshall BrunelPeter HobsonOxfordSusan Cooper CERNLes RobertsonQueen MaryTony Carter CambridgeJanet Carter Royal HollowayJohn Strong DurhamMike Whalley RALKen Peach EdinburghSteve PlayferSheffieldNeil Spooner GlasgowDavid SaxonSussexDavid Wark ICPeter DornanSwanseaChris Alton LancasterPeter RatoffUCLPeter Clarke Project Leader (in attendance)Tony Doyle e-Science Director (in attendance)Neil Geddes  Members are 'Group Leaders' of all GridPP Institutes  Chair elected by Board

Tony Doyle - University of Glasgow Experiments Board  ATLASRoger Jones, Andy Parker  CMSDave Britton, Dave Newbold  LHCbNick Brook (Chair), Glenn Patrick  ALICEIan Bloodworth  BaBarRoger Barlow (Deputy Chair)  D0Iain Bertram  CDFTodd Huffman  OthersDan Tovey  TheoryAlan Irving  CERNJamie Shiers  TB Cross Members:Pete Clarke, tbd  Members chosen by Experiments  Chair and Deputy elected by Board

Tony Doyle - University of Glasgow Recruitment Timetable  8th October - EB input deadline  Inputs: ·1. TECHNICAL CASE Complete ·2. ASSESMENT OF HOW MUCH EFFORT IS REQUIRED Complete ·3. WHO WILL CONTRIBUTE TO/LEAD THESE DEVELOPMENTS Complete ·4. WHAT IS REMAINING TO BE DONE - I.E. A POSTS REQUIREMENT Complete ·5. A JOB SPECIFICATION OF THE OUTSTANDING POST(S) INCLUDING DELIVERABLES AND MILESTONES Complete  19th October - TAG assessment document  Next step - technical deployment issues  26th October - PMB definition of tasks  Next step - deployment/resource issues  first review of CERN posts  2nd November - Tender document  5-6th November - Collaboration Meeting  30th November- Deadline for submission by Institutes  7th December - PRSC mtg.  14th December- Oversight Committee mtg. Ambitious timescales. Experiments Input? “good” to “very good”

Tony Doyle - University of Glasgow Experiments Board  New phase for EB – moving from focussing on post “creation” to the need for exploitation and utilisation of GridPP resources  Remit:  To ensure that the GridPP project is ‘experimentally-driven’  ‘Long term’ planning – assessment of needs and req ts from GridPP  Dissemination of experiences and information between UK collaborations  To facilitate close working relationship between expts.  Overall liaison between the expts & GridPP Nick Brook

Tony Doyle - University of Glasgow EB Meetings  Next meeting  Late March, after LCG startup workshop  Areas of discussion:  status of recruitment  status & experience of Grid work  long term plans (changes to original submissions)  reporting procedures to GridPP  testbed experiences  management & "booking" of UK Grid resources  quarterly reporting to TB Nick Brook

Tony Doyle - University of Glasgow CERN Status  The LCG Project is up and running  PEB, SC2 and POB have all met.  Launch Workshop, 11 th -15 th March  Hardware for 2002 Prototype is available  Scheduling ongoing.  RTAGs launched for Persistency, Managing Software, Maths Libraries and Grid Requirements.  Initial report from Persistency and Managing Software RTAGs at launch workshop.  RTAG reports will determine assignments for GridPP staff in relevant areas.  Fabric Management Activity Plan being prepared  Similarly, this will determine assignments for GridPP staff working on Fabric Management. Tony Cass

Tony Doyle - University of Glasgow CERN concerns  Ensuring compatibility between EU and US testbeds  Understanding impact if first physics moves to 2007  On Production system  On LCG Phase I  Scheduling of the Prototype  Understand needs and types of challenge - pushing the envelope or mass production for LHCC milestones  Fitting these in resources available  Tight schedule for new tape drives for ALICE  And we need to support running experiments in a reduced budget  Scheduling to allow test of Grid tools across multiple sites. Tony Cass

Tony Doyle - University of Glasgow Technical Board  Composition to date (reflects dynamic nature):  ChairPete Clarke  DeputyRobin Middleton  WP1Dave Colling  WP2Gavin McCance  WP3Steve Fisher  WP4Lex Holt/Paul Anderson  WP5John Gordon  WP6Andrew McNab  WP7Richard Hughes-Jones  WP8Frank Harris  MiddlewareAlex Martin  ExperimentsSteve Burke  SecurityDave Kelsey  EBNick Brook Pete Clarke

Tony Doyle - University of Glasgow  December  adoption of DataGRID reporting mechanisms for WP1-7 (3-monthly reports)  Jan 9 th  EDG: Post quarterly submission review of delivered resources  EDG: Review of UK deliverable responsibilities  Gantt chart : testbed functionality  Testbed rollout planning: first discussion  Application builders workshop planning  adoption of  Feb 8 th  Detailed discussion of Testbed rollout planning => SEE TOMORROW  First discussion of non-LHC exp support  GridPP DEVELOPMENT Grid TB Meetings Pete Clarke

Tony Doyle - University of Glasgow Resource Monitoring  Input = 3-monthly report  A work-segment is a structure something like:  {Quarter, Person, Institute, Experiment, Task, StaffMonths}  for example:  {2002Q1, P.Mealor, UCL, NONE, T7.2, 3.00}  Extract information such as:  1) Total delivered effort as a function of time.  2) Effort as a function of task or workgroup.  3) Effort as a function of Institute or Experiment (as a function of time).  4) Histograms of any of these quanitites (e.g. effort vs workgroup or Institute or experiment).  Technically - Input into Excel (visual basic macros)  Need to integrate into project management tools  Need to match to high level goals Next three-monthly reports in March (~18th).. Dave Britton

Tony Doyle - University of Glasgow  EDG resource management document produced and circulated  WP1-7 mechanisms now in place and being used usefully  Applications posts monitoring mechanisms now being reviewed  Testbed rollout planning and support document in preparation  iterated within TB  next step is circulation to EB and WPs affected (WP6) for comment  will then be circulated to all for comment  to be outlined by PC tomorrow [Note: TB-WWW pages being fixed now] TB Documentation Pete Clarke

Tony Doyle - University of Glasgow  Application Builders meeting 31 st RAL  aim was “how to use middleware” talks where appropriate (e.g. WP1)  in addition overview of what other middleware does  All future meetings will be as requested by GridPP (i.e. the audience)  Will probably only cover one theme  Next planned is half day (or more) on use of Job submission system  We will attempt to make the format more “training oriented” (perhaps use RAL or IC training rooms)  Will circulate request for interest (6 weeks from now)  Use other meetings, where appropriate e.g. RAL (SRB) and NeSC e-Science Institute (Globus)..  But try to minimise.. TB Meetings Pete Clarke

Tony Doyle - University of Glasgow

Project Management Board  Project LeaderTony DoyleAppointed by CB  Deputy Project Leader John GordonAppointed by Project Leader  Chair of Collaboration BoardSteve LloydEx officio  Chair of Technical BoardPeter ClarkeAppointed by Project Leader  Chair of Experiments BoardNick BrookEx officio  Deputy Chair of EBRoger BarlowEx officio  Resource ManagerDave BrittonAppointed by Project Leader  DataGrid UK Project LeaderRobin MiddletonEx officio  CERN LiaisonTony CassAppointed by CERN  e-Science DirectorNeil GeddesEx officio  As proposal plus Resource Manager + DataGrid UK Project Leader + Part time Secretary at Glasgow  Three year posts (except EB 1+2)  Meets weekly via VC and monthly face to face

Tony Doyle - University of Glasgow High Level Deliverables To be re-defined to extend to “The Grid” i.e. testbed Aspiration: all that you expect from a central service? Probably not.. Tomorrow’s Discussion..

Tony Doyle - University of Glasgow High Level Deliverables Requires input from WP contacts

Tony Doyle - University of Glasgow High Level Deliverables Pre-defined at high level, but requires further iteration

Tony Doyle - University of Glasgow Projected Staff Effort [SY] WP1 Workload Management 0.5[IC]2.0 [IC] WP2 Data Management 1.5[Ggo]1.0 [Oxf] WP12 WP3 Monitoring Services 5.0++[RAL, QMW]1.0 [HW] Security ++[RAL]1.0 [Oxf] Security ++[RAL]1.0 [Oxf] WP4 Fabric Management 1.5[Edin., L’pool] WP5 Mass Storage 3.5++[RAL, L’pool] WP6 Integration Testbed 5.0++[RAL/M’cr/IC/Bristol] WP7 Network Services 2.0[UCL/M’cr]1.0 [UCL] WP8 Applications 17.0 ATLAS/LHCb (Gaudi/Athena) 6.5[Oxf, Cam, RHUL, B’ham, RAL] CMS 3.0 [IC, Bristol, Brunel] CDF/D0 (SAM) 4.0 [IC, Ggo, Oxf, Lanc] BaBar 2.5 [IC, M’cr, Bristol] UKQCD 1.0 [Edin.] Total >20.0?6.0 Resource Management

Tony Doyle - University of Glasgow DataGrid Recruitment Status Tools being developed Dec 01 Robin Middleton

Tony Doyle - University of Glasgow DataGrid-UK Status  Recruitment  EU funded posts filled & in post (1 additional from 1/2/02)  All but 2 of 15 “unfunded” posts active  Reporting  UK templates defined, with TB meetings before & after quarterly submissions  81% of effort contracted delivered (but not uniform across WPs) for WP1-7 (WP8 hard to quantify)  UK Progress (in 2001)  Architectures, Testbed-1, network monitoring, certificates & security, R-GMA, LCFG, MDS deployment, Spitfire… … Robin Middleton

Tony Doyle - University of Glasgow DataGrid-UK Effort Robin Middleton

Tony Doyle - University of Glasgow DataGrid-UK Status  EU Review  All UK deliverables passed EDG internal review and now submitted to EU  Review at CERN on 1 st March  3 external reviewers  ~40 deliverables (inc. quarterly reports)  Summary presentations only  Can expect detailed feedback of areas to address  4 th Project Conference – Paris 4 th -8 th March  Plans for 2002 – TB2  Absorb initial feedback from EU review Robin Middleton

Tony Doyle - University of Glasgow DataGrid-UK News  Licensing is still an issue  ATF reconstituted with more Applications involvement  New project web site at original address  Based at CERN  Uses EDMS for document handling  Testbed (see elsewhere at meeting)  Minor releases every 2 months  V2 in September  Transatlantic testbeds coming closer to reality  Scope being clarified  Interoperability a major issue  architectures Robin Middleton

Tony Doyle - University of Glasgow InterGrid etc  Strictly HENP-InterGrid  HICB meeting on 17 th Feb Sunday (prior to GGF)  HIJTB meet monthly by phone  Projects  BaBar testbed  GRIT – Grid Reference International Testbed  ATLAS & CMS  DataTAG started  DataTAG : WP2=QoS, etc; WP4=m/w interoperability  Collaborates with iVDGL  CrossGrid, GridStart Robin Middleton

Tony Doyle - University of Glasgow GRIT Ruth Pordes

Tony Doyle - University of Glasgow “The Grid” is an evolving concept  enabler for transient ‘virtual organisations’  anatomy: a software infrastructure that enables flexible, secure, co- ordinated resource sharing among dynamic collections of individuals, institutions and resources  Foster, Kesselman & Tuecke (2001)  evolution of and integration with web services  physiology: everything is a Grid service i.e. a service that conforms to a set of conventions for management and exchanging messages  Foster, Kesselman, Nick & Tuecke (2002)   Global Grid Forum: define a standard Grid architecture  big business and big science starting to work together  GGF4 (Toronto) next week

Tony Doyle - University of Glasgow GGF Architecture  1) scope the problem - what are we trying to produce  2) decide on the require functionality  3) modularize the functionality  4) decide on the relationships among the modules  5) identify the data that must flow among the modules  6) decide what state has to be maintained, and where  7) design the protocols for communicating among and with the modules  8) APIs  The challenge in the Grid forum is  - picked a large and amorphous problem (wide area distributed computing)  - for which there is not agreement (yet) as to what all of the functionality should be  - and even where we think that we understand the functionality requirements (e.g. data and resource management) there is not yet consensus on the modules  ANSWERS??  DataGrid++ is “The Architecture” we have adopted - look here first

Tony Doyle - University of Glasgow e-Science and Grid  We are asked to map onto this plane…  Recognisable words, focussing on “middleware”  DataGrid++=middleware  (where ++=SAM, SLAC-based developments)  Experiments=Application Area  Middleware=Application Area??  Draft 2.0 status  Response should be built up from technical definitions

Tony Doyle - University of Glasgow 2002 Challenges  Complete rollout of TB-1 and plan future upgrades  Reconvened ATF to work closely with applications  Make TB-2 a success  Deploy and exploit Tier-1/A  Applications to make good use of testbeds  Solve interoperability issues  We are part of many larger collaborations/structures/groupings - we need to collaborate/discuss engage here, and  Focus on implementation in the UK… this will tell us what works (and what doesn’t) at any given point.

Tony Doyle - University of Glasgow Technical Management: See tomorrow’s discussion Status 13/2/02: Requires upgrades to functionality (extract hosts from info. services automatically) and maintainability (integrate into testbed workplan)

Tony Doyle - University of Glasgow Status 13/2/02: Expect that Experiments will deliver production monitoring systems e.g. AliEn for ALICE (integrate into testbed workplan)

Tony Doyle - University of Glasgow Guidelines and Policies  Disk/CPU resources allocated to each “group”  Grid is based on distributed resources - a “group” is an experiment  An institute is typically involved in many experiments  Institutes define priorities on computing resources via OPEN policy statements  e.g. Edinburgh/Glasgow RAL++  ScotGrid - Priority 1: ATLAS, LHCb, Computing Science Data MiningBaBar (2/3-1/2-1/3) LHC (1/3-1/2-2/3)tomorrow’s  - Priority 2: BaBar, CDF, UKQCDe-Sciencediscussions  BioInformatics  All jobs submitted via Globus authentication - Certificates identified by user and experiment  Need to implement “priority”  Minimum amount of data to deliver at a time for a job?  Where to store files?  Which data access/storing activities have the highest priority?  Sharing of the resources among groups?  Users belong to multiple groups?  How many jobs per group are allowed?  What processing activities are allowed at each site?  To which sites should data access and processing activities be sent?  How should the resources of a local cluster of PCs be shared among groups?  REQUIRES Discussion at the Collaboration Meeting… issues will arise which require ALL Institutes to publish a policy  Longer term goals… Grid Credits issued? * Implemented by site administrators, with OPEN policies defined at each site based on e.g. case to funding authority What’s new? Ability to monitor/allocate unused resources We will be judged by how well we work as a Virtual Organisation Its all in the small print.

Tony Doyle - University of Glasgow Summary  Most effort so far directed towards DataGrid=middleware  Significant UK contributions  Startup difficulties – largely overcome  Recruitment, late filling of posts – funded & unfunded  Lost ground being recovered..  All WP with UK involvement active & contributing  Testbed-1 : so far modest, predictable teething problems (many !)  Next major challenges focus on INTEGRATION  US experiments programme - planning for today/tomorrow..  LCG programme - planning for  Testbed (focus for tomorrow’s discussions) - requires input from system managers as well as software developers.  Positioned reasonably well ~6 months into GridPP..  with many challenges ahead...

Tony Doyle - University of Glasgow Closing thought… 14/2/02 Grids are everywhere.. With many hidden words, meanings, nuances… “communication is the answer to the problem” (apparently)

Tony Doyle - University of Glasgow 3rd Collab Meeting Agenda  Feb 14th  Opening Session (Chair - Andy Parker):  11:00-11:30 Welcome and Background - Steve Lloyd  11:30-12:00 GridPP Project Status - Tony Doyle  12:00-12:30 LHC Computing Grid Project Status - Tony Cass  Middleware Status Reports and Development (Chair - Tony Doyle):  13:30-13:50 Information Services (FTREE and MDS) - Alex Martin  13:50-14:10 Information Services (RGMA) - Manfred Oevers  14:10-14:30 Mass Storage - Jens Jensen  14:30-14:50 Fabric management - Michael George  14:50-15:10 Grid Data Management - Will Bell  15:10-15:30 Networking - Paul Mealor  Experiment Developments (Chair - John Gordon):  16:00-17:00 ATLAS, CMS, LHCb and theory - Nick Brook  17:00-17:30 WP8 status - Frank Harris  17:30-18:25 BaBar, CDF, D0, SAM and others - Roger Barlow  18:25-18:30 Logistics - Andy Parker  18:30-19:30 Project Management Board Meeting  Feb 15th  Cambridge Perspective and TestBed Introduction (Chair - Robin Middleton):  9:00-9:30 Cambridge e-Science Centre I - Andy Parker  9:30-10:00 Cambridge e-Science Centre II - Mark Hayes  10:00-10:30 TestBed Technical Development - Pete Clarke  Grid Deployment I (Chair - Pete Clarke):  11:00-11:30 Testbed Tools and Release in UK - Andrew McNab  11:30-11:40 Security issues - Linda Cornwall  11:40-11:55 Testbed status at RAL - Steve Traylen  11:55-12:10 Testbed status at M'cr - Andrew McNab  12:10-12:25 Testbed status at Bristol - Marc Kelly  12:25-12:40 Testbed status at IC - Dave Colling  Grid Deployment II (Chair - Steve Lloyd):  13:30-13:40 Testbed status at Edinburgh/Glasgow - David Martin  13:40-13:50 Testbed status at Liveropool - Girish Patel  13:50-14:00 Testbed status at Oxford - Pete Gronbech  14:00-14:10 Testbed status at UCL - Paul Mealor  14:10-14:20 Testbed status at Cambridge - Mark Hayes  14:20-14:30 Testbed status at Brunel - Peter van Santen  14:30-14:40 Testbed status at Lancaster - Roger Jones  14:40-15:00 Discussion  15:00-15:30 TestBed UK Organization - John Gordon  WELCOME!