GridPP2 Data Management work area J Jensen / RAL GridPP2 Data Management Work Area – Part 2 Mass storage & local storage mgmt J Jensen

Data Mgmt – J Jensen / RAL Data Management Work Area u Metadata management n See Gavin’s talk u Local Storage Management n Cleanup temporary space n Reservations? u Mass Storage Management n Enable Grid access to storage at UK Tier 1 and 2

Data Mgmt – J Jensen / RAL Overview Metadata Local Storage Storage management SRM2 dCache Deployment Support EGEE ARDA WSRF?

Data Mgmt – J Jensen / RAL Common u Links to external middleware groups u Common support & deployment requirements u Integrate with deployment software n LCFG-ng / Quattor / Manual install n Not contractually obliged to deliver n Must use tools from EGEE JRA1/SA1 (tar/make) n Can we use EGEE CVS ? n Savannah n Work with OMII

Data Mgmt – J Jensen / RAL Storage Group u Form a Storage Group within GridPP2 u Already set up! u Requirements, evaluation, development, deployment, support, evaluation u Work with storage staff at sites u RAL, Bristol, Edinburgh (, Glasgow) u Will set up web site, mailing list soon u Regular phone conferences

Data Mgmt – J Jensen / RAL Storage Management – Pf2 u Need to support MSS at UK Tier 2 sites u Evaluate existing solutions in the UK u Support Tier 1 first, then Tier 2 n Prioritize Tier 2 in order of storage committed… n …or ease of implementation u SRM 1 required u Move to SRM 2 later?

Data Mgmt – J Jensen / RAL Storage Management – Pf3 u SRM version 1 implementations n Enstore, dCache, CASTOR, EDG SE,… u Pick one to deploy initially n Initial focus on Tier 1 u Pick one to develop and support n Which is best for supporting storage systems at Tier-2 sites in the UK?

Data Mgmt – J Jensen / RAL Local Storage Management – Pf2 u Evaluations u Clean up temporary files u Reservations – difficult to support u May be necessary to integrate with WLMS u May be necessary to integrate with batch systems

Data Mgmt – J Jensen / RAL Local Storage Management – Pf3 u Various ways to provide partial functionality n EDG Zambo n dCache n Condor n Integrate something with batch systems u Initial deployment n dCache n For batch systems, focus on PBS initially, then LSF u Evaluate which is best for long term n Additional developments may be required

Data Mgmt – J Jensen / RAL Requirements input u Other GridPP areas u EGEE Architecture Group u HEPCAL, in particular HEPCAL I, ARDA,… u EGEE NA4? u Security - JRA3 u SRM group

Data Mgmt – J Jensen / RAL Before GridPP2 u EDG SE developments n More work on SRM 1 s Currently testing with dCache client and GFAL n Improve metadata implementation n Integrate VOMS, deploy ACL stuff u Set up support structures (started already!) u Set up EGEE tracking

Data Mgmt – J Jensen / RAL Support u Work with Tier-1 and Tier-2 support staff u Use Bugzilla? Or Savannah? n Bugzilla to be set up by Edinburgh if req’d u Provide mailing lists for n Announcements n Help – integration n Perhaps internally, for developments/discussions

Data Mgmt – J Jensen / RAL Deliverables Deployment and Support u Releases to match EGEE release m and LCGn, mє{1,2}, n є{3,…}. u PM03: Initial deployment n dCache for LSM n EDG SE (and SRM?) u PM05: Release for LCG3 u PM07: Release for EGEE release 1 u PM16: Release for EGEE release 2 u PM26: Final post-EGEE release n For EGEE follow-on project

Data Mgmt – J Jensen / RAL Deliverables Evaluation and Development u Evaluations, developments design u PM08: [Storage only] Report on mass storage used in UK u PM10: Report on integration with EGEE release 1 u PM14: [Local Storage only] Report on local storage management u PM20: Report on integration with EGEE release 2

Data Mgmt – J Jensen / RAL Deliverables Management and timeline u Plans, report on interactions with EGEE, e-Science, OMII u PM01: Initial project management plan n Risk analysis, plan for evaluating and supporting storage systems, etc u PM12: Annual report u PM24: Annual report u PM36: Final project report n Describing delivered and usable software, EGEE and LCG deployment and evaluation, dissemination, contributions to int’l standards, deviations from plans,

Data Mgmt – J Jensen / RAL Deliverables Integration with EGEE u “Higher level” EGEE tracking and integration (architecture, security, …) u PM01: Report describing requirements arising from EGEE Architecture u PM11: Report describing requirements arising from EGEE Architecture u PM18: Integration plan for EGEE follow-on

Data Mgmt – J Jensen / RAL SRM 1 u Mass storage control protocol u GridFTP for data transfer u SRM 1 req’d for interoperability with DoE Labs u Some interoperability problems between various implementations n Parts of protocol open to interpretation

Data Mgmt – J Jensen / RAL EGEE u Integrate directly with n JRA1 – middleware s Data Management “cluster” s Architecture Group n SA1 – support and deployment u Also watch n JRA2 – QA n JRA3 – security u And…? n NA2, 3, 4: dissemination, training, use cases

Data Mgmt – J Jensen / RAL LCG u LCG testbed – match LCG3 n Future LCG milestones less well defined (or secret)? u ARDA n Can provide early prototypes s What is “early” – pre-GridPP2? n For Mass Storage, SE to interface with RM n For Local Storage Management, dCache u GFAL

Data Mgmt – J Jensen / RAL LCG: GFAL u LCG (will) decide to use GFAL – the “Grid File Access Library” u It was decided to interface to EDG SE using SRM 1 interface u For now using EDG RM POSIX interface SRM 1 client EDG 2.1 Storage Element Mass Storage EDG SRM

Data Mgmt – J Jensen / RAL Web services u Already using web services for mass storage control interfaces u Need HTTPS-compatible delegation u Need to evaluate WSRF

Data Mgmt – J Jensen / RAL SRM version 2 u Protocol (very nearly) defined u More advanced optional functionality n Volatile, Durable, and Permanent files n Volatile, Durable, and Permanent space u Evaluate need/requirement for this as we go along u SRM Group to be in GGF as “GSM”

Data Mgmt – J Jensen / RAL Dissemination u Work with NA2 u Work with e-Science programme n We used AHM 2002, 2003 to promote GridPP, EDG SE (posters, papers) n We will be present at AHM 2004 (September)

Data Mgmt – J Jensen / RAL Non-LCG liaison u e-Science Core programme n Liaise with e-Science GSC, ETF, STF,…? u ADS – used by diverse scientific community u EGEE – WP formerly known as 10 n WP formerly known as 9 is not in EGEE u Babar n No Grid plans yet…

Data Mgmt – J Jensen / RAL EGEE: DICOM server support The Grid Storage Element WP10 DM2 DICOM Server Metadata Encrypt, anonymise Metadata Store keyStore patient metadata Access control on metadata required; different ACLs for different types of metadata