ILDG Middleware Status Presented By: Bálint Joó, Jlab, USA Working Group Members: G. Beckett (EPCC, UK) T. Boku (CCS Tsukuba, Japan) D. Byrne (EPCC, UK) P. Coddington (CSSM, Australia) M. Ernst (DESY, Germany) M.Sato (CCS Tsukuba, Japan, Convener) J. Simone (Fermilab, USA), W. Watson (Jlab, USA, Convener)

Contents ● Reminder of Our Task ● Reminder of Our Architecture ● Summary of Current Discussions ● Future plans

Our Mission ● We develop standard interfaces that facilitate the interoperation of a grid of grids ● Local grids – Control Local Policies (Eg Security) – Control Local Grid Implementations ● Choice of software components

Our Architecture ● Simple veneer over local implementations – Minimally intrusive (wrapper?) – Not too burdensome to implement ● Language and platform neutral if possible ● Our choice: SOAP Web Services

Our Architecture: Components Replica Catalogue Web Service (RC) Metadata Catalogue Web Service (MDC) Security Infrastructure (Identity management access control encryption) SRM service (data management) XML ILDG service description file Logical File Name (LFN, formerly GFN)

ILDG Service File ● Publishes Service Endpoints ● Each collaboration currently has entries for MDC and RC ● Started filling this out at MW Meeting in Tsukuba ● ● ● Entries for LatFor & CSSM

Logical File Names (LFN) ● Changed to LFN from GFN to fit better with MDWG usage ● Unique and Persistent name for files ● MWWG mandated URI format lfn:// / ● Collaboration indexes service file ● Collaborations control their own path

MDC Web Service ● Maps metadata to LFNs ● Provides “Virtual XML Database” ● Prototype WSDL definition by M. Sato ● Production MDC Web Services deployed by LatFor & CSSM ● Interface definition in flux – Raw queries – variable return types – Targeted Queries – return LFNs

Replica Catalog Service ● Maps (data) LFNs to URL ● Intention was to map to SRM URL – SRM component deals with transfer ● SRM requirement temporarily relaxed – RC can return any URL ● No standard download protocol ● Client must have appropriate tool ● Prototype implementation at Jlab

SRM Service ● SRM definition v3 almost ready – Fewer backward incompatible changes than v1->v2 transition. – Migration should be simple(r) ● Popular dCache storage management software still uses v1.x (Fermilab, DESY) but Jlab uses v2. v1 and v2 incompatible ● ILDG uses v2 SRM, but SRM use is not mandated currently

Security Infrastructure ● Currently quite nebulous ● Previously defined 'ILDG Group Files' – XML Documents containing user certificates – Not very grid-like, not best match with LatFor preference/implementation ● Currently considering VOMS service ● Security not explicit in current interfaces

WG Organization ● Had very nice and productive meeting in Tsukuba in Oct Minutes on ILDG web site. ● Instituted more frequent teleconference calls – first of these was on Dec 5. Minutes on ILDG web site. Next call Jan 23, ● Middleware BLOG set up on ILDG web site to aid reporting progress.

Plans ● Finish up discussions and make a decision on using VOMS by end of February ● US basic service up with some data by end of January. ● EPCC to convert MDC GUI browser to work with Web Services ● Interoperability testing Feb-May ● Interoperable system by June 2006.

Current Discussions ● MDC – Need to fix interface ● Small inconsistencies between WSDL from LatFor and Japan – to be fixed ● Clarify semantic meaning of query functions – Raw queries – return results appropriate to query – LFN queries – return markovChainUR or dataLFN – Need document retreival queries ?

lfn://myCollab/myEnsemble E+00 lfn://myCollab/myFilename Raw Query: number(/gaugeConfiguration/markovStep [dataLFN="lfn://myCollab/myFilename"]/avePlaquette) Returns: Raw Query: /gaugeConfiguration/markovStep [markovChainURI="lfn://myCollab/myEnsemble"]/dataLFN Returns: lfn://myCollab/myFilename XPath funcion XPath Number Type XML Tags Raw Queries:

lfn://myCollab/myEnsemble E+00 lfn://myCollab/myFilename LFN Query: /gaugeConfiguration/markovStep[update="010170"] Returns: { lfn://myCollab/myFilename } LFN Query: /gaugeConfiguration/markovStep [markovChainURI=”lfn://myCollab/myEnsemble”] Returns: { lfn://myCollab/myFilename, lfn://myCollab/myFilename2,... } LFN Queries : Single Return Type: Array of LFNs for all queries

Comparison ● Raw Queries – Powerful: Need Only 1 Query function ● Can use XPath functions eg: to count matches – Query returns different types ● Eg: numbers, strings, XML Snippets – Can use to implement LFN Queries – Also good for Web interfaces, testing

Comparison ● Domain Specific (eg LFN) Queries – Less Flexible ● Can't execute arbitrary XPath functions – More logic in clients – More complex ● Need more functions: – Query/Get Config XML, Query/Get Ensemble XML – Tailored to Particular Use Cases ● Eg: Command Line Tools (find & download) – Don't need XML database necessarily

Security Infrastructure ● Can use SSL to encrypt communications with services ● Can have certificate based authorization in Web servers, Application servers, and even some databases ● Use firewalls to secure back ends ● Credential forwarding for service aggregator portals may need thought

Service (MDC/RC) Client Certificate Exchange Encrypted Connection Client Encrypted Connection Certificate Exchange Aggregate Service (MDC/RC) US Service (MDC/RC) LatFor Service (MDC/RC) UK Service (MDC/RC) Australian Service (MDC/RC) Japanese Service (MDC/RC) Certificate forwarding? Proxying? How to maintain secure connection?? Actual Database Actual Database Actual Database Actual Database Actual Database Firewall Clear Thought Needed

VOMS: Virtual Organisation Management Service ● Technology for identity and role management ● Distributed and decentralized ● Part of LCG, OSG and gLite ● LatFor moving to production use ● To be trialled in UK and Japan ● In US, Fermilab will use infrastructure on collaborative project with DESY

VOMS ● Interest in VOMS expressed by Australia ● WG to come to a decision on standardising use of VOMS by end of February 2006 – Personal worry: This is choosing software rather than defining an interface. – Pragmatism: LatFor requires users' certificate in their VOMS server. Will eventually hold everyone's certificate

LCG Software Components ● Michael recommended use of LCG software: – Rich in features, presented as RPMs through software repositories, with installation tools (apt-get, yaim) – Can install just components we need – Can use any Red Hat Enterpise Linux like system (Scientific Linux (SL), RHEL, Fedora Core) CERN supports SL.

LCG Components – Large User Community (LHC, OSG) – Used by LatFor (File Catalog – LFC, and VOMS) – In US Fermilab will have infrastructure ● BUT – Don't want to force particular technology/software. That's not our mandate !

Conclusions ● Components coming along – individual collaborations will present own status reports ● MDC interface solidifying currently ● Most groups have received funding/effort ● Most groups have local infrastructures and are now working on ILDG issues ● I have a LOT of work ahead of me.

US Middleware status ● Effort recently revitalised with arrival of Bálint at the Jlab. ● General Web Service Infrastructure: – Apache Web Server – Tomcat Application Server – Axis Web Services Framework – X509 Certificates, SSL encryption ● Prototype Data: MILC Datasets from NERSC Gauge Connection

US Middleware Status ● RC Prototype ready for some time – Need secure connection and certificate based authentication for production use. ● RC needs data in it to be useful. – In discussion with MILC to choose suitable LFNs for data in NERSC archive. Recent helpful changes in metadata schema facilitate things.

US Middleware Status ● In analysis stage for MDC – Plan: use eXist native XML database ● Pure Java ● Integrates well with Tomcat and Axis ● Comes with Web Service API ● Sophisticated XML capabilities: – Validation against Schemata, XQuery, XUpdate ● Expect very thin ILDG layer on top

US Middleware Status ● Progress with ILDG File Format – Conversion routine from NERSC format to ILDG format exists within MILC code and QDP++ using QIO library. – MILC Code can read QDP++ files and those provided by Dirk Pleiter (LatFor) – Dirk could read files produced by QDP++ and MILC

US Middleware Status ● File format (cont'd) – Only minor discrepancy ● QIO uses LIME type: ildg-data-lfn ● Dirk's code & spec give: ildg-data-LFN ● Dirk suggests case insensitivity in codes – Backward compatible solution for all sides. – Carleton willing to change QIO to use case insensitive checking of LIME types.