John Gordon EMI TF and EGI CF March 2012 Accounting Workshop

EMI INFSO-RI Storage Compute SSM and existing cpu Outline

EMI INFSO-RI dCache StoRM DPM Castor BestMan Gratia Gstat Cloud Storage (Unicore) (ARC) The middleware in scope for EGI and WLCG Storage Solutions

EMI INFSO-RI To produce a design All have achieved this Now need to correlate that the designs are compatible in the fields published and the semantics used. And the timescales for implementation and testing. EMI Milestone

EMI INFSO-RI WLCG Use Case WLCG_GlueSchemaUsage-1.8.pdf WLCG_GlueSchemaUsage-1.8.pdf Required fields/values StorageSystem = Storage Element FQDN StorageShare = StorageArea – Needs checking for consistency SE, SA, Storage Token The SubjectIdentity property MUST be present in the record – It MUST contain Group = VO name – It MAY contain: UserIdentity = X.509 UserDN of the owner of the data. GroupAttribute = VOMS FQAN EGI Profile for StAR

EMI INFSO-RI Allocated Space – equivalent to wallclock time for cpu. Shows resource blocked by VO. – – Discussed and agreed in OGF since document published SITE – EGI and NGI deal with SITEs. Defined in GOCDB and used in many other tools inc BDII for resource discovery. – I thought this had been included after previous discussions. Surprised to see not. – Very Important to EGI StAR Issues

EMI INFSO-RI APEL Test service in place. A days work to add a new AMQ topic and database schema and tweak the record loader to the storage record format and schema APEL currently uses its own data format Fieldname: value Where the fieldnames match the UR fields Intention to accept XML document but not yet in place. APEL Specifics

EMI INFSO-RI CAR – latest APEL job schema has been changed to align fields with CAR names. Issues with semantics. APEL currently uses epoch timestamps, and seconds durations CAR has adopted ISO standard date and time forms. CREAM – doesn’t publish directly. Fieldname changes will be adopted in next emi-apel-client release. (not EMI-2.0) ArcCE - ???? Unicore – already adopted CAR so may have semantic issues mentioned above. CPU

EMI INFSO-RI Parallel Jobs MPI users pushing for consistent accounting of parallel jobs This is relevant to whole-node use and how one treats cpu/wallclock. Record nodecount and ncpus(cores) for batch jobs In new APEL schema for SSM publishers – Need new apel-client for others. Don’t understand how to present the data. – Portal deals with summaries. – What is a parallel job? UR doesn’t record this. – On many sites, parallel jobs will get swamped in all the serial jobs so average speedup and nodes/job etc will not be meaningful.

EMI INFSO-RI Summary Records – Gratia (OSG) – SGAS – EDGI – Gridsafe (Globus) JOB Records DGAS Unicore QCG (Mapper) CERN NL-T1 France The systems/infrastructures/sites testing SSM SSM Users

EMI INFSO-RI Summary Records SSM Users

EMI INFSO-RI Final Production testing today Issues raised by Tania this morning – Worldwide tree in portal – Verification – summary republish? Not instantaneous – SAM tests at site level. Gratia (OSG)

EMI INFSO-RI NorduGrid developer in Sweden left. Post unfilled Finland also run SGAS and have volunteered to implement SSM publishing Hope NorduGrid will adopt. SGAS

EMI INFSO-RI k+ compute nodes Looking at publishing summaries EDGI

EMI INFSO-RI Unit tested Globus to Gridsafe and Gridsafe to APEL End to end tests next. Gridsafe (Globus)

EMI INFSO-RI DGAS Have written a requirements document Some issues already addressed.

EMI INFSO-RI Unicore Have released a version of CE with APEL publishing Test status unknown

EMI INFSO-RI QCG Compute MAPPER End of April

EMI INFSO-RI CERN Moving to production now Slides

EMI INFSO-RI NL-T1

EMI INFSO-RI France CC-IN2P3 publish summaries by direct database insert. Are still considering options of SSM, apel- parser/client or publishing via a central French site which

EMI INFSO-RI Next One by one move systems from test broker network to production. Quarantine records for comparisons Swing production publishing When all existing database inserters have migrated, move database to new schema. – Don’t need to wait for new publishers – Apel-client will continue to work – Introduce apel-cleint based on SSM.

EMI INFSO-RI Other accounting issues?

EMI is partially funded by the European Commission under Grant Agreement RI