INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Padova site report Massimo Sgaravatto On behalf of the JRA1 IT-CZ Padova group.

Slides:



Advertisements
Similar presentations
INFN & Globus activities Massimo Sgaravatto INFN Padova.
Advertisements

Grid Workload Management (WP 1) Report to INFN-GRID TB Massimo Sgaravatto INFN Padova.
Work Package 1 Installation and Evaluation of the Globus Toolkit Massimo Sgaravatto INFN Padova.
Workload management Owen Maroney, Imperial College London (with a little help from David Colling)
WP 1 Grid Workload Management Massimo Sgaravatto INFN Padova.
EU-GRID Work Program Massimo Sgaravatto – INFN Padova Cristina Vistoli – INFN Cnaf as INFN members of the EU-GRID technical team.
GRID workload management system and CMS fall production Massimo Sgaravatto INFN Padova.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.
GRID Workload Management System Massimo Sgaravatto INFN Padova.
Globus activities within INFN Massimo Sgaravatto INFN Padova for the INFN Globus group
Workload Management Massimo Sgaravatto INFN Padova.
First steps implementing a High Throughput workload management system Massimo Sgaravatto INFN Padova
Status of Globus activities within INFN (update) Massimo Sgaravatto INFN Padova for the INFN Globus group
Evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova.
INFSO-RI Enabling Grids for E-sciencE XACML and G-PBox update MWSG 14-15/09/2005 Presenter: Vincenzo Ciaschini.
INFN-GRID Globus evaluation (WP 1) Massimo Sgaravatto INFN Padova for the INFN Globus group
Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.
INFSO-RI Enabling Grids for E-sciencE The US Federation Miron Livny Computer Sciences Department University of Wisconsin – Madison.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
EGEE is a project funded by the European Union under contract IST Testing processes Leanne Guy Testing activity manager JRA1 All hands meeting,
Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.
INFSO-RI Enabling Grids for E-sciencE CREAM: a WebService based CE Massimo Sgaravatto INFN Padova On behalf of the JRA1 IT-CZ Padova.
DataGrid WP1 Massimo Sgaravatto INFN Padova. WP1 (Grid Workload Management) Objective of the first DataGrid workpackage is (according to the project "Technical.
Grid Workload Management Massimo Sgaravatto INFN Padova.
INFSO-RI Enabling Grids for E-sciencE SA1 and gLite: Test, Certification and Pre-production Nick Thackray SA1, CERN.
Globus Toolkit Massimo Sgaravatto INFN Padova. Massimo Sgaravatto Introduction Grid Services: LHC regional centres need distributed computing Analyze.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks CREAM and ICE Massimo Sgaravatto – INFN Padova.
WP1 WMS rel. 2.0 Some issues Massimo Sgaravatto INFN Padova.
EGEE is a project funded by the European Union under contract INFSO-RI Practical approaches to Grid workload management in the EGEE project Massimo.
Summary from WP 1 Parallel Section Massimo Sgaravatto INFN Padova.
EGEE-III INFSO-RI Enabling Grids for E-sciencE SA3 All Hands Meeting 'Cluster of Competence' Experience SA3 INFN Cyprus May 7th-8th.
Status of Globus activities Massimo Sgaravatto INFN Padova for the INFN Globus group
Grid Workload Management (WP 1) Massimo Sgaravatto INFN Padova.
WP1 WMS release 2: status and open issues Massimo Sgaravatto INFN Padova.
EGEE 3 rd conference - Athens – 20/04/2005 CREAM JDL vs JSDL Massimo Sgaravatto INFN - Padova.
WP1 Status and plans Francesco Prelz, Massimo Sgaravatto 4 th EDG Project Conference Paris, March 6 th, 2002.
INFSO-RI Enabling Grids for E-sciencE gLite Test and Certification Effort Nick Thackray CERN.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Study on Authorization Christoph Witzig,
EGEE is a project funded by the European Union under contract IST LCG open issues Massimo Sgaravatto INFN Padova JRA1 IT-CZ cluster meeting,
INFSO-RI Enabling Grids for E-sciencE File Transfer Software and Service SC3 Gavin McCance – JRA1 Data Management Cluster Service.
First evaluation of the Globus GRAM service Massimo Sgaravatto INFN Padova.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks The Dashboard for Operations Cyril L’Orphelin.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
INFSO-RI Enabling Grids for E-sciencE Padova site report Massimo Sgaravatto On behalf of the JRA1 IT-CZ Padova group.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks CREAM: current status and next steps EGEE-JRA1.
A GOS Interoperate Interface's Design & Implementation GOS Adapter For JSAGA Meng You BUAA.
EGEE is a project funded by the European Union under contract IST Padova report Massimo Sgaravatto On behalf of the INFN Padova JRA1 Group.
CREAM Status and plans Massimo Sgaravatto – INFN Padova
INFSO-RI Enabling Grids for E-sciencE CREAM, WMS integration and possible deployment scenarios Massimo Sgaravatto – INFN Padova.
UNICORE and Argus integration Krzysztof Benedyczak ICM / UNICORE Security PT.
EGEE-II INFSO-RI Enabling Grids for E-sciencE IT cluster activity status (Status of WMS & CE) Francesco Prelz – IT.
JRA1/Job Submission and Monitoring Moreno Marzolla INFN Padova.
Resource access in the EGEE project Massimo Sgaravatto INFN Padova
Gri2Win: Porting gLite to run under Windows XP Platform
Workload Management Workpackage
CEMon
First proposal for a modification of the GIS schema
JRA1 IT-CZ cluster meeting Milano, May 3-4, 2004
CE-Monitor Luigi Zangrando INFN-Padova
Security aspects of the CREAM-CE
WP1 WMS release 2: status and open issues
Workload Management System ( WMS )
Preview Testbed Massimo Sgaravatto – INFN Padova
CREAM Status and Plans Massimo Sgaravatto – INFN Padova
Massimo Sgaravatto INFN Padova On behalf of the CREAM product team
The CREAM CE: When can the LCG-CE be replaced?
Francesco Giacomini – INFN JRA1 All-Hands Nikhef, February 2008
GRID Workload Management System for CMS fall production
Condor-G Making Condor Grid Enabled
Presentation transcript:

INFSO-RI Enabling Grids for E-sciencE Padova site report Massimo Sgaravatto On behalf of the JRA1 IT-CZ Padova group

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 2 CEMon & Information providers How to stay synchronous with LCG ? Last time we agreed to ask LCG to integrate our mods to support the CondorC based CEs, and then rely on the LCG information providers Actually we then found that LCG information providers can support CondorC based CEs as they are (i.e. no changes to the code) –Just a matter of configuration Tested on LSF and PBS CEs –The ones of the prototype CEMon software changed to support these new LCG information providers In the next integration build the support to the LCG information providers will be included In this process of integrating the LCG information providers also contributed in debugging some problems in the LSF dynamic provider

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 3 CEMon & R-GMA Publishing of CEMon info also in R-GMA –This can be done via R-GMA Gadget In (Gin)  Not necessary to have a LDAP server (GRIS) –This allows publishing into R-GMA CE (and possibly also CE-SE bind) information –Done and tested on the PBS CE of the prototype (lxb2077) –Instructions (to be used for the CE installer script) sent to the Iteam  No ideas when they will be integrated in the CE installer script CEMon & ServiceTools –This is basically the same service publisher stuff already done to publish the status of the WMS services –Already integrated in the CE installer script –We just needed to fix something in “getInfo” of CEMon

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 4 CEMon current activities Working to several enhancements in particular to better support the pull mode, as discussed last time –More powerful “condition” (specifying is the CE is willing or not to receive jobs) specification  More powerful than a simple RegEx  Also discussing with FrancescoP, followed approach to rely on classAds –Setting of expiration in CE info sent by CEMon –In particular, if the “condition” is false, send expired info to client (ISM)  To mark as expired the info in the ISM If the condition is false, it means that the CE doesn’t want to receive new jobs, and so that CE doesn’t have to be matched –Allowing modifying the static subscriptions (the one specified by the CE admin) without the need to restart tomcat

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 5 Simplified example of static subscription /CEMonitor GlueCEStateEstimatedResponseTime<150&&am p;GlueCEStateWaitingJobs<2&&GlueCEInfoLRMStype==&qu ot;lsf" ClassAd SEND_NOTIFICATION TRUE SEND_EXPIRED_NOTIFICATION FALSE

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 6 Simplified example of static subscription /CEMonitor GlueCEStateEstimatedResponseTime<150&&am p;GlueCEStateWaitingJobs<2&&GlueCEInfoLRMStype==&qu ot;lsf" ClassAd SEND_NOTIFICATION TRUE SEND_EXPIRED_NOTIFICATION FALSE Language used to express the “condition”: RegExp or ClassAds

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 7 Simplified example of static subscription /CEMonitor GlueCEStateEstimatedResponseTime<150&&am p;GlueCEStateWaitingJobs<2&&GlueCEInfoLRMStype==&qu ot;lsf" ClassAd SEND_NOTIFICATION TRUE SEND_EXPIRED_NOTIFICATION FALSE RegExp or ClassAds “Condition” expression Must be XML compliant

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 8 Simplified example of static subscription /CEMonitor GlueCEStateEstimatedResponseTime<150&&am p;GlueCEStateWaitingJobs<2&&GlueCEInfoLRMStype==&qu ot;lsf" ClassAd SEND_NOTIFICATION TRUE SEND_EXPIRED_NOTIFICATION FALSE If the condition is true, then sends notifications to client (e.g. ISM)

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 9 Simplified example of static subscription /CEMonitor GlueCEStateEstimatedResponseTime<150&&am p;GlueCEStateWaitingJobs<2&&GlueCEInfoLRMStype==&qu ot;lsf" ClassAd SEND_NOTIFICATION TRUE SEND_EXPIRED_NOTIFICATION FALSE If the condition is false then sends “expired” notifications to client (e.g. ISM)

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 10 Migration path First phase –CE ldif info “calculated” by info provider translated into classads –ClassAds evaluation to evaluate if the condition is true or false –Ldif sent to client  No change in the ISM purchaser; just needed to relink with the new API (which supports the new subscription) –Static subscription file can be updated and it is reread on the fly (no necessary to restart tomcat) –Code ready: under test –Not yet committed  Where ? RC1 or HEAD ? Second phase –Classads sent to client instead of ldif –ISM purchaser doesn’t have to perform anymore the ldif  classads translation –Also include expiration of information in classads  Default will be sending expired info when condition is false … but if we are going to work on HEAD and not in RC1, I guess we don’t need to have two distinct phases

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 11 CEMon: other issues How can the ISM CEMon purchaser reject notifications from misconfigured or just unwanted CEMons ? CEMon could send notifications on an authenticated channel (using CE credentials), and then AuthZ (via LCAS ?) in the ISM ?

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 12 CREAM Code in CVS but not yet linked with the build system (CruiseControl) yet because of a circular dependency Basically submission and cancellation implemented Worked on implementing sandbox staging also via DIME, but some problems found –Decided to postpone this item: with Axis 1.2 (which we hope to rely on after RC1) the scenario will probably be completely different AuthN and AuthZ integrated Missing piece is credential mapping, as we discussed last time –JRA3 contacted: they are planning to provide in the second year a su- exec wrapper  Looks like the right approach, even if all the details are missing  Doesn’t exist yet, and not too clear when it Not detailed plan yet in JRA3 –The other option would be integrating BLAH with LCMAPS Testing and debugging of submit and cancel –Mini testbed in Padova where CREAM and CREAM CLI are tested

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 13 CRM workshop Rome, February Goals –To enable information exchange about experiences and to discuss about opportunities for common approaches –To make progress towards the definition of standard "compute resource management" (CRM) interface allowing different implementations to co-exist and interoperate  as is happening for the different storage management systems thanks to the SRM interface Attendees –Condor, EGEE/Glite, Globus, INFN-GRID, LCG, NorduGrid, Unicore Outcomes –Agreement to work towards a common Job Description Language  “All the participants of this group are committed to a constructive evaluation of JSDL in the near future” –The area of Compute Resource Interface was considered more complex  Start by comparing the different interfaces  “All the participants of this group will meet again (in Rome in October?) to assess the possibility of developing an agreed set of interfaces for job submit and management” –Agreement to draft a document comparing the approaches to job description and submission of the different projects by 18th April 2005

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 14 Prototype and testing activities Maintenance of the WMS of the prototype Support to testing team Weekly prototype meetings Testing of weekly candidate release done according to what was discussed and decided in Catania –This is basically working –What’s the process now that we don’t have to release anymore weekly tags ? Other testing of new stuff Debugging of the CE installer script –Used to install the CE PBS (lxb2077)

Enabling Grids for E-sciencE INFSO-RI Massimo Sgaravatto - INFN Padova 15 Other MPI jobs –Working for Globus based (LCG2) CEs –Basically not tested yet with CondorC-Blah based CEs (because of other priorities)  Likely not be fully working with this week’s tag Work on configuration (JMX) –See report on the configuration meeting Distributed super-scheduling –See Matteo’s talk