SEE-GRID-SCI Branko Marovic JRA1 Leader University of Belgrade JRA1 overview and planning PSC01 Meeting, Athens, May 2008 The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no

PSC01 meeting, Athens, 22-23/05/082/28 JRA1 Objectives (DoW) Capture commonalities across scientific fields in terms of application requirements on Grid middleware Define development areas for middleware plug-ins and application-level services to cater to application demands and provide improvements to current infrastructure services Implement application-specific services and middleware extensions Provide a baseline of operational tools from the perspective of application and end-user friendliness and define development areas Develop and deploy operational tools easing the application- specific monitoring and control Quite vague objectives – must be grounded ASAP!

PSC01 meeting, Athens, 22-23/05/083/28 Common JRA1 goals “A seamless environment to a number of research communities”  Meteorology, Environment, Seismology (MES) Improve usability of the infrastructure and Grid services for the SEE- GRID-SCI user communities and their applications Address issues not addressed by the existing middleware and infrastructure. Services/plug-ins should  Have the capability to provide benefits to a wider range of ERA and Grid users  Some should be discipline-neutral to a certain extent  Built upon the existing middleware (gLite)  Incorporate present security and operational infrastructure. Improve manageability of the infrastructure  Research the application-focused features of operational tools  Develop new tools  Extend existing tools towards the needs of the scientific communities, VOs, and applications Tools should  Add functionality and improve both the flexibility and reliability of the tools  Facilitate the deployment, monitoring and control of applications. Development of tools and middleware plug-ins will be jointly supported by NA4 and JRA1 activities

PSC01 meeting, Athens, 22-23/05/084/28 JRA1 numbers 24 months 87.5PMs 395k€ 12.9% of the overall requested PMs, 12.3% of the total budget RTD focused on the development of few services and tools for a target group of application communities  Contributions to SA1 and NA4  Limited number of specialised partners  Strong commitments (RTD!)  Pay per performance  Others can contribute through SA1 and NA4 CERNIPPICITUBITAKSZTAKIUoBLUKIMUOBRBITOTAL 5,5015,009,0012,004,005,008,0027,002,0087,50

PSC01 meeting, Athens, 22-23/05/085/28 Tasks JRA1.1 – Application requirements analysis  Leader: UoB, B.M. JRA1.2 – Operational tools analysis  Leader: IPP, E.A. JRA1.3 – Implementation of application- specific services  Leader: TUBITAK, C.O. JRA1.4 – Design, implementation and assessment of the application-control tools  Leaders: IPP, E.A. (d/i), ICI, A.S. (asmnt) JRA1.5 – Application extensions deployment and assessment  Leader: UKIM, B.J. Metrics  MTJRA1.1 - Application-level services developed M12: 2, M24: per VO  MTJRA1.2 - Operational tools developed M12: 2, M24: 4

PSC01 meeting, Athens, 22-23/05/086/28 Risks R-JRA1.1: Dependency of application service development on integration with low-level gLite middleware components  Opening date: M06  The project significantly relies on the developments of EGEE for the provision of middleware components of gLite. Although previous agreements have been established with EGEE regarding the integration, if needed, of SEE-GRID-SCI application-level plug-ins with gLite releases, there is a potential risk in delays by EGEE integration activity.  Minimise software dependencies and produce truly pluggable application-level services to avoid the integration risk. If integration delays are inevitable, aim to run the application without specific application add-ons. R-JRA1.2: EGEE middleware components and operational tools  Opening date: M03  Developed operational tools that are extensions of and/or are based on the operational EGEE tools and middleware. In case of tools or middleware instabilities and/or major interface changes there is the risk that JRA1 developed tools will fail their purpose or be incompatible with the new tools and/or middleware.  Functional Grid middleware is necessary for any Grid operation. The fall-back solution: using latest stable middleware and tools versions. Consider this risk from the very beginning and ensure the development of lightweight and generic tools that can easily adapt to middleware changes. Close follow-up of EGEE development plans will ensure the adequate development planning.

PSC01 meeting, Athens, 22-23/05/087/28 Deliverables DJRA1.1 - Grid application requirements and commonalities assessment  M06, 7 PM, UoB-RCUB, Editor: B.M. DJRA1.2 - Application-specific monitoring and control tools design  M08, 12 PM, IPP, Editor: E.A. DJRA1.3 - Application services design report  M12, 10 PM, UoB-RCUB, Editor: B.M. DJRA1.4 - Application services implementation and deployment report  M22, 30 PM, UoB-RCUB, Editor: B.M. DJRA1.5 - Assessment of operational tools developed  M23, JRA1 28,5 PM, ICI, Editor: A.S.

PSC01 meeting, Athens, 22-23/05/088/28 Milestones MJRA1.1 - Baseline analysis of operational tools  M03, IPP: wiki, informal report MJRA1.2 - Application commonalities assessed and areas of development identified  M06, UoB-RCUB: DJRA1.1 MJRA1.3 - Operational tools implemented and running  M16, IPP: wiki, informal report MJRA1.4 - Selected application services implemented and deployed  M24, UoB: DJRA1.3, demonstration at review

PSC01 meeting, Athens, 22-23/05/089/28 JRA1 harmonisation Clear separation in DoW high-level PERT, JRA1 tasks, deliverables and milestones - try to establish a synergy between application-level services and operational tools developments through  Shared presentation of offered services and operational tools to user communities  Using operational tools to monitor some application-level services  Feed services with data from operational tools (statistics, configurations) Possible discrepancy between DoW JRA1 commitments and actual needs of applications  Must harmonise with NA4 now when we have user communities onboard Analyse the commonalities across disciplines and suggest common approaches  Can we transcend application and community borders with plug-ins and tools? Address above in NA4 (MES uses/apps) and JRA1 (services/tools) surveys

PSC01 meeting, Athens, 22-23/05/0810/28 Expected impact of JRA1 Ease the involvement of new users, applications and resource providers Automating and simplifying common and burdensome tasks Wider influences  Results available to other regional initiatives  Provide its most successful tools and components to European projects and initiatives that have the leverage to provide them to other players – EGEE, OMII  Take part in European and global standardization efforts and bodies, especially the Open Grid Forum Contributions to  Optimum development and use of eInfrastructures  Establishing a future rich Grid environment for the needs of European Research Aea

PSC01 meeting, Athens, 22-23/05/0811/28 Development of app. services 1.Assessment of target applications’ requirements and available components and solutions. 2.Definition of goals and scope; specification of key features, functional requirements and draft interfaces. 3.Plan for reuse, adaptation and modification of existing solutions and development of necessary additions; detailed specification and assignment of related activities. 4.Building of solution components and development documentation; producing functional specifications and testing procedures. 5.Testing in a realistic environment with pilot applications; handling of detected problems; review of development and deployment documentation. 6.Deployment of production-level solutions through activity SA1, starting operational usage and support; employment with all interested communities and target applications; assessing user experiences and accomplished results.

PSC01 meeting, Athens, 22-23/05/0812/28 Application-level services 1. Assessment of requirements and solutions (JRA1.1, DJRA1.1, MJRA1.2, M06)  Define development areas for middleware plug-ins and application-level services, initially  Data access and management  Multidimensional visualization  Interactivity  Inputs and expertise that go beyond local groups in all three communities  Usage, processing and visualization of acquired/calculated spatial data  Location and management of data used in collaboration and resource sharing 2. Selection and specification of commonalities (JRA1.1, DJRA1.1, MJRA1.2)  Based on expertise of participating partners and user communities 3. Implementation plan (JRA1.1–JRA1.3)  Selection of the designers, developers, testers and reviewers of services and plug-ins 4. Development (JRA1.3)  Design (DJRA1.3)  Implementation (DJRA1.4) 5. Testing (JRA1.3, DJRA1.4, MJRA1.4)  Test and consolidation with one application  Reuse  Inclusion into additional target applications if identified, or  Offer for usage to the wider community  Fine-tuning of plug-in interfaces, implementation, documentation and deployment 6. Production-level deployment & dissemination (JRA1.5, DJRA1.4, MJRA1.4)  Operational usage and support  Wider dissemination and assessment  Feeding to WLCG/EGEE/gLite teams for potential assessment (EGEE RESPECT?)  Integration with the gLite foundation services if necessary – although not envisaged

PSC01 meeting, Athens, 22-23/05/0813/28 Meteorological user community services Led by UOB Only expectation that are not specific for a single local group will be addressed Potential services mentioned in DoW  UOB (data access and management, interactivity - generic)  Extension of SEE-GRID File Management Java API with application-dependant metadata and dataset geo-referencing  Job-pooling?  Application-specific events  RBI (multidimensional visualization - generic)  Investigate multidimensional visualisation by defining a standardized inter- application communication  Integration with an meteorology application  FGA: cooperate in scientific visualization technologies and their development and implementation using grid with new scientific visualization methodology The above may bee too generic - more attention may be needed regarding  Streamlining of forecast ensemble runs using different models  Fusion of ensambles (spatial blending, identification of representative outcomes from ensembles)  Post processing/interpretation from the perspective of end consumers

PSC01 meeting, Athens, 22-23/05/0814/28 Environmental user community services Led by IPP IPP will analyze the possibility of ensuring (generic)  QoS for job start/job execution  Utilization of available performance data for improved throughput ICI and its third parties (UTCN, NCIT, UVT) will identify Grid requirements for satellite image processing in environmental applications and other similar domains  ESIP platform  Grid based satellite image processing software  SOA providing the basic functionality for Grid and Web applications  Develop a methodology for application development on this platform  Experiment this approach on a pilot application developed using this platform according to the NA4 activity workplan  Investigate usability for other application areas

PSC01 meeting, Athens, 22-23/05/0815/28 Seismological user community services Led by TUBITAK Design of plugins integrating applications and the middleware  TUBITAK  Extension to the high-level programming interface tested with SDA, to cater for Seismic Risk Assessment  Massive Digital Seismological Signal Processing with the Wavelet Analysis  UKIM (generic)  Middleware extensions which will enable easier and more reliable job execution. Seismological data repositories  4 Tbyte of data is foreseen to be produced per year from the stations of the participating seismology observatories. No foresight of computational requirements for applications  Storing and archiving of this huge amount of data will need deploying and testing of new grid services like dCache, DPM and FTS Applications are not computationally intensive, thus SEE-GRID- SCI computing resources will be adequate

PSC01 meeting, Athens, 22-23/05/0816/28 Analysis, extension and integration of services Overall process management by UOB TUBITAK  Contribute to the application requirements analysis which will assist to the application services design  As NA4 leader, oversee the applicability and feasibility of all designed tools and services for all core VOs.  NA4 survey of the 3 user communities & analysis of their requirements  Start immediately through detailed applications questionnaire  The questionnaire should also capture JRA1 related requirements In areas of seismology and meteorology SZTAKI will  Support the analysis of existing solutions, their extensions and integration, focusing the application-specific requirements by the help of its Grid Application Support Centre  Analyse the possibility of the design and integration of two plug-ins in P-GRADE Portal (already involved in the EGEE RESPECT program) in order to ensure the dissemination and exploitation CERN  Alignment of SEE-GRID-SCI JRA1 developments with EGEE SA1 and JRA1 through  Avoiding of overlapping in implementation  Prevention of implementing application services based on or oriented towards infrastructure elements and services that will be phased out  Check for existing analyses within other EGEE-related projects of user and application requirements not already covered by the existing middleware  R-JRA1.1 watchdog  Alignment of SEE-GRID JRA1 assessment practices with EGEE SA3  If needed, be a liaison between the SEE-GRID-SCI JRA1 team and EGEE SA3 (Integration, Testing and Certification) and JRA1 (Middleware Reengineering) M12: 2, M24: per VO

PSC01 meeting, Athens, 22-23/05/0817/28 Preliminary survey of application services and operational tools Piloting/testing of surveys proved to be very useful After the start of SEE-GRID-SCI we conducted a survey of those who are about to develop something within JRA1 13 propsals from 6 countries  4 tools  6 services  3 tool/services 7 with Concept status 11 usable in < 1 year

PSC01 meeting, Athens, 22-23/05/0818/28 Preliminary survey of application services and operational tools Acronym or short name SEE-GRID-SCI partner acronym CountryServiceToolStatusUsable after Logwatch modulesRBICroatiaXConcept2 - 6 months SELinux for gridRBI, CICCroatiaXConcept6 months - 1 year Advanced-workflow development tool and orchestration service UKIMFYR of MacedoniaXConcept1 - 2 years User-perspective monitoring tool UKIMFYR of MacedoniaXConcept6 months - 1 year P-GRADE PortalSZTAKIHungaryXXFinished (grid)0 / done ESIP (Environment oriented Satellite Data Processing) Platform UTCNRomaniaXIn development6 months - 1 year ESIP/UVT (front-end or app?)UVTRomaniaXConcept1 - 2 years No-mercy ticketing ICIRomaniaXConcept6 months - 1 year Data Management Web PortalUOBSerbiaXFinished (grid)Under a month Event LoggerUOBSerbiaXXFinished (cluster)2 - 6 months Job BinderUOBSerbiaXXIn development2 - 6 months SEE-GRID File Management Java API UOBSerbiaXFinished (grid)0 / done SDS (Seismic Data Server)ULAKBIM TUBITAKTurkeyXConcept2 - 6 months

PSC01 meeting, Athens, 22-23/05/0819/28 Operational tools Research the application-focused features of operational tools and develop new or extend existing tools  Facilitate the deployment, monitoring and control of applications  Complement the current monitoring tools  Add functionality to existing ones  Improve tools’ flexibility and reliability The core of tools should, from the point of view of application developers and users  Collect data relevant to the infrastructure operations and make it available to the operators and users  Optimize the performance and reliability  Automate  Operational tasks and procedures related to application deployment and running  Follow-up of discovered problems by creating tickets

PSC01 meeting, Athens, 22-23/05/0820/28 Tools extension and development Analyze and evaluate existing operational tools (JRA1.2) against the SEE-GRID-SCI infrastructure and application requirements through an analysis of the existing operational tools (MJRA1.1, M03) Design and implement the operational tools easing the application- specific monitoring and control (JRA1.4, DJRA1.2, MJRA1.3) Finally, the activity will carry out the assessment of the operational tools developed (DJRA1.5) IPP will be responsible for the design and implementation of the application-specific monitoring and control tools ICI will contribute CERN  Check for existing operational tools that in some other EGEE-related contexts address scientific communities and applications (not only infrastructure)  Support the analysis of existing EGEE operations and user support tools  Investigate their usability and areas of improvement or extension  Be a lead liaison between the SEE-GRID-SCI JRA1 team and the EGEE SA1 and, if needed, EGEE SA3 (Integration, Testing and Certification)

PSC01 meeting, Athens, 22-23/05/0821/28 Candidate tools from DoW Collect operational application data and make it available to the operators and users from a web-based and web-service based front-end  IPP  Lead the design and implementation of a centralized operational tool providing web interface to relevant operational data focusing on application features  UoBL  Web interface integrated with existing GRID service monitoring tools such as BBmSAM, to create one-stop-shop for easy monitoring of applications and resources  IPP and UKIM  Mechanisms for gathering statistics about use and performance of the main Grid services (CPU, storage, file transfers) by the target user applications and communities, integrated into a single point of access for users via the above web service interface. Develop tools that automate some operational tasks and procedures which are particularly related to the deployment and running of the grid applications  ICI  Study the interactions between existing operational tools  Design and implement an interface between the monitoring and control tools and the trouble ticketing system  Further developing the trouble ticketing system in order to provide a better application-specific support. Other extensions  UoB  Web-based tool for application-specific event logging and analysis  Assess the operational aspects of usage of SEE-GRID File Management Java API and Web-based Data Management Front End in administration of storage elements and management of files, file replicas and metadata.  SZTAKI  Focus on a workflow oriented middleware portal extension, which enables all the research communities of SEE-GRID-SCI to use a grid-aware workflow repository M12: 2, M24: 4

PSC01 meeting, Athens, 22-23/05/0822/28 User operational tools (1/2) ARDA dashboard -  The dashboard collects information on job processing, data management, transfer tests, monitoring of the reconstruction at Tier-0, or monitoring of site efficiency of LHC experiments (ATLAS, ALICE, CMS and LHCb). It works on different grid flavors (OSG, LCG, gLite). CRAB - Cms Remote Analysis Builder -  CRAB generates scripts and additional data files for each job which are submitted directly to the Grid using BossLite to interface to the Grid scheduler, as well as for logging and bookkeeping.  CRAB supports any CMSSW based executable, with any modules/libraries, including user provided ones, deals with the output produced by the executable, provides an interface to CMS data discovery services (DBS and DLS), which are completely hidden to the final user. It also splits a task (such as analyzing a whole dataset) into smaller jobs, according to user requirements. FCR  FCR allows the VOs to define a preference on Grid resources, optionally taking the SAM test results in account as well. They can select the set of Critical Tests for all services, set of Site Resources (CEs and SEs) to be used by the VO and set of Central Service Nodes ( Note : this will be used in the future) UIPnP – User Interface Plug & Play  This is the easiest way to access the grid and it will allow you to use the shared resources of nodes that already belong to the Grid. UIPnP contains all files in a unique directory without any RPM. It is intended to be used by any user (no root privileges are needed!) of commonly used RedHat platforms, without changements of system configuration. Parallel Pipe -  Tool for executing piped commands in a parallel environment (Is this a service/commonality?)

PSC01 meeting, Athens, 22-23/05/0823/28 User operational tools (2/2) Migrating Desktop -  Migrating Desktop is a framework that allows the user to access transparently the Grid resources, run sequential or interactive, batch or MPI applications, monitoring and visualization, and manage data files. The graphical user interface integrates and makes use of number of middleware and integrates the individual tools into a single product providing a complete grid front-end. OCM-G - Online Monitoring of Grid Applications using OCM-G  The purpose of the OCM-G is to provide on-line information about a running parallel / distributed application to application-development-support tools, specifically performance analysis tools, like the GPM tool. Using the information from the OCM-G, tools are enabled to obtain performance measurements of the monitored application, related to, for example, delay and volume of communication, CPU usage, etc. PHPMySRB –PHPMy Storage Resource Broker -  PHPMySRB is friendly web user interface for defining and manipulating collections of files, as well as for creating personal data collections and sharing, searching, replicating, and more without installing any client software. SRB is a system in Academia Sinica, Taiwan used for the long-term preservation of the digital contents. ELFI - EGEE Grid storage in a local filesystem interface -  ELFI is a filesystem interface to the LFC catalog and EGEE SE's (both SRM-enabled and "classic SE"). With ELFI, you can see the entries in the LFC catalog as files in a locally-mounted filesystem, and directly operate on the replica contents: read/write operations on the local filesystem are acted as read/write operations on a remote SE via the SRMv2 and GSI-RFIO protocol. GridAssist -  This tool provides access to Grids directly from the user desktops. It has an intuitive graphical user interface which makes it very easy to use in order to create and run scientific data processing jobs and it takes care of the transfer of the data between the resources.

PSC01 meeting, Athens, 22-23/05/0824/28 Conducting collaboration Collaboration in development and use of JRA1 results   Gridification guide  Piggybacking NA4 and VO collaboration activities  Workshops/trainings?

PSC01 meeting, Athens, 22-23/05/0825/28 Intellectual property rights Additions/modifications to readily available software  Use Free / Libre / Open Source Software (FLOSS) whenever possible  The same user license as the original software, commit back to the source tree  Copyright of new source code: the consortium (+original authors)  Changes in the a non-FLOSS  Require permission from the copyright  Negotiate status of changes and copyrights Development of completely new software  FLOSS license  Copyright: the consortium or authors  Later external community additions - shared FLOSS copyright

PSC01 meeting, Athens, 22-23/05/0826/28 Actual issues NA4 applications survey JRA1.1 application requirements and commonalities survey MJRA1.1 – Baseline analysis of operational tools - M03 DJRA1.1 – Grid application requirements and commonalities assessment - M06 MJRA1.2 – Application commonalities assessed and areas of development identified – M06 Discussion  Dependency of application service development on integration with low-level gLite middleware components.  Maintenance of existing gLite services, development in standardisation area  Recommendation: Minimise software dependencies, do not hardwire to the middleware  Test-environment needed?  JRA1 developments are initially deployed locally  Test-environment for services – significant overhead  Most likely compromise of the infrastructure by DoS-like behaviour  Disperse load among VO services  Are current monitoring and ticketing tools sufficient?  Quick fix – can we use Training VO?  Review this decision when more experience is gained  Can we transcend application and community borders with plug-ins and tools?  Important element of commonalities analysis Immediate actions  Further articulation of service/tool proposals  Articulation of the needs of user communities

PSC01 meeting, Athens, 22-23/05/0827/28 Articulation of the needs of user communities (1/2) Must be completed by the end of the summer! Finding of other analyses of user and application requirements and examples of integration of operational tools with applications (CERN) would this process a lot. 1.Preliminary offering of common services and monitoring tools – done  DoW  Preliminary survey (UOB)  Draft list of candidate tools (CERN) 2.Outlining of community, applications, and infrastructure (SA1) involvement –NA4 survey + 2 weeks  Surveying of target applications (NA4 survey, TUBITAK, C.S.)  Infrastructure-driven requirements for the tools (SA1, MJRA1.1, IPP, E.A.)  Analysis of existing operational tools and their possible extensions towards applications and communities (inputs from SA1, CERN, VOs and applications). This is also the input for MJRA1.1 artifacts that will be later incorporated into DJR1.2 (IPP, E.A.) 3.Presentation of offered services and operational tools – June  Service (UOB, B.M.) /tool (CERN, D.S.) descriptions (refine the current list, agree ob required details with SA1 and VOs)  Input on other analyses (CERN, D.S.)  Survey among JRA1 developers (UOB, B.M.)  All who are intending to develop services and tools should fill the questionnaire or/and prepare introductory presentations of services and tools (All JRA1 developers)  Mandatory service/tool presentations for (3)?  If so, do we really need a questionnaire? …

PSC01 meeting, Athens, 22-23/05/0828/28 Articulation of the needs of user communities (2/2) 4. Matchmaking with interested applications (second phase of NA4 survey?) (TUBITAK, C.S, UOB, B.M.) - July-August  Review outputs from survey (3) from the perspective of user communities, applications, and their interaction with the infrastructure  Identification of known/expected infrastructure related issues (directly from SA1 after NA1 applications survey)  Conducting the survey among applications –Offer already identified preliminary services and tools –Identification of possible outstanding commonalities (non-covered issues) –To be filled by developers of all applications (Representative developers of all MES application) 5. Analysis of commonalities (MJRA1.2, DJRA1.1, UOB, B.M.) – September-next PSC  Analysis of survey (4)  Assignments for developments not covered by offered services and tools, if any (4) and (5) will provide the necessary inputs for both DJRA1.1 "Grid application requirements commonalities assessment“ and DJR1.2 "Application-specific monitoring and control tools design“