VO Tools and Portals: Summary and Conclusions Flavia Donno, CERN

Session Organization The session was organized in several sub-sessions Virtual Organization Support: Dedicated to VO and user support support with a special focus on real experience and open issues. The Experiment Integration and Support (EIS) Team presented its experience mainly serving HEP but also other VOs such as GEANT4 and UNOSAT. General VO and User Support provided by the EGEE Global Grid User Support (GGUS). VO Portals Portals make Grid technology transparent to users. They provide for a very practical way to have users approach the Grid. The NICE company presented its product, EnginFrame, on which also Genius is based. VO Monitoring Monitoring tools are very needed by operators, middleware experts, users, and VOs to monitor resource usage and correctly tune the Grid. The INFN Team presented GridICE, present status and future developments. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Session Organization The session was organized in several sub-sessions VO Application and Software Management Among the tools needed to VO communities, the ones to integrate and run legacy applications on the Grid and manage and distribute application software at all sites participating to provide resources to a VO are fundamental. The P-Grade Portal and the GEMLCA bridge between different Grids to solve Grid interoperability problem at the workflow level and at the same time provide a way of transparently run legacy applications on different Grid infrastructures. The ETICS project aims to provide a building, testing and certification service for Grid middleware with the possibility of extensions to VO software environment and applications. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Session Organization The session was organized in several sub-sessions Other Tools and infrastructures: This subsession was dedicated to other interesting talks presenting ideas sometime not yet explored in EGEE. A proposal was presented to introduce Universal Accessibility of the Grid via a Metagrid Infrastructure. Ideas on a methodology for VO design and management F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support EIS : Experiment Integration and Support Team Help LHC Experiments integrating their production environment with the Grid Middleware and utilities. Offer support during all steps of integration process understanding of the middleware functionality testing new prototypal components getting on the LCG Infrastructure. Production is the main focus. Experiment Integration Support does not mean User Support (However….Because of the familiar and friendly relationship installed with the experiments very often we really represent the quickest first line support ….). Experiment Integration Support does not mean GOC nor CIC (nonetheless we provide customized monitors of the GRID, we manage production grid services and/or we are active in chasing site related problems weekly meeting with Operational). F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support Help with middleware functionality and usage. Integration Help with middleware functionality and usage. Propose new grid technologies to the experiments. Perform functionality tests on new technologies. Provide customized distributions and missing tools Discuss requirements and bring them to the attention of the developers Understand issues and channel them to developers or M/W experts. Experiment and User Support Provide documentation: Manuals, Guides, FAQ, tutorials. Answer first line User Support questions Monitoring experiment specific production system. Provide infrastructure expertise Monitoring/Managing GRID and Experiment Specific Services Chasing mis-configured sites and solving site-related problems Service Challenge Second Level Support (on shift) F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support Comments: The EIS model has been proven to be very incisive and very useful for High Energy Physics communities. Other VOs have tried to follow the same example, however the shortage of internal human resources has made the entire exercise difficult The need to have a more wide Grid User Support infrastructure to assist all communities during their integration and exercises with Grid has clearly emerged. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support A possible answer ? Global Grid User Support is the support infrastructure for Grid users, deployment and operation problems It offers a great variety of services to satisfy user needs at all levels It does not substitute but integrates existing infrastructures and coordinates support efforts http://www.ggus.org F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Central Application (GGUS) VO and User Support The support model in EGEE can be captioned “Regional Support with Central Coordination" The ROCs and VOs and the other project wide groups such as the Core Infrastructure Center (CIC), middleware groups (JRA), network groups (NA), service groups (SA) are connected via a central integration platform provided by GGUS. ….. … Central Application (GGUS) Deployment Support RC 1 RC X Middleware Network Operations Support TPM BIOMED ESR DS 1 DS 5 MS 1 MS 8 ROC 1 ROC 12 ROC… VO Support ALICE Interface Webportal Other Grids F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support GGUS provides a single entry point for reporting problems and dealing with the grid. In collaboration with the EGEE EIS team, the EGEE User Information Group, NA3, and the entire EGEE infrastructure, GGUS offers a portal where users can find up-to-date documentation, and powerful search engines to find answers to resolved problems and examples. Common solutions are stored in the GGUS knowledge database and Wiki pages are compiled for frequent or undocumented problems/features. GGUS offers hot lines for users and supporters and a VRVS chat room to make the entire support infrastructure available on-line to users. Special tools and grid middleware distributions are made available by the NA4/EIS team for GGUS users. GGUS is interfaced with other grids’ support infrastructures such as in the case of OSG and NorduGrid. GGUS is used for daily operations to monitor the grid and keep it healthy. Specific user problems can be directly communicated to the Grid Operation Centers and broadcasted to the entire grid community. GGUS is used also to follow and track down problems during stress testing activities such as the HEP experiments production data challenges and the service challenges. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO and User Support Comments: The GGUS system offers quite a satisfactory response to user needs. However VO Dedicated support is missing. More funding is necessary to establish dedicated support groups. Connection between GGUS and Local Helpdesks in terms of knowledge and statistics seems missing. Even if knowledge is somehow available, statistics are missing. GGUS is working with the metrics group. There is still little involvement from the VOs. This is due to many reasons: missing human resources, most of the effort for HEP covered by EIS. This is already changing. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Portals F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Portals 1.x - LAN Integration (1998) Unix-Windows integration for simplified parallel and distributed computing 2.x - WAN MetaComputing interface (1999) Standards convergence, dynamic contents management 3.x - Computing Portal (2000-2003) Web architecture, JS independence, Globus integration 3.5 – Enterprise Grid Portal (2004) Fault tolerance, Auth delegation, Authorization 4.x – Grid Gateway (2005) WS-I compliancy, user mapping, enhanced Data management, RSS notification, JSR168 portlet gateway F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Portals F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Portals F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Portals VO Portals Comments: Are a very practical way to have users approach the Grid. Different portal with different functionalities can and should be integrated. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Monitoring A VO wants to monitor a Grid in order to: to observe the composition, state and features of available resources available to its users to analyze their behavior and performance to track user activity as regards computing/storage/network resource usage to detect fault situations In the context of Grid computing, two important categories of monitoring systems are: Application monitoring Infrastructure monitoring We focus on F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Monitoring GridICE Server Site – Administrative domain Lemon or other fabric mon. tools charts HTML XML notification data aggregation and abstraction Grid Inf. Service persistent storage discovery consumers scheduler Grid Discovery Service Site – Administrative domain Monitored Entity Site Collector local publisher site consumer site publisher sensor site persistent storage F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

We focus on the following categories of users: VO Monitoring We focus on the following categories of users: VO manager actual set of resources accessible to VO members Grid operator all resources under responsibility of a Grid Operator Center Site administrator site resources offered to a Grid F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Monitoring Comments: Technical questions about the usage of GridICE. - The notification service is a very useful one: it should be integrated in the current support system. Developers working on the new notification service. Only specific events are monitored at the moment. In future working on an XML description.Integration in GGUS and CIC portal. There are more than one tool. Is this OK? In general it is good. EGEE is a service oriented project. There should not be a monopoly of a given service. The same applies to WMS, DM, etc. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management The P-Grade Portal General purpose, graphical, workflow-oriented Grid portal Supports the development and execution of workflow-based Grid applications Components of the workflows can be Sequential jobs Parallel jobs (MPI, PVM) Legacy code (GEMLCA) services Enables the exploitation of two levels of parallelism intra-job inter-job Enables to control and observe the execution of the workflow applications Enables fault-tolerant workflow execution F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management The P-Grade Portal Enables application-oriented extensions by new portlets Hides the low-level Grid details Enables the access of every important feature of the underlying Grid, e.g.: brokering service information service, etc. Easy porting of legacy and other applications to the Grid Solves the interoperability of various Grids at the workflow level and hence enables: easy porting of legacy and other applications between various Grids (e.g. from LCG to gLite) parallel execution of a workflow in several Grids F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management Proxy management Grid resources management Workflow creation Job mapping to Grid resources Workflow management and execution visualization F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management ETICS Vision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applications Mission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software. Set up the foundations for a certification process to help increasing the quality and interoperability of such software F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management Web Application Web Service Via browser Report DB Project DB Build/Test Artefacts NMI Scheduler Via command- Line tools NMI Client Wrapper WNs ETICS Infrastructure Clients F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

VO Application and Software Management Comments: There are different and sometimes more complex problems connected with VO software management and distribution. Can ETICS address those ? The ETICS team has stressed that the project focuses on application software management as well. Important is to establish QA metrics depending on the middleware/software in question. ETICS intends to do so. P-Grade is an interesting proposal for hiding the details of different flavors of Grids. Based on fat clients. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Other Tools and infrastructures UNIVERSAL ACCESSIBILITY OF THE GRID rooted in the concept of Design for All in Human Computer Interaction addresses accessibility problems in human interaction with software applications and telematic services. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Other Tools and infrastructures Various application domains benefited from research and development in this area including: interactive television and media children educational games interfaces for the physically handicapped interfaces for various devices F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Other Tools and infrastructures Methodology for the definition of a VO: VO definition in terms of resources needed, policies to be applied, software environment, human resources, workflow, scheduled events, etc. VO Modification: Define procedures to add more resources, get feedback from past experience, etc. Can some of this task be automated ? F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Other Tools and infrastructures Comments: The proposal is to extend the concept of interoperability toward universal accessibility. Universal accessibility has not been taken into account at design time Is it probably too early to speak about universal accessibility ? VO methodology definitions should be connected to operations and translated into practical procedures. Connection to support infrastructure is also needed. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006

Conclusions Very useful workshop. Quite some ideas about commonly used tools came out. Unfortunately VO communities were not really represented. We look forward to the next EGEE User Forum!. F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006