A DΙgital Library Infrastructure on Grid EΝabled Technology ETICS Usage in DILIGENT Pedro Andrade

Digital Libraries Powered by the Grid 1st ETICS Review 2 Outline  DILIGENT Project Overview  Usage of ETICS  Build and testing  Adoption timeline  Components and users  Difficulties  Benefits  New Requirements  Conclusion

Digital Libraries Powered by the Grid 1st ETICS Review 3 Project Overview  DILIGENT goal is to develop a Digital Library (DL) test-bed infrastructure that allows members of dynamic virtual research organizations to create on- demand DL based on shared computing, storage, multimedia, multi-type content and application resources  39-month project (from Sep 2004 to Nov 2007)  13 partners in 9 countries

Digital Libraries Powered by the Grid 1st ETICS Review 4 Project Overview  2nd year of the project completed  Close cooperation established with other projects  EGEE: gLite middleware, PPS infrastructure  ETICS: build and test system  Software release  release Alpha available  DILIGENT two user communities portals (cultural heritage and environmental studies) running  developments ongoing for release Beta

Digital Libraries Powered by the Grid 1st ETICS Review 5 Project Overview  DILIGENT is a large consortium developing a complex software and services infrastructure  the infrastructure is composed of 10 sites running DILIGENT and gLite services  3 sites part of EGEE PPS infrastructure  different technologies and standards  gLite, GT4, WSRF, WSRP  20 service classes  ~100 sw components  3 architectural layers

Digital Libraries Powered by the Grid 1st ETICS Review 6 Project Overview Certification Build Deployment Testing System Testing Packaging Pre-Production Development and Bug Fixing Available BuiltIntegratedTested Released BugsBugsBugs

Digital Libraries Powered by the Grid 1st ETICS Review 7 ETICS Usage - Build and Testing  Build activity  modeling of build components and their dependencies  creation of night builds (automatically every night)  creation of integration builds (upon decision)  generation of javadoc API documentation  generation of reports (from ETICS logs)  Deployment testing activity  modeling of deployment test dependencies  execution of project-level deployment tests  generation of reports (from ETICS logs)

Digital Libraries Powered by the Grid 1st ETICS Review 8 Nov Sep Jul May Mar ETICS Usage - Adoption Timeline ETICS officially adopted project structure and first components modeled first experimental builds first experimental deployment tests 2006 all components modeled successful build of all components successful deployment testing of all components release Alpha completed web app edit functionality access given to all partners ETICSDILIGENT read-only web app and CLI edit functionality direct DB access read-only CLI

Digital Libraries Powered by the Grid 1st ETICS Review 9 ETICS Usage - Components and Users  102 DILIGENT software components  53 WSRF services (43 release Alpha)  40 libraries (35 release Alpha)  9 portlets (4 release Alpha)  described in ETICS by  23 ETICS subsystems  300 ETICS components  ~500 ETICS configurations (~280 Head and ~220 Alpha)  21 users from 8 different locations (6 countries)  3 project administrator  18 subsystem administrator

Digital Libraries Powered by the Grid 1st ETICS Review 10 ETICS Usage - Interface

Digital Libraries Powered by the Grid 1st ETICS Review 11 Difficulties  Early adoption  progressive availability of ETICS clients functionalities  web app edit functionality available late (now solved)  web app and CLI functionalities not synchronized (now solved)  some needed functionalities still not available (requirements)  lack of documentation  Starting with ETICS requires some investment  Clients user-friendliness can be improved  System is more build-oriented then test-oriented  Project dependency (ETICS short-term lifetime)

Digital Libraries Powered by the Grid 1st ETICS Review 12 Benefits  Automatization of processes  build and deployment testing automatization  defined dependencies are automatically solved  build repository can be automatically populated  automatic report generation  Support for distributed developments  different code repositories  Integration activity facilitated  management of different builds simplified  subsystem integrations possible

Digital Libraries Powered by the Grid 1st ETICS Review 13 Benefits  Distribution of responsibilities  project administrators  subsystem administrators  Being an early adopter, our feedback is well accepted and considered as relevant by ETICS  test functionalities less exploited by other projects  several meetings with ETICS people  constant feedback  Effective support from the ETICS team  Large public infrastructure (not yet exploited)  Multi-platform support (not yet exploited)

Digital Libraries Powered by the Grid 1st ETICS Review 14 New Requirements  Usage of private resources in ETICS infrastructure  privacy is guaranteed  enables the usage of the ETICS reports  Notification mechanism  Cloning of configurations  configuration dependencies should also be cloned  Locking of configurations  useful when different external code repositories are used to guarantee code freezing  Improve dynamic dependencies update

Digital Libraries Powered by the Grid 1st ETICS Review 15 Conclusion  DILIGENT release Alpha successfully built and integrated using ETICS  The collaboration has been working well  The benefits achieved through the adoption of ETICS show that the effort invested is being repaid  Future releases are planned to exploit ETICS

Digital Libraries Powered by the Grid 1st ETICS Review 16 Questions ?