1. 1 Open Science Grid Middleware at the WLCG LHCC review Ruth Pordes, Fermilab

2. 2 OSG Commitments to the WLCG OSG is relied on by the US LHC as their Distributed Facility in the US. Resources accessible through the OSG infrastructure deliver accountable cycles for the US LHC experiments. OSG interoperates with many other infrastructures in managerial, operational and technical activities. OSG cooperates specifically with the EGEE to ensure an effective and transparent distributed system for the experiments.

3. 3 Current OSG deployment 96 Resources across production & integration infrastructures 27 Virtual Organizations including operations and monitoring groups >15,000 CPUs ~6 PB MSS ~4 PB disk

4. 4 OSG Consortium & Project Consortium policies are to be open to participation by all researchers. Project responsibilities are to operate, protect, extend and support the Distributed Facility for the Consortium. OSG is co-funded by DOE and NSF for 5 years at $6M/year starting in Sept ‘06 currently including major deliverables for US LHC, LIGO and STAR; well used by CDF and D0; and potentially other experiment major deliverables in the future.

5. 5 Project Management CS/technology providers part of management teams. LHC well represented in the management structure. External project technical leads (Globus, CDIGS, SOS, LIGO-PIF) part of the Executive Board. Partners (WLCG, TeraGrid, EGEE etc.) represented through liaisons on the Council. # # # # # # # # # # # # #

6. 6 Will review distribution each year 34.3Total FTEs 4.0Management & Staff 8.5Extensions in capability and scale. 2.5Education, outreach & training 2.5Engagement (Year 1 only) 6.5Software release and support 4.5Security and troubleshooting 5.8Facility operations FTEs Effort Distribution in Year 1

7. 7 OSG Middleware Infrastructure Applications VO Middleware Core grid technology distributions: Condor, Globus, Myproxy Virtual Data Toolkit (VDT) core technologies + software needed by stakeholders: e.g.VOMS, CEMon VDS, MonaLisa, VO-Privilege. OSG Release Cache: VDT + OSG specific configuration + utilities. ATLAS Panda, DQ etc CMS cmssw, LFC, etc. Bio blast, charmm etc. LIGO LDR, Catalogs etc. User Science Codes and Interfaces Existing Operating, Batch systems and Utilities. OSG Middleware is deployed on existing farms and storage systems. OSG Middleware interfaces to the existing installations of OS, utilities and batch systems. VOs have VO scoped environments in which they deploy applications (and other files), execute code and store data. VOs are responsible for and have control over their end-to-end distributed system using the OSG infrastructure.

8. 8 Applications & Middleware OSG does not provide application-level tools  OSG provides midleware  VOs build on top of the middleware OSG packages, tests and distributes common middleware needed by multiple VOs

9. 9 VO Responsibilities VOs make their priorities clear for the common middleware. VOs contribute to the testing of new OSG releases. VOs consider commonality of and are prepared to contribute middleware that they develop or adopt. VOs often use (and therefore harden) new components in the VO Environment before they are part of the common middleware.

10. 10 The VDT

11. 11 What software is in the VDT? Security  VOMS (VO membership)  GUMS (local authorization)  mkgridmap (local authorization)  MyProxy (proxy management)  GSI SSH  CA CRL updater Monitoring  MonaLISA Accounting  OSG Gratia Job Management  Condor (including Condor-G & Condor-C)  Globus GRAM Data Management  GridFTP (data transfer)  RLS (replication location)  DRM (storage management)  Globus RFT Information Services  Globus MDS  GLUE schema & providers  gLite CEMon

12. 12 Client tools  Virtual Data System  SRM clients (V1 and V2)  UberFTP (GridFTP client) Developer Tools  PyGlobus  PyGridWare Testing  NMI Build & Test  VDT Tests And.. Support  Apache  Tomcat  MySQL (with MyODBC)  Non-standard Perl modules  Wget  Squid  Logrotate  Configuration Scripts

13. 13 Licences All s/w can be freely redistributed. All BSD or GNU licences except:  Monalisa kernel.  Pacman copyright. We work with the software providers on agreements.

14. 14 Agreements for S/W in the VDT When software is added a questionnaire is filled out. Addition of the software is agreed to by the Executive Director. In future we are including a security audit.

15. 15 Integrated Security Risk assessment of middleware is included in security management, operations, and technical controls:  Auditing of s/w  Checking of the software repositories. Etc. Security contacts at each site for communication and incident response. The OSG Security Officer decides, with consultation, the priority and mechanisms for addressing a vulnerability. Security patches are treated with high priority for testing and deployment.

16. 16 Software Support VDT/OSG team provides front line support, then works with software providers to solve problems reported by administrators and users. Issues:  Some of the s/w is supported as best effort.  Some of the s/w has support funding for the short term only. Current plans:  “Open Source” like support is ok for some components.  OSG Consortium members may step up to provide support for others.  OSG will have to consider support within the project.

17. 17 Support for S/W in the VDT ComponentOwnerContact ApacheApache Foundationhttp://www.apache.org/ Berkeley Disk Resource Manager (DRM)LBNL SDM ProjectArie Shoshani CEMonEGEEClaudio Grandi/JRA1 Certificate Authorities in the VDTOSGAlain Roy Certificate ScriptsLBNLDoug Olson Clarens & jClarensCaltech/US-CMSConrad Steenberg CondorCondor ProjectMiron Livny EDG Make GridmapEGEE Claudio Grandi/JRA1 Fault Tolerant ShellCondor ProjectMiron Livny Fetch CRLEUGridPMADavid Groep Generic Information Provider (GIP)UofIowaShaowen Wang GlobusGlobus ProjectIan Foster GLUE information from Laurence FieldWLCGLaurence Field GLUE SchemaEGEESergio Andreozzi/JRA1 GratiaFermilabPhilippe Canal GUMSBNLJohn Hover

18. 18 And.. ComponentOwnerContact KX509NMIUniversity of Michigan MonALISACaltech/US-CMSIosif Legrand MyProxyNCSAmyproxy-users@ncsa.uiuc.edu list. NetloggerLBNLBrian Tierney PacmanBUSaul Youssef PRIMAFermilabGabriele Garzoglio PyGlobusLBNLKeith Jackson RLSGlobusCarl Kesselman Squidhttp://www.squid-cache.orgsquid-bugs@squid-cache.org. SRMCPFermilabTimur Perelmutov Tclon sourceforgehttp://tcl.sourceforge.net TomCatApache Foundation http://tomcat.apache.org UberFTPNCSAgridftp@ncsa.uiuc.edu. VDT – scripts, utilities OSGAlain Roy Virtual Data System (VDS)UofC, ISIIan Foster, Carl Kesselman VOMS and VOMS AdminEGEEVincenzo Ciaschini/JRA1

19. 19 Release Process (Subway Map) Gather requirements Build software Test Validation test bed ITB Release Candidate VDT Release Integration test bed ( test services, system, applications,interoperability) OSG Release Time Day 0 Day 180 Decrease errors in production Make installations easy, quick and foolproof. Make releases and updates faster Support different sub-sets of software for different purposes.

20. 20 Gather Requirements Example Gathering US LHC requirements are given high priority.

21. 21 Steps to Release the VDT Build the software and distributes binaries. Integrate the software packages:  Figure out and deal with dependencies and differences in version dependencies between software.  Provide for easy installation using Pacman (which installes and configures) and RPM (installs only)  Provide automated installation configuration scripts to allow useful defaults and accept parameters at install time.  Make functional packages (Client, WN, Server, VOMS etc) Test everything on thirteen Linux OS platforms (and growing) using NMI build and test facility.

22. 22 Using NMI Sources (CVS) Patching NMI Build & Test Condor pool (~100 computers) … Build Test Package Build RPMs Pacman Cache Binaries Test Users Contributors

23. 23 OSG Integration and Provisioning Before a VDT release, we deploy it on a small validation/VDT testers grid with the additional OSG layers:  Check end to end functionality of OSG software.  Install and check OSG software layer  Check bugs are fixed in the OSG environment. ~15 sites on Integration TestBed deploy new release to validate:  Major applications  Interoperability (between OSG releases and between OSG and EGEE, TeraGrid)  Scalability  Write documentation OSG Release DocumentationOSG Release Documentation

24. 24 OSG Releases After testing and validation, a production release is made. Sites are free to upgrade “when ready”: not in lock-step. OSG supports the two most recent major releases No guarantee of backwards compatibility in all services, and new services bring new capabilities. Knowing the specific environment an application finds at each site becoming more important!

25. 25 Next Steps Schedule and function priorities driven by needs of US LHC. Balanced by need to prepare for medium- longer term sustainability and evolution. Goal of 2 major releases a year with easier minor updates inbetween.

26. 26 Upcoming VDT Work Provide smooth updates to a running production service:  In-place vs. on-the-side  Preserve old configuration while making big changes.  Make updates an hour not a days work. Support for more platforms?  More linux variants and needs of EGEE.  AIX needed by NERSC.  Mac OS X being tested by STAR, TIGRE, …  Solaris? Add dCache/SRM packaged in RPMs and tested on existing storage/data management testbed.

27. 27 OSG 0.6.0 TaskPriority Gratia1 Improve ease of upgrade of one VDT version to another1 BDII2 About 50 small bug fixes and enhancements2 New GIPs2 gLite client tools for interoperability purposes2 New VDS2 dCache/SRM as "RPM"3 tclGlobus3 Update platforms we build on3 New Globus - WSGRAM and GT43 CEMON 4 SQUID 4 Globus Virtual Workspaces,for Edge Services Framework5 Job managers/GIPs for Sun Grid Engine5 Switch from MySQL to MySQL-Max5 Upgrade Monalisa to the latest version5 MDS 45 VOMRS5 glexec6 Grid Exerciser6 Production release of the VDT (1.4.0)6 Expect to rely on VDT 1.3.12 or 1.4.0 Release early 2007. Some components being tested for earlier installation:  Gratia accounting  CEmon for Resource Selection Service.

28. 28 Summary The Open Science Grid infrastructure is a part of the WLCG Collaboration. OSG supports and evolves the VDT core middleware in support of the EGEE and the WLCG as well as other users. The OSG software stack is deployed to interoperate with the EGEE and provide effective systems for the US LHC experiments among others.