1. 1 Open Science Grid.. An introduction Ruth Pordes Fermilab

2. Open Science Grid @ Maryland 2 Why am I at Maryland today ? The US LHC Experiments, which will take data at CERN starting next year have a global data processing and distribution model based on Grids. US CMS (and US ATLAS) are relying on and contributing to the Open Science Grid as their distributed facility in the US. Maryland is a US CMS facility. Maryland physics also includes: D0 - already using Grids for data distribution and processing; Icecube - already collaborating with Condor in Wisconsin.. I am attending the Open Grid Forum standards meeting in Washington - *and yes, I met David McNabb*

3. Open Science Grid @ Maryland 3 CMS Experiment LHC Global Data Grid (2007+) Online System CERN Computer Center USA Korea Russia UK Maryland 0.1 - 1.5 GB/s >10 Gb/s 10-40 Gb/s 2.5-10 Gb/s Tier 0 Tier 1 Tier 3 Tier 2 Physics caches PCs Iowa UCSDCaltech U Florida  5000 physicists, 60 countries  10s of Petabytes/yr by 2008  1000 Petabytes in < 10 yrs? FIU Tier 4

4. Open Science Grid @ Maryland 4 Computing Facility Tiers in CMS Tier-1 : One Center in each Geographical region of the world. Well provisioned with Petabytes of Archival Storage to accept data from and provide data to Tier-2s/Tier-Ns. Tier-2s: Several (in the US 7) University facilities that accept responsibility to deliver to the CMS experiment Monte Carlo simulations and analysis; and provide 200 TB of online data caches for analysis; and mentor the Tier-3s. Tier-3s: University facilities providing processing and online storage for the local physics and research groups, and provisioned to get data from and send data to Tier-2s.

5. Open Science Grid @ Maryland 5 What is Open Science Grid? It is a supported US distributed facility The OSG Consortium builds and operates the OSG. The farms and storage are not owned by OSG - we integrate existing resources. 5 years of funding $6M/year (~35 FTEs) starting 9/2006 from DOE SciDAC-II and NSF MPS and OCI. Cooperation with other distributed facilities / Grids to give our research groups transparent access to resources they are offered worldwide.

6. Open Science Grid @ Maryland 6 The OSG Map Aug-2006 Some OSG sites are also on TeraGrid or EGEE. 10 SEs & 50 CEs (about 30 very active)

7. Open Science Grid @ Maryland 7 Open Science Grid in a nutshell  Set of Collaborating Computing Farms - Compute Elements. Commodity Linux Farms; & Disk. optional MSS From ~20 CPUs in Department Computers To 10,000 CPU SuperComputer Any University Local Batch System OSG CE gateway OSG & the Wide Area Network

8. Open Science Grid @ Maryland 8  Set of Collaborating Storage Sites - Storage Elements. Mass Storage Systems And Disk Caches From 20 GBytes Disk Cache To 4 Petabyte Robotic Tape Systems Any Shared Storage OSG SE gateway OSG & the Wide Area Network

9. Open Science Grid @ Maryland 9 OSG Services  Grid wide monitoring, information, accounting services in support of a System.  Based on:  X509 GSI Security  Globus Toolkit  Condor products  Storage Resource Management (SRM) storage interface  VOMS/Prima/GUMS Role Based Attribute and Access Control  S/w to make it interoperable with Europe and TeraGrid.

10. Open Science Grid @ Maryland 10 Supported Software Stacks  Integrated Supported Reference Software Services — Most Services on Linux PC “Gateways” -- minimal impact on compute nodes. — Loose coupling between services, heterogeneity in releases and functionality. Independent Collections for Client, Server, Administrator

11. Open Science Grid @ Maryland 11 Middleware Me: thin user layer My friends: VO services VO infrastructure VO admins The Grid: anonymous sites & admins Common to all. Me & My friends are usually domain science specific. Middleware & Service Principles: Me -- My friends -- The grid

12. Open Science Grid @ Maryland 12 Grid - of - Grids  Inter-Operating and Co-Operating Grids: Campus, Regional, Community, National, International  Open Consortium of Virtual Organizations doing Research & Education

13. Open Science Grid @ Maryland 13 Data Transfer for LHC crucial Core Goal to Deliver to US LHC & LIGO scale in next 2 years 1 GigaByte/sec

14. Open Science Grid @ Maryland 14 Who is OSG ? Large global physics collaborations: US ATLAS, US CMS, LIGO, CDF, D0, STAR Education projects e.g. Mariachi,I2U2. Grid technology groups: Condor, Globus, SRM, NMI. Many DOE Labs and DOE/NSF sponsored University IT facilities. Partnerships e.g. TeraGrid, European Grids, Regional/Campus Grids e.g. Texas, Wisconsin…

15. Open Science Grid @ Maryland 15 OSG Consortium Partners Contributors Project

16. Open Science Grid @ Maryland 16 Current OSG deployment - across integration and production grids 96 Resources 27 Virtual Organizations

17. Open Science Grid @ Maryland 17 Smaller VOs

18. Open Science Grid @ Maryland 18 Large VOs... CMS CDFATLAS

19. Open Science Grid @ Maryland 19 Running a Production Grid

20. Open Science Grid @ Maryland 20 Running a Production Grid

21. Open Science Grid @ Maryland 21 OSG Core Program of Work Integration: software and systems. Operations: common support and procedures. Inter-Operation: across administrative and technical boundaries.

22. Open Science Grid @ Maryland 22 Release Process (Subway Map) Gather requirements Build software Test Validation test bed ITB Release Candidate VDT Release Integration test bed OSG Release Time

23. Open Science Grid @ Maryland 23 What is the VDT? A collection of software  Grid software (Condor, Globus and lots more)  Virtual Data System (Origin of the name “VDT”)  Utilities  Built for >10 flavors/versions of Linux An easy installation  Goal: Push a button, everything just works  Two methods: Pacman: installs and configures it all RPM: installs some of the software, no configuration A support infrastructure Build Software

24. Open Science Grid @ Maryland 24 What software is in the VDT? Security  VOMS (VO membership)  GUMS (local authorization)  mkgridmap (local authorization)  MyProxy (proxy management)  GSI SSH  CA CRL updater Monitoring  MonaLISA  gLite CEMon Accounting  OSG Gratia Job Management  Condor (including Condor-G & Condor-C)  Globus GRAM Data Management  GridFTP (data transfer)  RLS (replication location)  DRM (storage management)  Globus RFT Information Services  Globus MDS  GLUE schema & providers

25. Open Science Grid @ Maryland 25 Client tools  Virtual Data System  SRM clients (V1 and V2)  UberFTP (GridFTP client) Developer Tools  PyGlobus  PyGridWare Testing  NMI Build & Test  VDT Tests What software is in the VDT? Support  Apache  Tomcat  MySQL (with MyODBC)  Non-standard Perl modules  Wget  Squid  Logrotate  Configuration Scripts And More!

26. Open Science Grid @ Maryland 26 VO Registers with with Operations Center. User registers with VO. Sites Register with the Operations Center. VOs and Sites provide Support Center Contact and join Operations groups We’re all fun people!

27. Open Science Grid @ Maryland 27 The OSG VO A VO for individual researchers and users. Managed by the OSG itself. Where one can learn how to use the Grid!

28. Open Science Grid @ Maryland 28 Due diligence to Security Risk assessment, planning, Service auditing and checking Incident response, Awareness and Training, Configuration management, User access Authentication and Revocation, Auditing and analysis. End to end trust in quality of code executed on remote CPU -signatures? Identity and Authorization: Extended X509 Certificates  OSG is a founding member of the US TAGPMA.  DOEGrids provides script utilities for bulk requests of Host certs, CRL checking etc.  VOMS extended attributes and infrastructure for Role Based Access Controls.

29. Open Science Grid @ Maryland 29 Role Based Access Control

30. Open Science Grid @ Maryland 30 Training - e.g. Grid Summer Workshop Year 4 Hands on. Technical trainers. Nice Setting (Padre Island). Students got their own applications to run on OSG!

31. Open Science Grid @ Maryland 31 Use commodity networks - ESNet, Campus LANs Well network provisioned sites e.g. connected to Starlight to low bandwidth connections e.g. Taiwan Connectivity ranges from full-duplex, outgoing only, to fully behind firewalls. Network Connectivity

32. Open Science Grid @ Maryland 32 Bridging Campus Grid Jobs - GLOW Dispatch jobs from local security, job, storage infrastructure and “uploading” to wide-area infrastructure.

33. Open Science Grid @ Maryland 33 FermiGrid? Interfacing All Fermilab Resources to common Campus Infrastructure Gateway to Open Science Grid  Unified and reliable common interface and services through one FermiGrid gateway - security, job scheduling, user management, and storage. Sharing Resources  Policies and Agreements enable fast response to changes in resource needs by Fermilab users. More information is available at http://fermigrid.fnal.gov

34. Open Science Grid @ Maryland 34 Access to FermiGrid OSG General Users Fermilab Users CDF Farm FermiGrid Gateway OSG “agreed” Users D0 Farm Common Farms CMS Farms

35. Open Science Grid @ Maryland 35 Web sites www.opensciencegrid.org http://vdt.cs.wisc.edu http://osg.ivdgl.org/twiki/bin/view