1 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Bockjoo Kim U of Florida Bockjoo Kim U of Florida CMS Software Installation

2 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration CMSSW Installation Sites on OSG T3 T2 9 T3 9 T2 (Caltech) (FIU)

3 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration CMSSW on US T3 Sites(as of 10/21/08) SitenameCE (OSG)CPU slotsCMSSW T3_US_Buffalo?? u2-grid.ccr.buffalo.edu/fork 21122_1_10, … T3_US_Colorado heposg01.colorado.edu/? 162_1_10, … T3_US_Cornell nys1.cac.cornell.edu/fork 1202_1_9, … T3_US_FIT uscms1.fltech-grid3.fit.edu/fork 1602_1_10, … T3_US_FIU ?/? T3_US_FNALLPC ?/? T3_US_Kansas ?/? T3_US_Minnesota ?/? T3_US_Olemiss?? umiss001.hep.olemiss.edu/fork 422_1_10, … T3_US_Princeton ?/? T3_US_Rutgers ?/? T3_US_TTU antaeus.hpcc.ttu.edu/fork 2522_1_10, … T3_US_UCLA saxon.hosted.ats.ucla.edu/fork 442_1_10, … T3_US_UCR top.ucr.edu/condor 402_1_10, … T3_US_UIowa ?/? T3_US_UMD ?/? T3_US_Vanderbilt vampire.accre.vanderbilt.edu/for k 14082_1_10, …

4 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Centralized CMS Installation  CMS requires CMSSW to be installed centrally for T2  For T3, one can install it locally or centrally  non-CMS OSG sites, it can be done centrally  All installations are uniform ($OSG_APP/cmssoft/cms/$SCRAM_ARCH/cms/c mssw/CMSSW_X_Y_Z)  Central Installation on OSG done by me  LCG/EGEE counter part (serveral people) is doing this on LCG/EGEE

5 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Life Cycle 1. Software Development 2. Release Build 3. APT Packaging 4. Tagging Release in XML for Publication 5. Release Announcement 6. Deployment on Grids/Local 7. Release Deprecation Pre-Announcement 8. Tagging for Release Deprecation Release 9. Release Deprecation Announcement 10. Remove Deprecated Release

6 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Features of Software Deployment Tool  Condor-G Job Submission with Customized Installation/Verfication Scripts  Central Run and Bookkeeping DB  Possible Install via Grid Proxy Based Portal: Different DN Can Install Different Site Simul.  Cronized Installation In Parallel per Release  Production CMS Soft Release on OSG T2/T3  Twiki Pages :  Portal :

7 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Consideration For Installation  Different Linux Flavors  64-bit at 32-bit mode  Network Insulation (non-CMS sites)  Missing Tools on WN : apt-get, rpm, rpmbuild (non- CMS sites)  Different Shared File Systems (Lustre, AFS, etc)  Dedicated Slot : Needs to be done before any other CMS job starts  Disk Space Issue : ~50GB required typically  Many files ==> Installation time gets longer  Keeping many releases ==> Deprecation becomes terrible and should be done on time

8 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration APT and File Systems  APT (RPM tool) uses lots of locks  At least, four different FS accros OSG sites : NFS, AFS, GPFS, lustre  GPFS, lustre, other not well-known FS requires special treatment for locks using local FS ( not many exotic FS, though)  Many files ==> Installation time gets longer  Keeping many releases ==> Deprecation becomes very important ==> (64-bit OS/8GB required in the worst case)

9 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Deprecation  Mostly same as installation  Most time consuming part is find dependency and dependency bookkeeping  NFS stale file handle and GPFS : ‘rm -rf’  Recycle dependency calculation from one site  Deprecation is also automated and cronized

10 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration CMS Software Packaging Tool  CMS employs APT packaging for software distribution  CMS provides packaging tools for initial setup and update  RPMs are installed in a non-root area  CMS provides release publication : this allows deployment people for immediate deployment

11 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Automated CMS software Deployment  Well-established OSG CMS sites needs software deployment promptly  CRON is used for automation  List of well-established OSG CMS sites  Database(DB) for bookkeeping  Scripts checks and executes:  New release that needs to be deployed  Deployment status from DB  Deployment job submission/resubmission  Installation job self-monitoring  notification  Repeatition of all if necessary

12 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Implementation of the Deployment Tool Local Scripts A Remote OSG Site Application Area Info. Publication CMS Grid Users OSG Software GridCat GridCat Client CMS Pkg Tool RPMs CMS Pkg ToolRPMs CMS APT Repository Tool Design OSG CMS X509 Web Portal MySQL DB Deploy Script Condor-G Job Execution Script CRON Site Catalog Site Availability Site List DB Update DB Check Execution Script Job Status

13 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Grid Portal Screen Shot

14 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration SAM and SW Installation Monitoring Related with SW Installation SAM monitors SW Installations

15 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration List of Problems and Solutions  RPM version mismatch -> Rebuild RPM DB  rpm-wrapper error 88 -> insufficient disk  rpm-wrapper error 92 -> permission problem  “Could not get lock” -> FileSystem, use local disk  memory alloc (4byte)-> Remove releases  memory alloc (8byte) -> Use 64-bit apt-get

16 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Statistics of CMS SW Deployment Installed Releases _4_1 TO 1_2_ _2_0 TO 1_7_ _6_10 TO 2_1_ SitesOSG T2, T3, and Opportune 33 sites Installations (ReleasesxSites) Removals (ReleasesxSites) Years

17 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Timing and Files SiteTierTime to Install*FSNo of FilesSize (GB) Caltech254 minXFS1.7 x Florida2868 minNFS/lustre1.3 x MIT266 minNFS1.5 x Nebraska257 minNFS1.5 x Purdue252 minNFS1.6 x SPRACE2100 minNFS1.7 x UCSD298 minNFS1.6 x UERJ251 minNFS1.4 x Wisconsin271 minAFS1.9 x TTU380 minlustre1.3 x UCR356 minNFS?1.5 x Mississippi3186 minNFS?1.6 x UCLA348 minNFS?1.6 x Vanderbilt378 minGPFS1.2 x Buffalo345 minNFS?1.1 x FIT3763 minNFS?1.1 x Cornell3---- FIU3----

18 CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab CMS Commissioning and First Data Stan Durkin The Ohio State University for the CMS Collaboration Summary  More than 717(1024) installations/removal for CMS software have been deployed on OSG,  Automated installation works quite efficiently with almost no problem theses days  Recently, most installation problem comes from rpm db limitation and can be fixed via 64-bit apt-get  T3 sites are not required to deploy CMSSW centrally. At the moment, there are 9 sites under the regular/central deploy list.  If other T3 sites wish to be included, please let me know