U.S. ATLAS S&C Planning Meeting - June ATLAS Software Infrastructure : Requirements and Goals at Run 2 Period Alex Undrus

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Outline  Current status  Plans for the next year  Long term perspectives This presentation reports for the following WBS items:  – Software Validation  – Librarian and Infrastructure Services

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Current Status  Some statistical data  Tendencies  U.S. contribution to ATLAS infrastructure

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Number of files in ATLAS offline release Calculated by cloc-1.62 for ATLAS nightly development release created on June 15, 2015 (cmt, InstalllArea, NICOS, platform specific. genConf, dict areas excluded). External packages are not included.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Number of lines in ATLAS offline release Calculated by cloc-1.62 for ATLAS nightly development release created on June 15, 2015 (cmt, InstalllArea, NICOS, platform specific. genConf, dict areas excluded, comments and blank lines excluded). External packages are not included.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Number of submitters and commits to offline SVN repository

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June New and persistent submitters to offline SVN repository (period of 05/16 – 06/15, 2015)  number of unique submitters during 1 month (05/16/15 – 06/15/15)  number of unique submitters during 1 year (06/16/14 – 06/15/15)

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June ATLAS Nightly System at a glance 59 in total NEW

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June ATLAS Nightly System at a glance (2) Number of ATLAS nightly jobs. Record high number of 100 daily jobs is registered on 08/01/2014. As of 06/22/2015 the Nightly System run 83 daily jobs.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Statistics Interpretation  Athena development releases contain 7 million lines  The number of lines is not growing last 5 years, but they keep changing  Software development activity was always high and it is increasing since the Run 2 start  70% increase in SVN commits  50% increase of developers community  Hundreds of new developers joined  Many new nightly branches are opened (total number reached the record high 67)  Unprecedented rate of new stable releases: 1.4 per day (~ 250 stable release Jan. 1, 2015 – June 24, 2015)  New kinds of release for targeted use

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Intensity of Run 2 Software Development ... is high  It is considerable higher than at LS1 period:  Some important LS1 software projects continue: CMAKE, ROOT 6 integration  New kind of releases for targeted use  Athena Simulation  Athena Analysis  AtlasP1Mon (for Tier 1 online monitoring)  RootCore based releases are actively developed  New compilers are probed (gcc 4.9, clang)  Release configuration management is under major change (cmake)

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Infrastructure Challenges and U.S. Contribution  All areas (code configuration, documentation, externals, platforms&compilers, QA/QC, release building and distribution, code repositories, information protection, etc.) must cope with increased workloads when the work force remains the same  Infrastructure needs both support and development (tools must be updated in sync with software they support)  ATLAS Software Infrastructure Team includes 19 persons contributing 7.5 FTE (with 50 % shortage: additional 3.7 FTE are needed for an adequate support) as of 04/02/2015  U.S. ATLAS contribution to SIT is currently 1 FTE as it appears in OTP (Alex Undrus and Shuwei Ye)  U.S. contributes to critical areas: the Nightly System (Alex Undrus), environment setup (Shuwei Ye), LXR service

 Expertise Gain  Influence on ATLAS-wide policies and decisions  Parallel and effective user support for U.S. based physicists  Librarian and User services at U.S. Analysis Center, BNL PROOF farm and Tier I Center  Capture innovations and new ideas Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Rationale of U.S. Participation in Infrastructure Projects

14 Plans for Next Year  Permanent goal: create supportive environment for code development, data processing and analysis jobs across all ATLAS sites and file systems (local, afs, cvmfs)  Key areas:  Nightly builds (details in the next slides)  Build and run-time (details in Shuwei's presentation)  Criteria of success: users satisfaction and absence of complaints  Explore new innovative technologies and tools Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

15 Plans for the Nightly System Further improve ATLAS Nightly web and database services brought by successfully completed Nightly System LS1 upgrade Add new on-demand functionality to the System Current system is for daily builds at fixed times Software coordinators increasingly request urgent nightly builds Up to 100 nightly jobs are manually restarted monthly Some branches need irregular builds separated by few days New on-demand functionality will be demonstrated on July 9 at the Annual Nightlies Workshop ( )

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 NEW ADMIN PANEL Administrative functions for privileged users, authentication via CERN SSO Main task button (restart, cancel, etc.) Current and detailed progress information

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Plans for the Nightly System (2) Complete projects according to the schedule New Nightlies CVMFS server Hot spare for the Nightlies CVMFS server (greatly improves the system reliability) ATLAS Nightly Mail Facility (personalized s about nightlies results) Adapt the System for cmake builds Experimental cmake nightlies support compilations, no tests yet Assess recent requests Expand doxygen documentation builds git repository support Continue to encourage users to use ATLAS Nightlies DB for customized views (successful experience with Trigger developers)  example on the next slide

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 New panel of Yasu Okumura with summary of Trigger related problems across major nightly branches

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June Plans for the Nightly System (technical work)  Update NICOS for new Tag Collector 3  Builds on CC7 (CERN CentOS 7)  Optimization and testing new machines on the nightly farm  CERN IT plans to replace all real hardware machines with VMs  VMs performs differently, need a lot of testing and optimization  Key problem: I/O bottleneck prevents full CPU usage  New VM machines with SSD disks allow to reach 60% CPU utilization (a success after some VM types could not be loaded above 20%)

20 Long Term Perspectives  Rising concerns:  Unclear relationship between releases  Confusion where essential parts of software are located (simulation, digitization, reconstruction, derivations)  Dissatisfaction with standard release coordination bureaucracy, ways for development of new algorithms and techniques  Single platform support, essentially no software portability  All-inclusive offline releases ( installation size 12 GB, with few tens of externals ) Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

21 Long Term Perspectives (2)  Increasing requests for smaller releases with targeted purpose (Simulation, Athena Analysis)  Ideas about software restructuring Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 Core Event Reco Analysis Core Simula tion HLT Analysis 2 Analysis 1 Reco ???

22 Long Term Targets  Improved software structure  Compact releases with targeted purpose  Expanded multiplatform support  Decrease of centralized tag validation bureaucracy and effort  Discussion on July 2 at 13: :00 – "Release build: technical session" Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

23 Perspectives for the Nightly System  Expansion to new platforms (e.g. PowerPC)  90% of the Nightly System is already portable  Moving some operations to the GRID  Making and testing releases on sites where jobs run would bring better results and save human and machine efforts  Nightly testing is already available on the GRID but attracted few tests so far  Desirable: developing communality with CMS (CMSSW), LCG (Jenkins) nightly builds  Keep users and management happy with the System is always a priority

24 Summary  U.S. contributes to key areas of ATLAS software infrastructure Infrastructure  Despite thin manpower the ATLAS Nightly System and Environment Setup procedures sustain increasing load and demand successfully  Keeping abreast of new technologies trends is at the top of U.S. contributors priorities  The Nightly System LS1 upgrade brought new database and web technologies and greatly improved user experience  The Nightly System is proactively prepared for meeting demands of new platforms (PowerPC), builds tools (cmake) and software restructuring Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015