Ian Bird CERN, 17 th July 2013 July 17, 2013

Agenda HEP Software collaboration HEP Computing future July 17, 2013

Background The Concurrency Forum has successfully brought the community together (  Pere) - To focus on problems of more optimally using modern computer architectures - Sharing knowledge across HEP - Forum for dissemination The scope of the problem we need to address for the next 10+ years is significant - Needs to draw in available expertise - Needs to provide training and education for next generation of physicists We should not re-invent the wheel – HEP may have lost some skills in this area - Other sciences have not – there is expertise and experience available  We need a more formal framework within which to plan our (HEP) activities and which allows us to construct collaborative activities where necessary July 17, 2013

Some of the risks There are a lot of activities on HEP-HPC The danger of - Diverging - Reinventing the wheel (inside and outside HEP) - Spending time in wrong directions...is very real Relations with other labs & sciences & industry is essential for items such as - HPC & concurrency programming - Data preservation - Algorithms - Big Data - Future of Grid / Cloud / … - and so on… July 17, 2013

Resource constraints Optimistically(!) funding for computing in the next few years will be ~flat - Worse – it competes with detector upgrades – there is only 1 pot of money, and this sharing is not being managed by the experiments - we must optimise our use of [CPU, memory/storage, bandwidth, power] & human resources HEP cannot go it alone - External funding for computing people will be limited, - HEP must build synergies with other sciences and organisations - Must be sure to focus our scarce efforts on the real problems that we have July 17, 2013

HEP is not (just) LHC LHC – long term programme with significantly increasing data rates ILC/CLIC TLEP FAIR Etc. Will all be facing the same problem – significant need for resources in a time of limited funding July 17, 2013

A HEP software collaboration Formalise the Concurrency Forum as a collaboration Provide a framework to - Increase level of collaboration and involvement - Coordinate and review existing efforts - Recognise and credit the contributions of FAs, institutes and individuals (shown to be important in the past) - Roadmaps, priorities for investigation and development - Create new projects, make them visible, and provide review - Launch new projects where R&D need is clear - Clarify that the scope is not limited to LHC – but is HEP-wide - Also a mechanism to collaborate and bring in existing expertise from other sciences and industry - Would very much help in bidding for funds (especially if this was joined up EU/US etc) July 17, 2013

A Software Collaboration We should also recognise that HEP is a very successful community – we are well integrated compared to almost every other science BUT: we are fragmented in computing - Notable exceptions – GEANT4, ROOT - All HEP experiments do essentially the same workflows, but we always (without fail) start over again for the software for every experiment We should not miss the huge opportunity to build commonality July 17, 2013

More commonality? Common distributed computing models Common software frameworks for processing and analysis Common libraries – now that incorporate all the best practices of high performance computing (vectorised, multi-threaded, parallel, …) This would allow us to have a common optimisation team/expertise/… to help experiments optimise their code - Commonality will help bring the best practice to all of HEP codes (in the same way GEANT4 incorporates physics understanding) HEP is also missing a huge opportunity to provide our expertise to the wider scientific community - ROOT to supplement R, Matlab, ??? - etc July 17, 2013

Commonality – why? Cannot expect to train the entire community to become experts in HPC/Parallel software;  must encode the developments/optimisations in common frameworks and libraries Developing concurrent frameworks and parallel algorithms will be a significant effort and we cannot afford to duplicate it July 17, 2013

R&D Techlab PH/SFT and IT are establishing a lab to support the development and benchmarking of parallel software on a variety of architectures - Would be activity as part of the software collaboration - Cluster of machines of interesting architectures - Environment for benchmarking and comparison of software developments - Provide access to range of code development and analysis tools to aid in parallelising and vectorising code - Means to collaborate with interested industrial partners, eg via openlab July 17, 2013

But what about broader HEP computing? Software optimisation is only 1 aspect How will HEP manage computing in years? - How should the models change, with Terabit networks, computing advances across the board in science, consumer, social, … - How can HEP make the best/optimal use of developments from other areas? - For how much longer does HEP run all/part of its own computing? July 17, 2013

How does HEP computing adapt? HEP-wide project on future computing? - Coordinate all the projects in HEP that address future evolution of computing and software - Governance by the HEP community itself - Launch new projects in areas where there are identified holes - Act as the focal point for collaboration with other sciences and industry We have experience in this level of collaboration We have a bigger problem now than we have had in the past HEP computing needs to take a change of direction now July 17, 2013

Timelines The software collaboration should be formed by the end of this year The broader scope of a project for HEP future computing needs some community discussion - Aim for a workshop in Q4 2013? July 17, 2013