DCC Workshop Input from Computing Coordination Simone Campana, Torre Wenaus Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week Introduction The mandate for this talk: “We thought that this could be a blue skies talk looking at Run 3 and beyond, what does and doesn’t scale, event streaming services, etc.  How could we reimagine data management and production systems interacting in the future, what granularity will we need to control data processing for HPC workflows, etc.” … and so this is (not) what I will talk about. Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week Introduction We are defining the roadmap for a Run-4 computing model We like an adiabatic approach, where new concepts and components are prototyped and evaluated in the context of the existing ecosystem Such components can go to production much earlier than Run-4 or be abandoned. This is how we think we should do R&D
 Data Curation and Characterization plays a central role in the model we are defining. Simone Campana – ATLAS Week 13/03/17

Axioms (https://en.wikipedia.org/wiki/Axiom) The granularity of our data processing is the event: we process one event at the time. We organize events in files because this is what file systems supports, we organize files in datasets because this is practical, we organize datasets in containers to characterize data
 Most of our data is “cold” data. We write it once, we access it O(10) times, peaked in time. Treating all data as equally “hot” has a cost. Treating all metadata as “hot” has a cost 
 Most of our data is reproducible. RAW data is not reproducible, all the rest is reproducible. “Very complicated, labor intensive, organizationally expensive, error prone to reproduce” == “irreproducible”. Treating all data as irreproducible has a cost Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week Events The event service works today with fine grained data at the event (range) level. Presently this fine granularity is short-lived DCC could take on a scalable, flexible, extensible means of recording such fine grained information. (Quasi) persistently Simone Campana – ATLAS Week 13/03/17

Event Streaming and Caching The Event Streaming Service would be the complement of the Event Service, for asynchronous delivery of data to be processed at fine granularity. Events or event collections A first implementation could deliver client-side pre fetch. In a more sophisticated scenario, a central intelligence mediates the dialog between the WFMS and the DDM systems, complementing the data transfer capabilities of DDM with a server side data access system It can be complemented with hierarchical caching based on data meta-information and access pattern statistics Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week The DCC “whiteboard” Supplements the capability to “annotate” meta information ‘Data in play’ whiteboard: at any given time, a subset of datasets/collections is ‘in play’, in use in the system. As this data is manipulated, replicated etc. knowledge about it could be dynamically cached in a whiteboard at the event collection level The whiteboard can flexibly receive arbitrary information associated with particular event collections (eg via tags). Information can be auto-generated from the system or from users 
 Usage Examples Physics collections in use by an analysis group could be annotated as such in the whiteboard, with consequent special treatment Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week The DCC “whiteboard” “Processing in play” whiteboard Tasks currently (or recently) in active processing could also benefit from a whiteboard. e.g. for monitoring purposes. Also, others could add to task whiteboard entries with annotations and information.
 A “request level” whiteboard would have its uses also
 could cross correlate prodsys requests with spreadsheets entries managed by PMG/MC prod (via programmatic API) to automate the refreshing of information on submission and processing status
 Requests associated with a particular analysis group, paper or CP group could be tagged as such, and this tagging propagated through the downstream processing and data products Simone Campana – ATLAS Week 13/03/17

Simone Campana – ATLAS Week The DCC “whiteboard” The definition of “in play” is flexible. Annotations can be made persistent after the entity they refer to is not “in play” anymore. If we accept the concept of hierarchical storage, we have to accept the concept of hierarchical meta-storage (different latency for different levels) How many whiteboards do we need, which technology, which architecture... You do not expect me to answer all that, right? The ‘whiteboard’ approach is an R&D/exploratory approach that gives an easy means of adding and accessing information to play with possible uses, quickly prototype an idea. Simone Campana – ATLAS Week 13/03/17

More ideas and use cases DCC being an essential piece in ensuring reproducibility. Needs to be implemented consistently across components DCC as a great opportunity to reduce complexity: simple decisions like “how to call a campaign: MC16c of MC17” today are forced upon us by complexity (information being sometime disperse, eventually consistent, hard coded) Simone Campana – ATLAS Week 13/03/17