1. Co-Design Breakout A. Maccabe & M. Sato Park Vista Hotel Gatlinburg, Tennessee September 5-6, 2014

2. Participants Dr. Mitsuhisa Sato, RIKEN Barney Maccabe, ORNL Robert Harrison, BNL Galen Shipman, ORNL Scott Klasky, ORNL Shaun Gleason, ORNL Dr. Kenji Ono, RIKEN, AICS Jeff Larkin, Nvidia Bronson Messer, ORNL Tjerk Straatsma, ORNL (Materials) Hooney Park, ORNL (Life Sciences) Sadaf Alam, CSCS (Life Sciences) Pat Worley, ORNL (Global Change) Seung-Hwan Lim, ORNL (Engineering) Jeremy Archuleta, ORNL (Engineering) David Bernholdt, ORNL (Fusion) Miwako Tsuji, RIKEN

3. Break-out Agenda TimeActivityWho 8:30Overview of the breakout Send out emissaries Life Sciences: Hooney Park, Materials by Design: Tjerk Straatsma, Global Change Prediction: Pat Worley, Computational Engineering: Jeremy Archuleta, Nuclear Sciences: Miwako Tsuji Fusion Sciences: David Bernholdt 8:45Robert Harrison, Scott Klasky, Kenji Ono, Mitsuhisa Sato 10:00Break 10:30Introductions of remaining participants with reactions to overview talks 11:30Identification of key challenges, from a computer science perspective

4. Break-out Agenda (continued) TimeActivity NoonLunch 1:00Re-write charge questions 1:30Draft answers to charge questions 3:00Break 3:30Finalize answers to charge questions 4:30Integrate issues identified by emissaries 5:30Adjourn

5. Co-design Models for co-design –Embedding Embed computer scientists in each application team Embed application scientists in each computer science team –Define and provide abstractions possibly subject to abuse Obsolescence Symmetry: What’s good for Apps is good for CS –CS needs abstract representations of applications Benchmarks, mini apps, skeletons, motifs, etc –Apps need abstract representations of machines Programming models and languages (parcels, X10, Chapel), simulators, performance prediction tools (e.g., Aspen), etc The role of software engineering? –What is the research challenge? Why co-design now? –Computation is changing –Explore flexibility on both sides: apps and systems Re-examine boundaries –Need to ensure continuity into the future

6. Examples of Systems co-design questions Hardware/NIC –What if the NIC can inject messages directly into an L1-cache? –What if the NIC provides computational capabilities? Collectives, matching, atomics, etc Streaming remote data to memory (bypass storage system)? Application profiles to guide compiler optimizations? How to manage different types of memory/storage? –What is the memory access pattern? –How much cache? –How much NVRAM? Which data reduction strategies can be deployed in storage or communication? Where can we tolerate overheads? –OS Noise, etc

7. Programming Models Programming models must be more holistic –Encompass more than nodes and machines Not MPI+X, but integration of I/O, storage, rest of world –End-to-end design Expose the whole context for computation so we can address the real problem No point in doing co-design in areas that aren’t the bottleneck We need a hierarchy of abstractions –MPI + X Good for nodes & machines Misses the broader context –Hadoop Encompasses the full data context and enables moving computation to data Too inefficient for scientific applications Network to the outside world –On-node interconnect, In-machine interconnect, Machine room interconnect, “rest of world” Multiuser, Interactive Supercomputing

8. Commodity – why it’s hard to define the programming model What is the right commodity –CPU + Cell did not become the commodity –CPU + GPU has shown to be much more sustainable commodity What is the right level for commodity –Commodity within a node (individual components/processors) –Commodity in software layers (e.g., Linux) –Commodity in infrastructure (e.g., cloud technologies)

9. Breakout Charge Questions, continued 1.What technical breakthroughs in science and engineering research can be enabled by exascale platforms and are attractive targets for Japan-US collaboration over the next 10 years? Exascale Challenges –Support for many core architectures –Communication layers e.g., PGAS –Application composition (beyond communication) –Memory / storage hierarchy (including staging and file systems) –Workflow management –Lightweight / micro kernels –Energy efficient scheduling –Performance measurement, modeling and prediction –Resilience: Fault models and minimizing duplication in detection and response

10. Breakout Charge Questions, continued 2.What is the representative suite of applications systems in your research area, available today, which should form the basis of your co-design communication with computer architects application teams? a)How are these applications currently constrained by compute and data resources, programming models, or available software tools? b)What are the gaps in available applications and application workflows, and requirements to fill these gaps? c)Which of these are ripe for collaboration within the context of Japan-US cooperation? –I/O systems (and frameworks), e.g., ADIOS –Communication layers, e.g., PGAS, UCCS –Application composition, e.g., COMPOSE/Hobbes –Visualization and analysis services –Simulation, emulation and modeling tools

11. Breakout Charge Questions, continued 3.How can the application systems research community, represented by a topical breakout at this workshop, constructively engage the vendor community in co- design? a)How should these various aspects of the application and architecture be optimized for effective utilization of exascale compute and data resources? b)Consider all aspects of exascale application: formulation and basic algorithms, programming models & environments, data analysis and management, hardware characteristics. –Full engagement of vendors and application teams, –Connected to procurement process CAAR at ORNL (one way, applications adapt to architecture) Co-design for post-K (two way, apps influence architecture and vice versa)

12. Breakout Charge Questions, continued 4.How can you best manage the “conversations” with computer designers/architects application developers around co-design such that (1) they are practical for computer design, and (2) the results are correctly interpreted within both communities? a)What are the useful performance benchmarks from the perspective of your domain? b)Are mini-apps an appropriate and/or feasible approach to capture your needs for communication to the computer designers? c)Are there examples of important full applications that are an essential basis for communication with computer designers? d)Can these be simplified into skeleton apps or mini-apps to simplify and streamline the co-design conversation

13. Breakout Charge Questions, continued 5.Describe the most important programming models and environment in use today within your community and characterize these as sustainable or unsustainable. a)Do you have appropriate methods and models to expose application parallelism in a high-performance, portable manner? b)Are best practices in software engineering often or seldom applied? c)Going forward, what are the critically important programming languages? d)On which libraries and/or domain-specific languages (DSL) is your research community dependent? e)Are new libraries or DSL’s needed in your research domain? f)Are these aspects of your programming environment sustainable or are new models needed to ensure their availability into the future?

14. Breakout Charge Questions, continued 6.Does your community have mature workflow tools that are implemented within leadership computing environments to assist with program composition, execution, analysis, and archival of results? If no, what are your needs and is their opportunity for value added? a)For example, do you need support for real-time, interactive workflows to enable integration with real-time data flows?

15. Breakout Charge Questions, continued 7.What are the new programming models, environments and tools that need to be developed to achieve our science goals with sustainable application software?

16. Breakout Charge Questions, continued 8.Is there a history, a track record in your research community for co-design for HPC systems in the installed machines in the past, and is there any co-design study done for these systems to document the effectiveness of co- design?