1. Future Trends in Nuclear Physics Computing
David Skinner
Strategic Partnerships Lead
National Energy Research Scientific Computing
Lawrence Berkeley National Lab

2. Overview and Background
HPC is extending further into experimental science.
It is however newly needed by 1) more teams 2) smaller teams 3) for “bursty” workloads
Computing center is a great place to observe and inform computing strategies adopted by science teams
PTF Super-nova
Planck CMB
Alice LHC
Atlas LHC
Carbon Flux*
Sensors
Daya Bay
JGI & KBASE
...

3. Trends in Experimental Facility Computing
NERSC Presentation
4/11/15
Trends in Experimental Facility Computing
Terabit detectors
Programmable Networks
In situ analysis
Detectors capable of generating terabit data streams.
Integration of experimental and computational facilities in real time, using programmable networks.
Computational tools for analysis, data reduction, feature extraction in situ, using advanced algorithms and specialized hardware.
Data platforms
High performance, configurable analysis and storage platforms
Robotics and automation
Increased scientific throughput from robotics and automation software.
Sharing and management
Trends in the worlds of light sources that are motivating
Offline analysis
Data management and sharing, with federated identity management and flexible access control.
Post-processing: reconstruction, inter-comparison, simulation, visualization.

4. Relevant Future Computing Concerns
NERSC Presentation
5/6/2018
Relevant Future Computing Concerns
Probably the worst thing you could do would be technology selection
Moore’s Law is ending. What’s next is….lower power.
Find the key algorithms which involve data motion in the end-to-end workflow.
How do these scale?
Would you like better algorithms?
Track / Participate in the ASCR Requirements Reviews
And other forward looking computing venues
Thank you for the invite. Its great to see a science community having its computing discussion before an emergency arises. When the EIC is in full bloom i think you'll be glad this meeting happened. Rather than react to a data problem tomorrow, you can today plan to leverage data to make EIC operationally great and imagine the data artifact you'd like EIC to generate. The worst thing you could do at this turn is to make technology selections.

5. Free Advice
Presuppose little about computing other than it having a power budget.
Focus instead on the metrics for success of the overall instrument, some of which will probably involve a computer.
In 2040 computers and networks will be different. Focus on the power demands of the end-to-end workflow.
Focus on algorithms that minimize data movement.
Pay attention to the plumbing.

6. Near Future HPC techs In Package Memory (DRAM on socket) NVRAM
Cheap persistent memories
Containers
Docker / shifter for software sanity
SDN
Self Driving Networks

7. Three Examples PTF, JGI, and LCLS

8. Palomar Transient Factory Retrospective
PTF Camera 92 Mpixels, 1” resolution R=21 in 60s – take 300 exp./night
PTF Science : Key Projects
Various SNe
Dwarf novae
Transients in nearby galaxies
Core collapse SNe
RR Lyrae
Solar system objects
CVs
AGN
AM CVn
Blazars
Galactic dynamics
LIGO & Neutrino transients
Flare stars
Hostless transients
Nearby star kinematics
Orphan GRB afterglows
Rotation in clusters
Eclipsing stars and planets
Tidal events
H-alpha sky-survey
The power of PTF resides in its
diverse science goals &follow-up.
Faster is better.
Scan the sky every night, rapidly ingest and analyze.
Fast turnaround enables wide ranging research agendas

9. PTF Pipeline: Faster/Integrated Workflows
128 MB/90s
50 GB/night

10. PTF: Earliest-ever Detection of Supernova Fast Workflows  More Science
Observed within hours of its explosion, 11KLY will allow a rare glimpse at the supernova’s outer layers that contain hints about what kind of star exploded
The last time a supernova of this type occurred so close was Before that: , 1898 and 1572.
“The supernovae of a generation”
- Josh Bloom UCB
23 August
24 August
25 August
NERSC resources used:
NERSC data transfer nodes accept 300GB/night; HPSS archives;
Data ( > 150 TB and growing ) reside on NERSC Global Filesystem;
NERSC HPC runs subtraction pipeline to process images and detect new transients;
Science gateway nodes serve the DeepSky database to astronomers worldwide.
NERSC Repo m937, Peter Nugent PI, Palomar Transient Factory
See
The repo has run ~4,300 jobs on Carver this AY, mostly 1-2 nodes, some 8, 16, 32-nodes
“Before” and “After” images showing the 11KLY discovery.
Peter Nugent project PI and Realtime Transient Detection Lead

11. JJoint Genome Institute’s HPC at NERSC
NERSC Presentation
JJoint Genome Institute’s HPC at NERSC
8/25/15
ssh genepool.nersc.gov
login nodes
web services
workflow nodes
database services
compute nodes
filesystems
high priority
& interactive nodes
Katie
fpga
Computing Sciences Area

12. X-ray Superfacility Motivations: Great Science
NERSC Presentation
5/6/2018
X-ray Superfacility Motivations: Great Science
“The model of users applying for one-time access to single-user facilities does not promote the coordinated, interdisciplinary approach needed to solve today’s grand challenge problems. Next-generation light sources and other user facilities must learn to accommodate the interdisciplinary, cross-platform needs of modern grand challenge science”
This close coupling of synthesis and light source characterization requires establishing new modes of facility operation that go well beyond the traditional application-approval model with its months-long turnaround times.
How? In that context, the strong integration of theory and experiment, like that already established in high energy physics, is a worthy goal to strive for at future BES photon sources.

13. Emerging XFEL Workflow Requires Superfacility
stream
XTC format
HPSS
Global Scratch
Lustre
Global Scratch
/Project (NGF)
HPSS
DAQ
multilevel data acquisition and control system
Science DMZ
Compute Engine
Cray XC30
Cornell–SLAC
Pixel Array
psana
Workflow Needs
Prompt Analysis
Offline Analysis
Community
Data Access …
100GB/s
hitfinder
spotfinder
index
integrate
hitfinder
spotfinder
index
integrate
hitfinder
spotfinder
index
integrate
100GB/s
Diffraction Detector
Injector
Science Gateway Nodes

14. Is this Success Generalizable?
Scientific workflows are diverse and complex.
Scientific workflows belong to scientists.
No one-size-fits-all solution from HPC facility
Facilities can provide workflow components
A network (thank you Esnet!)
Single Sign-On to computing and instrument
Schedulable bandwidth, storage, and computing
Automation in data analysis (fire and forget)
Integrate and scale the composition of these building blocks
Much work on building blocks is underway, more needed

15. Super Facility: Emerging Areas of Focus
Automated Data Logistics
Your data is where you need it, automated by policies
Federated Identity
Single sign on is a big win for researchers. Globally Fed ID we may leave for others.
Real-time HPC
“real-time” means different things, but given enough insight into your workflow we intend make the computing happen “when you need” Bursty workloads are a great place to exercise this new capability.
APIs for Facilities
Interoperation through REST not . Please.

16. Superfacility Concepts for Lightsources
NERSC Presentation
5/6/2018
Superfacility Concepts for Lightsources
Photon science has growing data challenges. An HPC data analysis capability has potential to:
Accelerate existing x-ray beamline science by transparently connecting HPC capabilities to the beamline workflow.
Broaden the impact of x-ray beamline data by exporting data for offline analysis and community re-use on a shared platform
Expand the role of simulation in experimental photon science by leveraging leading-edge applied mathematics and HPC architectures to combine experiment and simulation
Let’s examine one recent example in applying these concepts

17. Superfacilities Deliver for Workflows
150TB “image” of photosystem II reconstructed with 130K cpu hours
“Thank you to all who participated from the Photosystem II team. Having NERSC facilities at our disposal really helped to give us an early picture of the X-ray results. I look forward to applying this experience in the future.” –Nick Sauter of the Photosystem II team

18. Thank You