A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Introduction APS Engineering Support Division –Beamline Controls and Data Acquisition Pete Jemian (Group Leader) Kenneth Evans, Jr. (Scientific Software Section Leader) Brian Tieman (Software Engineer) –Information Technology Ken Sidorowicz (Group Leader) Roger Sersted (Engineer) X_Ray Sciences Division Gabrielle Long (Division Director) –Chemisty, Environmental and Polymer Science Peter Chupas (Beamline Scientist) –Materials Characterization Ulrich Lienert (Beamline Scientist) Jon Tischler (ORNL Resident Scientist) –Time Resolved Research Michael Sprung (Beamline Scientist) Alec Sandy (Beamline Scientist) –X-Ray Microcopy and Research Francesco DeCarlo (Beamline Scientist) Wah Keat Lee (Beamline Scientist)

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Current Operational Workflow

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Preferred Operational Workflow

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Local HPC Resources--What Tomo –32 processor cluster –12TB disk –Sector 2 dedicated to tomography Blacklab –16 processor cluster –2TB disk –Development Orthros –58 processor cluster –30TB disk –On demand data reduction

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Local HPC Resources--Why On Demand Processing –Data reduction immediate part of workflow –May waste CPU cycles-bad for larger clusters Alignment (10s of minutes) Between Samples (<5 minutes) Between Shifts (??) High Throughput –Dozens of samples per day Automated sample changers Sometimes Unattended –Multiple beamlines Semi-Long Term Storage –3 to 6 months Low Processor Demand per Application Low Latency—True Real-Time Processing

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Remote HPC Resources--Why Modeling applications are generally more demanding –Trickier algorithms—more advanced math –Scale to many more processors –Need to be run many times More need for remote collaboration –Interpret results –Compare with theory/other results Beyond APS Resources –It’s all about efficient use of money Manpower Hardware Space Etc…

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. A Survey of Use Cases Tomography –TomoMPI –Paraview/MCS solution –DEJ_TextureAnalysis –Local Tomography –Laminography 3D X-Ray Diffraction Microscopy –xdmmpi –Near Field Peak Finder –ImageD11/Fable –Grainspotter/Fable –Box Scans X-Ray Photon Correlation Spectroscopy –xpcsmpi X-Ray Micro-Diffraction –Reconstruct –Euler –Rindex

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Common Features of Our Use Cases Need for HPC resources –On demand clusters –Large scale clusters Most have need for large data volumes –Archival –Transport Most still need algorithm development –Parallelism –Optimization –Robustness/Portability Many used by many unrelated scientific disciplines –Open access –Intuitive interfaces –Tailored interfaces

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Where MCS Can Help… HPC resources –Help target appropriate systems How to find them How to develop for them How to generate proposals for them –Help understand management of HPC resources How to use HPC for On Demand computation How to tuning for performance What to buy

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Where MCS Can Help… Dealing with the Data –Archival solutions Central repository Nearline/Offline storage –Fast/Reliable data transfer HPC resources End users –Ethernet –Sneakernet

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Where MCS Can Help… Algorithm Development –Help enable scientists to parallelize code More training sessions Assistance with initial parallelization –Help with code optimization Maybe codes exist Maybe new routines need development –Robustness/Portability Libraries we should be using Languages we should be using Operating Systems we should target

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Ways MCS Can Help… User access –Service Oriented Architecture Hide complexity Intuitive interfaces Remote access Collaboratory Experience –Help users set up HPC software on their systems

A U.S. Department of Energy laboratory managed by UChicago Argonne, LLC. Ways APS Can Help… Look for Joint MCS/APS LDRDs Explore possibility of APS providing operational funding for MCS –New Hardware –R&D Effort Station APS FTE in MCS –Information exchange –Provide MCS effort on projects of direct benefit to APS Conduit to end users –Collect new use cases –Explore potential new funding opportunities –Scheduling Meetings Training