1. Framework support for Accelerators Sami Kama

2. Introduction Current Status Future Accelerator use modes Symmetric resource Asymmetric resource 09/11/2015 TIM Meeting Berkeley 2

3. Offloading now Accelerators are exposed to framework through use of OffloadSvc and APE server Athena algorithms prepare necessary data and do an offload request to OffloadSvc OffloadSvc sends request to APE server. Server executes the request on data by selecting appropriate module, and returns the results to OffloadSvc OffloadSvc passes response to algorithm, which convert results back to Athena EDM 09/11/2015 TIM Meeting Berkeley 3

4. Example HLT Demonstrator 1.HLT Algorithm asks for offload to TrigDetAccelSvc 2.TrigDetAccelSvc converts C++ classes for raw and reconstructed quantities from the Athena Event Data Model(EDM) to GPU optimized EDM through Data Export Tools 3.Then it adds metadata and requests offload through OffloadSvc 4.OffloadSvc manages multiple requests and communication with APE server 5.Results are converted back to Athena EDM by TrigDetAccelSvc and handed to requesting algorithm TrigDetAccelSvc Export Tool Athena EDM GPU EDM OffloadSvc Data+MetaData HLT Algoritm APE Server Request Result Export tools and server communication need to be fast (serial sections) 09/11/2015 TIM Meeting Berkeley 4 1 2 3 4 5 Implemented for each detector such as TrigInDetAccelSvc Module

5. Current status Multiple Athena’s and AthenaMP are supported Some glitches after forking, possible race condition in Yampl OffloadSvc can dump offload requests and responses for re-players. Needed for quick development and validation of accelerator algorithm Works fine for multi-process frameworks! 09/11/2015 TIM Meeting Berkeley 5

6. Offloading in future Client-Server approach is still valid if Accelerator is remote Framework can not include or work with Libraries Languages Compilers Programming paradigms needed by accelerator 09/11/2015 TIM Meeting Berkeley 6

7. Future Accelerator use modes Offloading should be integrated in scheduler. Could be a scheduler extension or plugin to modify graphs. Two different approaches depending on accelerator Accelerator is a symmetric resource Accelerator is an asymmetric resource 09/11/2015 TIM Meeting Berkeley 7

8. CPU Symmetric Resource Accelerator algorithms generate same output containers so next algorithm can execute in either CPU or accelerator. Easy to handle in current design May need differentiation of algorithms if different implementations are required (eg. CPU-GPU vs CPU-KNL) A B C D A B C D Accelerator 09/11/2015 TIM Meeting Berkeley 8

9. Asymmetric resource Accelerator algorithms execute several steps internally and create final containers There is no support for this mode currently Probably most efficient use of accelerator Scheduler has to have a separate graph segment for accelerator paths Event processing graph changes for depending on the availability of the resource May need data conversions steps CPU A B C D A-D Accelerator X Y time 09/11/2015 TIM Meeting Berkeley 9

10. Summary The choice of offloading model highly depends on availability of algorithms and accelerators Probably it will happen in an evolutionary way Plugin system can help Athena-APE will still work in new framework 09/11/2015 TIM Meeting Berkeley 10

11. BACKUP

12. Multi-Work resource management MODULE GPU 123 4 567 8 LB NLB M CONSTANTS Work1(LB-N) Work2 (LB-N) WorkX (LB-M) LB N ptr LB M ptr ptr 1 3 8 Scratch space for work to use Multi-event data such as MF and noise values Global constants such as geometry and hash tables addwork() Allocated by module at configure time depending on configuration create Works are processed and deleted returning resources to Module 12 09/11/2015 TIM Meeting Berkeley

13. WORK MutliGPU utilization Work contexts GPU1 areas GPU2 areas GPU3 areas GPU4 areas Module initializes multiple workspace on each GPU Queues them in cyclic manner Assigns front of the queue to new Works Workspace returns back to queue when job is deleted Front Back 13 09/11/2015 TIM Meeting Berkeley