1. EmPOWERing Software Porting Code to Run on our Power AI Cluster
Monday 8th July 2019 Simon Branford

2. POWER9 AI Cluster

3. POWER9 AI Cluster Eleven nodes Four NVIDIA V100s (16GB)
Dual IBM POWER9 CPUs
18 cores each
Running with SMT4
1TB system memory
NVIDIA NVLink
SMT = Simultaneous multithreading

4. Availability at UoB Is a part of BlueBEAR Batch HPC
Part central funded
Available to any researcher
Part CaStLeS funded
Available to life science researchers
Researchers from across the University
Compute and Storage for Life Sciences

5. Software on BlueBEAR Software installs are by request
Installed using EasyBuild
Not just AI software
GPU accelerated software

6. Software PyTorch: 1.1.0, 1.0.1 TensorFlow: 1.13.1 Theano: 1.0.4
GROMACS: (with PLUMED),
Amber: 18
Keras: 2.2.4

7. Some Software ... ... just installs and works TensorFlow Keras GROMACS
Test CPU/GPU work balance

8. Mesa / LAMMPS / Qt Get rid of the intel only options… Mesa: swr
LAMMPS: intel optimisations
Qt: QtWebEngine
Related to using Easybuild
most working on Intel

9. PyTorch Older versions needed tweaks to get installed
Newer versions seem fine

10. Amber / arpack-ng / Pillow
“ARPACK is a collection of Fortran77 subroutines designed to solve large scale eigenvalue problems”
“Amber is software for performing molecular dynamics and structure prediction.”
“Pillow is the 'friendly PIL fork' by Alex Clark and Contributors. PIL is the Python Imaging Library by Fredrik Lundh and Contributors.”

11. Amber / arpack-ng / Pillow
Needed to be told:
That this is a POWER9
Options to use on POWER9
Compile flags
Directories
Expect that these will 'just work' in the future

12. NumPy and SciPy Test failing Caused by bug in OpenBLAS
Use OpenBLAS for POWER9 support
(Don’t use unpatched OpenBLAS ever!)
Ed will give more details later!

13. Chiron
“Using a deep learning CNN+RNN+CTC structure to establish end-to-end basecalling for the nanopore sequencer.”
Uses TensorFlow
But has a dependency on mappy
“Minimap2 is a versatile sequence alignment program that aligns DNA or mRNA sequences against a large reference database.”

14. minimap2 “Minimap2 is optimized for x86-64 CPUs.”
“Minimap2 also works with ARM CPUs supporting the NEON instruction sets.”
“Minimap2 requires SSE2 instructions on x86 CPUs or NEON on ARM CPUs. It is possible to add non-SIMD support, but it would make minimap2 slower by several times.”

15. POWER Vector Library
Vector instructions ... accessed via ... intrinsic functions
For porting applications from other platforms
Supports POWER7, POWER8 and POWER9
Provides the header files
Tools to detect functions that need converting
POWER7, POWER8 and POWER9 processors support a wide range of vector instructions that can be accessed via built-in functions (also called intrinsic functions). For customers porting applications from other platforms, where the original source code included calls to built-in functions for those other platforms, the POWER vector built-ins can be challenging to map to typical vector functions such as those found in Open Source Software projects.
Fully supported POWER processors are POWER7, POWER8 and POWER9 (since VSX instructions were first introduced in POWER7). POWER6 processors are partially supported.

16. veclib https://github.com/IvantheDugtrio/veclib
Vector library for porting SSE2 instructions to other architectures
Though it only does POWER
Allows direct use of SSE2 with minimal work

17. Combined Compile veclib into a library
Replace <emmintrin.h> with <vec128intlib.h>
Add -maltivec
Removed hard coded options
And minimap2 works on POWER9

18. IBM Guide
Porting x86 vector intrinsics code to Linux on Power in a hurry
What I just described is approach 3

19. Other Options Approach 4: “SIMD Everywhere”
Approach 2: Power Vector Library
Approach 1: intrinsics compatibility implementation in GCC
Rewrite the code!
Approach 4 - Lofty idea - API above all intrinsic libraries - not sure it supports POWER
Approach 2 - is do the conversion with the first part of what I've just described
Approach 1 - GCC 8 - I've not tried this yet
Likely to be best - but will be most time consuming.

20. POWER Vector Library
Not to be confused with the POWER Vector Library previously mentioned!
"Header files that contain useful functions leveraging the PowerISA Vector Facilities: Vector Multimedia Extension (VMX AKA Altivec) and Vector Scalar Extention (VSX)."
A library of useful vector functions for POWER. This library fills in the gap between the instructions defined in the POWER Instruction Set Architecture (PowerISA) and higher level library APIs. The intent is to improve the productivity of application developers who need to optimize their applications or dependent libraries for POWER.

21. Other Interesting Links
The IBM Software Development Kit for Linux on Power (SDK)
Find open source packages built for IBM Power
Porting x86 Linux applications to IBM POWER
The Migration Advisor
The Migration Advisor helps on moving Linux applications to Power Systems servers. Its locates potential migration problems within a C/C++ project, such as source code that might produce different results when run on Power Systems servers.

22. Future
Whatever our users ask for?!

23. Thanks! ?