1. Instrumenting Genomic Sequence Analysis Pipeline Mothur on Shared Memory Architecture Junqi Yin, Bhanu Rekepalli, Pragneshkumar Patel, Chanda Drennen, and Annette Engel XSEDE 14, Atlanta GA, July 15, 2014

2. Outline  Introduction – Motivation --- ECSS – Bioinformatics tool --- Mothur – SGI UV1000 --- Nautilus  Porting OTU analysis pipeline – Pre-clustering denoise – Distance matrix calculation – Sequence clustering  Performance results on Nautilus  Summary 2Porting Mothur to Nautilus

3. ECSS community code project The effect of the Macondo oil spill on coastal ecosystems  The ultimate goal is to improve society’s ability to understand how to respond to and mitigate the effects of petroleum pollution and related stressors on marine and coastal ecosystems of the Gulf of Mexico.  The challenge is analyzing rapidly growing pyrosequencing data (millions of sequences), which are beyond the capability of a typical workstation.  The solution is to develop a downstream analysis pipeline capable for HPC. 3Porting Mothur to Nautilus

4. Mothur  Schloss, P.D., et al., Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Appl Environ Microbiol, 2009. 75(23):7537-41 Cited by 2453  Mothur is an expandable C++ code that re-implements a large number of popular algorithms within the community into a single, standalone executable for different platforms.  However, it is not HPC ready. 4Porting Mothur to Nautilus

5. Mothur  One important goal is to categorize sequences  3 ways to bin sequences in Mothur – Operational Taxonomic Units (OTUs): sensitive to errors, but is independent of any previous knowledge. – Taxonomic: bins sequences based on what they’re named – Phylogenetic: builds trees and uses the branching structure to bin sequences  For more information – Wiki: http://www.mothur.org/wiki/Main_Pagehttp://www.mothur.org/wiki/Main_Page – User forum: http://www.mothur.org/forumhttp://www.mothur.org/forum 5Porting Mothur to Nautilus

6. Nautilus Single system image:  1024 cores  Intel Nehalem EX processors  4TB of global shared memory  8 NVIDIA Tesla GPUs  NUMA 6Porting Mothur to Nautilus

7. Nautilus  A single node system with large global shared memory  Offloading thread synchronization, data sharing, and massage passing overhead from CPUs  Scalable interconnect with other blades via NUMAlink5  For more information, see http://www.nics.utk.edu/computing-resources/nautilus http://www.nics.utk.edu/computing-resources/nautilus 7

8. d d NICS and Nautilus: Darter 11,968 physical cores 8

9. OTU Analysis Pipeline  Clustering 16S rRNA sequences into operational taxonomic units (OTUs) is a critical step for the bioinformatic analysis of microbial diversity – Pre-clustering denoise – Distance matrix calculation – Sequence clustering 9Porting Mothur to Nautilus

10. Pre-clustering denoise (pre.cluster)  Remove sequences due to errors: if 2 sequences that are each 1 bp different from a big group, this assumes that it’s due to sequencing error.  Time complexity O(N 2 ); two loops are not independent, and the OpenMP directive is applied to the inner loop. 10

11. Distance matrix calculation (dist.seqs)  Calculate pairwise distance between sequences ( O(N 2 ) )  Using MPI in Mothur with embarrassingly parallel scheme  A shared MPI-IO pointer is employed and every MPI process writes to a single file in a line-by-line fashion, which cause writing contentions.  Solution: file per process; scale up to the number of Object Storage Targets (OSTs) of the parallel file systems 11

12. Sequence clustering (cluster)  Unweighted Pair Group Method with Arithmetic mean (UPGMA) – Search the distance matrix and find the minimum cell ( O(N 2 ) ) – Treat the found cell as a node and update its distance to other cells ( O(N) ) – Repeat first two steps N times or until the found minimum distance is larger than a predefined cutoff value  Time complexity O(N 3 ) ; memory complexity O(N 2 ); sequential implementation in Mothur 12

13. Sequence clustering (cluster)  To use more than one socket, each thread should work on part of matrix allocated on local memory  Distance matrix is represented by STL vector  “first touch” policy is enforced for NUMA  Solution: customizing memory allocation by overwriting the allocator in std::vector > numa_seqVec 13

14. Sequence clustering (cluster)  Most important methods in custom allocator class in allocate()  Object with dynamic data are problematic, e.g. can’t use vector::erase method 14

15. Sequence clustering (cluster)  The hot spot (over 90%) in cluster is SparseDistanceMatrix:: getSmallestCell method  Same static scheduling  Set OMP_PROC_BIND or use dplace 15

16. Performance results on Nautilus MethodSeqscoresTime(s) 11185 Pre.cluster500002668 4392 8243 131.8 Read.dist5000218.8 411.3 87.7 16 Scaling of distance matrix calculation 10000 sequences on Nautilus

17. Performance results on Nautilus 17 Ratio of run time for 16 cores with respect to up to 160 cores for 5000, 10000, and 2000 sequences Run time and speedups for 5000 Sequences on up to 128 cores

18. 18

19. Summary  Pre-clustering and matrix loading have seen over 4x speedup on Nautilus  Distance calculation shows linear scaling up to the number of the OSTs  Sequence clustering shows 7x speedup when number of cores increased by 10x  Overall, OTU pipeline being accelerated by orders of magnitude on Nautilus, and the optimization is generally applicable for other shared memory machines. 19HPC in Physics