Database Allocation Strategies for Parallel BLAST Evaluation on Clusters Louis Woods

24/05/2008Database Allocation Strategies for Parallel BLAST2 Outline What is BLAST? How does BLAST work? Some details about the BLAST algorithm PART I PART II Parallel BLAST: Motivation Parallel strategies Implementation issues Q & A

24/05/2008Database Allocation Strategies for Parallel BLAST3 Sequence Comparison DNA and protein sequences can be represented as finite strings over a restricted alphabet, e.g. {A, T, C, G} Given a query sequence molecular biologists are interested in finding similar sequences (evolutionarily related) in a sequence database

24/05/2008Database Allocation Strategies for Parallel BLAST4 BLAST – Basic Local Alignment Search Tool BLAST was developed in 1990 by The National Center for Biotechnology Information (NCBI) Due to its speed BLAST has become the most popular tool for doing sequence comparison BLAST uses a heuristic approach which makes BLAST a lot faster than other algorithms Trade-off: Results are good (in terms of accuracy), but not optimal

24/05/2008Database Allocation Strategies for Parallel BLAST5 Sequence Database >NT_ dna:supercontig supercontig::NT_113887:1:3994:1 TTGTTTGTCCCACGACAGAGCTGGGCTGAATTATTAATGTGG ATTTTGTCCAACAATGGACTGAAAAGGGAGAAGCCCATGAAC TCTGTGAGGAGTGCATGACAGGTGCTCGTGAGATGA... >NT_ dna:supercontig supercontig::NT_113947:1:4262:1 CATACATTTAATATACCCTCACCATACAGAATGTTCTTTCCC TATTACATAAGGAGTATATGTATTAAGCACTAAATCTTTGGA ATAATAAAAGACTATATTCATATTTGGTAACTTATT... >NT_ dna:supercontig supercontig::NT_113903:1:12854:1 ACCAGTTCTCACAGGAACTAATAAGAGTGAGAGCTCACTCAC CACTGTGAGGAAGGCACCAAGATGTCCATGAGTGACCTGTCC CTAAGACCCAAAGACCTCCCATTAGGCCCCACCTCC... >NT_ dna:supercontig supercontig::NT_113908:1:13036:1 CGGCCCTGCTGAGGCCGGGCATGGAGCTGGGGGTCAGGCCCT TCAGTCTCTTGAGGGTGTTCAGGACCACGTTCAGGTACCTGT TCTTGTTGGGGCTGCAGTCGTAGGCCACCTTCTCCT... >NT_ dna:supercontig supercontig::NT_113940:1:19187:1 CTTTGCATTCAACGCACAGTGTTGAACCTTTCTTTGATAGTT CAGATTTGAAACACTCTTTTTGTAGAAACTGCAAGTGGATAA CTGCACTTCTTTGAGGCCTATCGTAGTAAAGGAAAT... >NT_ dna:supercontig supercontig::NT_113917:1:19840:1 CCCCCAACTCTCGCCGGCTGTCCCTAGGAGTCCACGGGCCGT CCGGGGCCCCCCCCAGGCCTGGCGGGACCAGGATGCTGCCCT GTCACCTGCCCCCCAGCCCCACACAACGCCCCCACC... >NT_ dna:supercontig supercontig::NT_113963:1:24360:1 TACCTTTCCTCGTTGTGCGTCCGGTCTTCTGAGCCATGTGCT AATATCCTGGGACTCTGTCTGGTTTTGTTTTTTTTTTTAACA TTGCCTAAATCATATTTTCCATTTAAAGAAATTTCTAA... Extract form the sequences database: Homo_sapiens.NCBI36.48.dna.nonchromosomal.fa A sequence database is a textfile An simplified analogy to BLAST from the SQL- world: SELECT* FROM sequence_table WHEREsequence LIKE '%TGTTG%' ORDER BY$score; TTGTTTGTCCCACGACAGAGCTGGGCTGAATTATTAATGTGG ATTTTGTCCAACAATGGACTGAAAAGGGAGAAGCCCATGAAC TCTGTGAGGAGTGCATGACAGGTGCTCGTGAGATGA... CATACATTTAATATACCCTCACCATACAGAATGTTCTTTCCC TATTACATAAGGAGTATATGTATTAAGCACTAAATCTTTGGA ATAATAAAAGACTATATTCATATTTGGTAACTTATT... ACCAGTTCTCACAGGAACTAATAAGAGTGAGAGCTCACTCAC CACTGTGAGGAAGGCACCAAGATGTCCATGAGTGACCTGTCC CTAAGACCCAAAGACCTCCCATTAGGCCCCACCTCC... CGGCCCTGCTGAGGCCGGGCATGGAGCTGGGGGTCAGGCCCT TCAGTCTCTTGAGGGTGTTCAGGACCACGTTCAGGTACCTGT TCTTGTTGGGGCTGCAGTCGTAGGCCACCTTCTCCT... CTTTGCATTCAACGCACAGTGTTGAACCTTTCTTTGATAGTT CAGATTTGAAACACTCTTTTTGTAGAAACTGCAAGTGGATAA CTGCACTTCTTTGAGGCCTATCGTAGTAAAGGAAAT... CCCCCAACTCTCGCCGGCTGTCCCTAGGAGTCCACGGGCCGT CCGGGGCCCCCCCCAGGCCTGGCGGGACCAGGATGCTGCCCT GTCACCTGCCCCCCAGCCCCACACAACGCCCCCACC... TACCTTTCCTCGTTGTGCGTCCGGTCTTCTGAGCCATGTGCT AATATCCTGGGACTCTGTCTGGTTTTGTTTTTTTTTTTAACA TTGCCTAAATCATATTTTCCATTTAAAGAAATTTCTAA...

24/05/2008Database Allocation Strategies for Parallel BLAST6...TATATAGGTAT......CCGAAAGTCTT... BLAST Search...TACCTAGTCTA......GTTGTAGTCTA... Score = TTAATTTTCAC......AAATGTCCTAA......TCAACCCCGAC... Query Set...GGCCTAATTTT......AACGTAGTCTT... Score = 3 Score = 6 Sequence Database Threshold S = 4

24/05/2008Database Allocation Strategies for Parallel BLAST7 Scoring Matrices Scoring matrix: 4x4 matrix (DNA) or 20x20 matrix (protein) Assume these matrices as given. Not further discussed in this talk Score of an alignment: Sum of pairwise scores

24/05/2008Database Allocation Strategies for Parallel BLAST8 BLAST Algorithm - Phase 1 Compile a list of high scoring words: break query sequence into words of length w and add all w-length words that score above some threshold T. Example: query sequence = ACTTAG, w = 3, T = 12 ACTTAG ACT CTT TTA TAG,ACC,TTT A|A| C|C| T|T| Score = = 13 A CC

24/05/2008Database Allocation Strategies for Parallel BLAST9 BLAST Algorithm - Phase 2 The sequence database is searched for sequences with exact matches to words from the list from phase 1. List form phase 1: {ACT, ACC, CTT, TTA, TTT, TAG} Example: sequence match = AGTTAC

24/05/2008Database Allocation Strategies for Parallel BLAST10 BLAST Algorithm - Phase 3 Extend hits: for every pair (word, match) extend the pair in both directions until a score >= S (other threshold) is reached. All sequences with score >= S will be listed in the final results. Example:query sequence= ACTTAG database sequence= AGTTAC TTA ||| TTA C|GC|G G|CG|C A|AA|A Score = = 12

24/05/2008Database Allocation Strategies for Parallel BLAST11 Why Parallelize BLAST? Moore's Law outpaced by growth of sequence databases. BLAST alone is not fast enough anymore. Souce:

24/05/2008Database Allocation Strategies for Parallel BLAST12 Parallel Environment Hardware: Distributed memory architecture (workstation cluster) Parallel Interface: Message Passing Interface (MPI) Set-up: Master-Slave

24/05/2008Database Allocation Strategies for Parallel BLAST13 Parallel Strategies Advantages of clusters: (1) a lot of memory available (2) parallelize data, not the BLAST algorithm Focus: How can the data be distributed? Partition query set: replicated approach. Partition sequence database: fragmented approach.

24/05/2008Database Allocation Strategies for Parallel BLAST14 Replicated Approach Souce: [1] Database allocation strategies for parallel BLAST evaluation on clusters

24/05/2008Database Allocation Strategies for Parallel BLAST15 ✔ Very simple and easy to implement Result merging: concatenate results from slaves Low communication overhead: For a query set of m sequences, m messages have to be sent to the slaves Linear speed-up: (1h 7 min -> 2 min 20 sec with 30 slaves) Replicated Approach: Analysis ✔ Doesn't solve the database size problem (e.g when database doesn't fit into main memory) ✔ Load balancing depends on composition of the query set

24/05/2008Database Allocation Strategies for Parallel BLAST16 Fragmented Approach Souce: [1] Database allocation strategies for parallel BLAST evaluation on clusters

24/05/2008Database Allocation Strategies for Parallel BLAST17 ✔ Attacks problem of exponential sequence database growth ✔ Superlinear speed-up with large databases. (Whole database in main memory) ✔ Load balancing independent of query set Fragmented Approach: Analysis ✔ Partitioning strategy required ✔ Result merging phase more complicated ✔ Higher communication overhead (query set of m sequences sent to n slaves)

24/05/2008Database Allocation Strategies for Parallel BLAST18 Static Partitioning Prior to executing BLAST a program computes the fragments of the sequence database The slaves then copy their fragment from the shared storage to their local storage Maste r Shared Storage Sequence database Slave 1 Slave 2 Slave 3 Slave 4

24/05/2008Database Allocation Strategies for Parallel BLAST19 Disadvantages of Static Partitioning A large number of smaller files are created, which are harder to manage, migrate, and share The database needs to be re- partitioned if the number of slaves in a parallel BLAST search exceeds the number of existing fragments Creating a large number of fragments for running on different number of slaves is not a good option (1)I/O overhead (2)Result merging- and retrieving overhead

24/05/2008Database Allocation Strategies for Parallel BLAST20 Dynamic Partitioning with Parallel I/O Instead of physically computing the fragment files only the offsets are calculated Offsets are distributed over the network to the slaves Slaves read sequences directly form shared storage via parallel I/O into their main memory Maste r Shared Storage Sequence database Slave 1 Slave 2 Slave 3 Slave 4 Communicates file offsets Read sequences [0,29 ] [60,89 ] [30,59 ] [90,117 ] [0,29] [30,59] [60,89] [90,117]

24/05/2008Database Allocation Strategies for Parallel BLAST21 Load Balancing – Adaptive Distribution Dynamic adaptive approach: i nitially not the whole database is distributed (e.g. only 70%) When the fist slave has processed all queries on its fragment and reports back to the master, the master assigns a new fragment to that slave The master gradually decreases the size of the fragments until the query has been processed on the entire database

24/05/2008Database Allocation Strategies for Parallel BLAST22 Result Merging Phase: Problem Problem: the master has to regroup and filter result sequences from slaves Master waits for all of the results and then generates the global output file itself Slaves are idle and don't make use of their I/O capacity in the result merging phase

24/05/2008Database Allocation Strategies for Parallel BLAST23 Result Merging Phase : Solution Slaves cache results and send only relevant information for filtering to the master Master does filtering and informs slaves which results to write at which offset to the global output file Slaves use parallel I/O and write global output file simultaneously No I/O bottleneck at the master Communication overhead by the slaves reduced

24/05/2008Database Allocation Strategies for Parallel BLAST24 Bottleneck Master Mast er SSSSSSSSSSSS

24/05/2008Database Allocation Strategies for Parallel BLAST25 Introducing a Supermaster Supermas ter SSSSSSSSSSSS Mast er

24/05/2008Database Allocation Strategies for Parallel BLAST26 Blue Gene/L The world's top supercomputer Parallel BLAST was implemented on a cluster with 4096 nodes To utilize as many nodes as possible the fragmented approach was combined with the replicated approach

24/05/2008Database Allocation Strategies for Parallel BLAST27 Replicated & Fragmented Approach Supermas ter SSSSSSSSSSSS Mast er Fragmented Approach Replicated Approach

24/05/2008Database Allocation Strategies for Parallel BLAST28 Summary BLAST: popular tool for sequence comparison (good performance) Problem: exponential growth of sequence databases Solution: parallelization, distribute data Replicated approach Fragmented approach Implementation issues: partitioning, load balancing, I/O bottleneck, message overhead MPI & BLAST = a good solution

24/05/2008Database Allocation Strategies for Parallel BLAST29 References 1. R. de Carvalho Costa and S. Lifschitz. “Database allocation strategies for parallel BLAST evaluation on clusters”, Distributed and Parallel Databases, 13(1), H. Lin, X. Ma, P. Chandramohan, A. Geist, and N. Samatova, “Efficient Data Access for Parallel BLAST,” Proc. 19th Int'l Parallel and Distributed Processing Symp. (IPDPS '05), The European Bioinformatics Institute (EBI), 4. Karl Jiang, Oystein Thorsen, Amanda Peters, Brian Smith and Carlos P. Sosa, “An Efficient Parallel Implementation of the Hidden Markov Methods for Genomic Sequence Search on a Massively Parallel System”, A. E. Darling, L. Carey, and W. Feng, “The design, implementation, and evaluation of mpiblast,” In proceedings of CluterWorld and Expo, Huzefa Rangwala, Eric Lantz, Roy Musselman, Kurt Pinnow, Brian Smith, Brian Wallenfelt. “Massively Parallel BLAST for the Blue Gene/L” Presented at the High Availability and Performance Computing Workshop held in conjunction with the 6th Los Alamos Computer Science Institute Symposium, Santa Fe, New Mexico (October, 2005).

24/05/2008Database Allocation Strategies for Parallel BLAST30 Questions