Parallelizing the graph isomorphism portion of an automatic reaction mechanism generation algorithm Geoff Oxberry Project, Spring 2009

Automatic reaction mechanism generation yields models quickly  Reaction mechanisms are used to model chemistry in a wide range of applications  Generating first principles reaction mechanisms can take years, requires lots of expertise  The Bill Green group developed software (RMG) that automatically generates these models based on rules

For some problems, RMG takes days to generate a mechanism  We want it to take a day or less on a cluster  Big bottleneck for us is that we have to repeatedly solve a colored graph isomorphism (GI) problem  If we can speed it up, we can solve many more interesting chemistry problems  Parallelism is one option

We want to see if parallelism can be used to speed up RMG  Want to see if a parallel version of RMG is faster than a serial version  Due to time constraints, I chose to implement skeletal prototypes of serial and parallel versions of RMG in Python  Idea is to use results for the prototypes to see if it is worth parallelizing the production-scale code

Parallelism does speed up RMG on intermediate-sized case studies  When searching for graph isomorphisms in collections of 20 or fewer graphs, serial code is faster  When searching for graph isomorphisms in collections of ~100 graphs, parallel code is faster  When searching for graph isomorphisms in collections of ~2000 graphs, serial code is faster again

Outline  Brief overview of graph isomorphism  Discussion of existing RMG algorithm and how to parallelize  Python prototypes of serial and parallel versions of RMG algorithm  Results  Discussion of obstacles  Conclusions

Two graphs are isomorphic if there exists a bijection between their nodes  These two graphs are isomorphic:  Bijection here (L-R): 1-1, 3-4, 2-5, 5-2, 4-3

In RMG, ChemGraphs represent species  ChemGraphs are graphs with node labels and edge labels  Species are represented by a class of graphs equivalent under isomorphism  Example (methane): Node labels refer to atom types, edge labels refer to bond types

RMG classifies species as one of three types  Core species make up all of the reactants of the reaction mechanism  Edge species are products of the reaction mechanism not included in the core; they may be added to the core over the course of the algorithm  Postulated species are proposed species that may be added to the edge over the course of the algorithm

RMG algorithm manipulates graphs to generate a reaction mechanism Initialize set of core species Add remaining postulated species to edge species. Determine if any edge species should be added to core. Is termination criteria met? Use GI to discard postulated species based on various criteria Generate postulated species using some rules. No Yes

Checking for duplicate graphs using GI looks parallelizable  For example, could scatter postulated species over all processors and check for duplicates against core species in parallel  Could also do this with forbidden configs, etc. Use GI to discard postulated species based on various criteria Use GI to check for forbidden configurations. Use GI to check for duplicates among postulated species. Use GI to check that postulated species aren’t duplicated in core. Discard any duplicates.

Instead of working with RMG directly, I created a prototype  RMG takes 18 mos. for a developer to get up to speed; this project was ~6 wks.  To save time, I built a prototype in Python because its syntax and available libraries enable rapid development  Also enabled me to focus on the parts of the code that matter (GI algorithms) and ignore the rest

Serial prototype throws out everything but GI checking Initialize set of core species Add remaining postulated species to core species. Is termination criteria met? Use GI to discard postulated species based on various criteria Select postulated species from existing RMG output. No Yes

Parallel prototype parallelizes part of the GI comparisons  Checking postulated species against core species is embarrassingly parallel  Postulated species are essentially independent in that step Use GI to discard postulated species based on various criteria (in prototype) Use GI to check for duplicates among postulated species. Use GI in parallel to check that postulated species aren’t duplicated in core. Discard any duplicates.

Prototypes were implemented in Python/MPI on a cluster  Software: Python 2.5 (w/ C extensions) igraph module (graph data structure, GI algorithms) mpi4py module (MPI bindings for Python)  Hardware: 64-node cluster (pharos.mit.edu) 8 GB RAM per node Each node has 2 quad- core Xeon processors (either 2.33 GHz or 2.66 GHz)

Parallel prototype was faster on intermediate-sized problems  Species database was obtained from existing RMG output  Initial set of core species was 50% of database, randomly chosen  Program ran until all species in database were moved into core, or it reached 100 iterations Database Size (species) Serial time/ parallel time (a.u.)

Communication is slow in large test cases due to passing graph objects  Graphs are implemented using a class in the igraph library  mpi4py converts non-native Python objects using cPickle, which is compute-intensive  cPickle is probably why the serial code is faster in large test cases  Alternative approach would use NumPy and define an MPI derived data type; would be faster

Many technical problems occurred during the project  Laptop experienced hardware failures  Difficulties installing igraph and mpi4py on pharos System libraries had to be recompiled Environment variables were reset so igraph and mpi4py could be recognized on all nodes  Incomplete mpi4py documentation  Python extended debugger not installed; no graphical front-end

Parallelism can be used to speed up RMG for some case studies  Saw speed up for intermediate-sized case studies on parallel prototype  Additional opportunities for parallelism within RMG algorithm  Can also decrease MPI communication costs w/ additional development, use of debugger/profiler

Future Work  Install extended Python debugger/profiler  Use NumPy and MPI derived data type to reduced communication overhead  Try alternative strategies for parallelization: Reorganize algorithm (check core species, then postulated species) Parallelize checks of postulated species against themselves

Acknowledgments  RMG team: Franklin Goldsmith Sandeep Sharma Josh Allen Richard West Michael Harper Greg Magoon  Ray Speth  Kushal Kedia  Prof. Bill Green  DOE CSGF for funding