1. Parallelization of a Non-Linear Analysis Code Lee Hively and Jim Nutaro (mentors) Computational Sciences and Engineering Travis Whitlow Research Alliance in Math and Science Program Alabama Agricultural and Mechanical University August 8, 2007 Oak Ridge, Tennessee

2. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 2 Outline  Problem statement  Research/objectives  Solution approach  MPI (Message Passing Interface)  Results  Summary

3. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 3 Problem FORTRAN-based, research code Predicts biomedical events (seizure) Searches over large parameter space 1≤ w ≤ 500(500) 5000 ≤ N ≤ 100,000(95,000) 2 ≤ S ≤ 100 2 ≤ d ≤ 26 2 ≤ λ ≤ 250(250) 0 ≤ uE ≤ 1(2) (231) “Lots of combinations”

4. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 4 What is Parallel Computing? Theory Executable task can be divided into smaller chunks and run on multiple processors to achieve faster results Each job is calculated simultaneously Output Initialization Sent to processorsGather results

5. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 5 Parallel Computing Dependencies  Software compatibility  Memory  Each program segment independent of other segments

6. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 6 Research ORNL developed and patented novel forewarning technology Biomedical events Machine failures FORTRAN-based research code Accurately predicts epileptic seizures 4.5 hours of forewarning R & D 100 Award

7. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 7 Design Patterns of the Code Initialization Begins with process-indicative, time-serial data Rejects inadequate data quality to avoid the garbage-in-garbage- out syndrome Removes confounding artifacts Eye blinks Muscular artifacts Converts filtered data Statistical distribution function Captures a baseline signature Compares to signatures of dissimilarity Indicates condition change that detects or predicts the event Accurate prediction requires that the algorithm use a good parameter set

8. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 8 Goal  Perform Monte-Carlo search over large space of possible parameters  Analyze each parameter-set choice  Compare predictions made with each parameter set to known data sets How well can this parameter set predict events? True positive(s) True negative(s) The determination of the total true rate (sum of true positives and true negatives) for a specific set of parameters is independent of every other parameter set – Ideal problem for parallel computing!

9. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 9 Seizure Prediction Process 1. EEG data gathered and sent to SeizAlert device 2(a). Artifact removal done and discrete points generated 2(b). Analysis of dissimilarity 3. Forewarning result 1 2 3

10. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 10 Objectives 1.Parallelize existing research-class, nonlinear statistical FORTRAN code 2. Analyze Brain wave seizure data Statistical parameters for seizure forewarning 3.Find parameter sets that maximize total true rate

11. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 11 Approach  Note all possible options  Choose appropriate? suitable? option  Develop ways to implement that option MPI

12. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 12 MPI (Message Passing Interface) What is message passing? Programming paradigm used widely on parallel computers with distributed memory and on networks of workstations (NOWs) Head node

13. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 13 How Does MPI Work? Program Example integer :: a a = 0 do I = 1,10 a = a+1 end do print*, a FORTRANFORTRAN + MPI include ‘mpif.h’ integer myrank,a call MPI_INIT (…) call MPI_COMM_RANK(…) … call MPI_FINALIZE end

14. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 14 Results After each iteration, a single-line forewarning summary is given detailing attributes: Parameter set values True positives and true negatives The total true rate A standard forewarning summary would be written as, 1 189 19640 2 3 28 0 10 1 6 5 0.826 0.9167 5 0 controls search areaforewarning details

15. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 15 Isolating each value, abcdefghijklmnp 11891964023280101650.8260.916750 Where, a = montage (0 = monopolar, 1 = bipolar) b = half width of the artifact filter window (w) c = # pts per cutset d = # of dimensions in the PS* e = # symbols (S) f = time delay lag (λ ) g = uE (0 = uniform, 1 = equiprobable) h = # of base-case cutsets i = inter-PS*-symbol lag j = # DM above threshold within successive occurrence window k = # successive occurrences above l l = threshold m = total true rate (%) n = # false negatives p = # false positives *PS - phase-space, DM - dissimilarity measures

16. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 16 Also, successive runs performed on 2,4,8,16,32,64,128, and 256 processors Each run had unique random seed generator Head node not included (a) # N# PN * PCPU time (s)Relative CPU time 50021000 32889.36 1.0000 50042000 32943.71 1.0017 50084000 38938.10 1.1840 500168000 ------------------------- 5003216000 69948.41 2.1268 5006432000 41355.70 1.2574 50012864000 41197.871.2526 500256128000 34560.74 1.0508 N = parameter sets P = processors 234000 a

17. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 17 Plotting the relative CPU time…

18. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 18 Analysis Examined the forewarning summaries noticed that it was possible to choose better parameters by graphing the total true rate versus each parameter. By doing so: Visible parameter value region is shown See which values generate higher total true rates #NODES w N d S λ UE 128 283 19659 3 3 64 0 128 166 30199 2 39 202 0 64 356 39039 4 4 84 1 32 256 20104 2 3 143 0 TTR=.90

19. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 19 Total true rate vs. parameter(s)

20. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 20 Summary Research highly dependent on ability to produce numerous results without sacrificing time. Achieved with MPI (Message Passing Interface) Obtained multiple results in single result time frame Chose better parameter visually Graphed total true rate vs. parameter(s)

21. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 21 Acknowledgements Dr. Lee Hively Dr. Jim Nutaro Dr. Nancy Munro Mark Elmore Debbie McCoy Dr. Z.T. Deng and Dr. Cathy Qian

22. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 22 The Research Alliance in Math and Science program is sponsored by the Mathematical, Information, and Computational Sciences Division, Office of Advanced Scientific Computing Research, U.S. Department of Energy. The work was performed at the Oak Ridge National Laboratory, which is managed by UT-Battelle, LLC under Contract No. De-AC05-00OR22725. This work has been authored by a contractor of the U.S. Government, accordingly, the U.S. Government retains a non-exclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S Government purposes.

23. O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY 23 ??