1. Center for Subsurface Sensing & Imaging Systems HP Visit @ UPRM February 27, 2003 Overview of Research Thrust R3 R3 Fundamental Research Topics R3A Parallel Processing Middleware Tools FPGA Acceleration R3B Solutionware Development Image and Sensor Data Databases Subsurface Toolboxes R3 Fundamental Research Topics R3A Parallel Processing Middleware Tools FPGA Acceleration R3B Solutionware Development Image and Sensor Data Databases Subsurface Toolboxes David Kaeli, Northeastern University

2. CenSSIS Barriers Addressed by R3 Projects Lack of Computationally Efficient, Realistic Models Barrier 6 Lack of Rapid Processing and Management of Large Image Databases Barrier 7 Lack of Validated, Integrated Processing and Computational Tools Lack of Validated, Integrated Processing and Computational Tools Barrier 4

3. Middleware Tools  Parallelization using MPI – a software pathway to exploiting GRID-level resources  Presently exploiting NCSA resources (Mariner Center at BU) and Internet-2  Profile-guided program instrumentation/optimization  Developing parallelization strategies for MATLAB  Utilizing MPI-2 to address barriers in I/O performance Impact on SSI applications:  Reduced the runtime of a single-body Steepest Descent Fast Multipole Method (SDFMM) application by 74% on a 32-node Beowulf cluster Hot-path parallelization Data restructuring  Reduced the runtime of a Monte Carlo scattered light simulation by 98% on a 16-node SGI Origin 2000 Matlab-to-C compliation Hot-path parallelization  Parallelization using MPI – a software pathway to exploiting GRID-level resources  Presently exploiting NCSA resources (Mariner Center at BU) and Internet-2  Profile-guided program instrumentation/optimization  Developing parallelization strategies for MATLAB  Utilizing MPI-2 to address barriers in I/O performance Impact on SSI applications:  Reduced the runtime of a single-body Steepest Descent Fast Multipole Method (SDFMM) application by 74% on a 32-node Beowulf cluster Hot-path parallelization Data restructuring  Reduced the runtime of a Monte Carlo scattered light simulation by 98% on a 16-node SGI Origin 2000 Matlab-to-C compliation Hot-path parallelization Soil Air Mine

4. Techniques for Parallelizing MATLAB  Manage completely independent MATLAB processes distributed over different processors  Message passing within MATLAB (e.g., MultiMATLAB)  MATLAB calls to parallel libraries (multi-threaded LAPACK, PLAPACK)  Backend compilers can convert MATLAB to C, and automatically inserting MPI calls (e.g., RTExpress) Multiple MATLAB sessions A Single MATLAB session

5. Our Approach for Parallelizing MATLAB Convert MATLAB to C using the MATLAB mcc compiler Profile the C program to capture both data flow and control flow Parallelize the “hot” regions of the the application using MPI Convert array structs (generated by mcc) to pointer-based structs where needed Main 0/2502 Tfqmr 1/1604 3/35 Mult1 1/1602 Mult 4/1599 Multfaflone4m 523/6 Multfaflone4 359/5 Multdifl 706/0 0/0 18/0 16/0 0/0 1 1 1 1 273 54K 2.4M 0.6M Function Self /Children Runtime Number of Calls

6. Eliminating I/O Barriers in Parallel SSI Applications  Many SSI applications tend to be file bound or memory bound (or both)  While we can use MPI to parallelize processing, and use MPI-I/O to perform collective I/O, we still are limited to access to a file on a single physical disk  Our present work looks at parallelizing I/O by partitioning files associated with MPI processes  We attempt to utilize, slower and commodity (IDE) local secondary storage  Many SSI applications tend to be file bound or memory bound (or both)  While we can use MPI to parallelize processing, and use MPI-I/O to perform collective I/O, we still are limited to access to a file on a single physical disk  Our present work looks at parallelizing I/O by partitioning files associated with MPI processes  We attempt to utilize, slower and commodity (IDE) local secondary storage

7. Eliminating I/O Barriers in Parallel SSI Applications  Parallelize application using MPI  Parallelize I/O using MPI collective I/O  Profile I/O access patterns on a per process basis  Use profile to guide partitioning to reduce overall execution time by 27%-82%  Presently targeting both file- bound and memory-bound applications

8. Some Recent Publications “Profile-based Characterization and Tuning Subsurface Sensing Applications”, M. Ashouei, D. Jiang, W. Meleis, D. Kaeli, M. El-Shenawee, E. Mizan, M. and C. Rappaport, Special issue of the SCS Journal, November 2002. “Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM) on a Beowulf Cluster for Subsurface Sensing Application”, D. Jiang, W. Meleis, M. El- Shenawee, E. Mizan, M. Ashouei, and C. Rappaport, IEEE Microwave and Wireless Components Letters, January 2002. “Electromagnetics Computations Using MPI Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM)”, M. El-Shenawee, C. Rappaport, D. Jiang, W. Meleis, and D. Kaeli, Applied Computational Electromagnetics Society Journal, August 2002. “Profile-Guided I/O Partitioning”, Y. Wang and D. Kaeli, Submitted to ICS’03. Other techniques being assessed in this work:  Globus Toolkit – NSF Middleware Initiative  MatlabMPI – MIT LL  RTExpress - Parallelizing compiler for Matlab  Sparse matrix representations “Profile-based Characterization and Tuning Subsurface Sensing Applications”, M. Ashouei, D. Jiang, W. Meleis, D. Kaeli, M. El-Shenawee, E. Mizan, M. and C. Rappaport, Special issue of the SCS Journal, November 2002. “Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM) on a Beowulf Cluster for Subsurface Sensing Application”, D. Jiang, W. Meleis, M. El- Shenawee, E. Mizan, M. Ashouei, and C. Rappaport, IEEE Microwave and Wireless Components Letters, January 2002. “Electromagnetics Computations Using MPI Parallel Implementation of the Steepest Descent Fast Multipole Method (SDFMM)”, M. El-Shenawee, C. Rappaport, D. Jiang, W. Meleis, and D. Kaeli, Applied Computational Electromagnetics Society Journal, August 2002. “Profile-Guided I/O Partitioning”, Y. Wang and D. Kaeli, Submitted to ICS’03. Other techniques being assessed in this work:  Globus Toolkit – NSF Middleware Initiative  MatlabMPI – MIT LL  RTExpress - Parallelizing compiler for Matlab  Sparse matrix representations

9. Embedded Processing for Accelerated Vessel Enhancement in Retinal Images The Application  Processing live video of the patient retina during laser retinal surgery  Highlighting the vascular structure helps the surgeon avoid damage  Retinal vascular tracing; detection of blood vessels in images of the retina  Utilizes a k-means matched filter to find blood vessels and traces out structure  Embedded into a Xilinx Virtex-based Firebird Computing Engine  Data are 1024x1024 images can be processed at 30 frames/sec  Embedded processing, combined with parallel processing, provides a powerful heterogeneous computing platform for discovery  Presently being applied to identify vascular structures present in sub- skin tumors @ MGH The Application  Processing live video of the patient retina during laser retinal surgery  Highlighting the vascular structure helps the surgeon avoid damage  Retinal vascular tracing; detection of blood vessels in images of the retina  Utilizes a k-means matched filter to find blood vessels and traces out structure  Embedded into a Xilinx Virtex-based Firebird Computing Engine  Data are 1024x1024 images can be processed at 30 frames/sec  Embedded processing, combined with parallel processing, provides a powerful heterogeneous computing platform for discovery  Presently being applied to identify vascular structures present in sub- skin tumors @ MGH NU/RPI Research Team Miriam Leeser – NU-ECE Badri Roysam – RPI-ECSE

10. CenSSIS Solutionware Toolbox Development  Support the development of CenSSIS Solutionware that demonstrates our “Diverse Problems – Similar Solutions” model  Delivered a software-engineered Multi-View Tomography Toolbox, developed in OOMATLAB  Developing three new CenSSIS Toolboxes  Registration – RPI/WHOI  Hyperspectral Imaging – UPRM  3-D Modeling - NEU  Establish software development and testing standards for CenSSIS Image and Sensor Data Database  Develop an web-accessible image database for CenSSIS that enables efficient searching and querying of images, metadata and image content  Develop image feature tagging capabilities Toolbox Development  Support the development of CenSSIS Solutionware that demonstrates our “Diverse Problems – Similar Solutions” model  Delivered a software-engineered Multi-View Tomography Toolbox, developed in OOMATLAB  Developing three new CenSSIS Toolboxes  Registration – RPI/WHOI  Hyperspectral Imaging – UPRM  3-D Modeling - NEU  Establish software development and testing standards for CenSSIS Image and Sensor Data Database  Develop an web-accessible image database for CenSSIS that enables efficient searching and querying of images, metadata and image content  Develop image feature tagging capabilities Matlab 6

11. CenSSIS Toolbox Development CenSSIS Applications written in MATLAB, C, C++, Java, MPI, VHDL and Verilog MVT MSDLPMModeling  Identify classes of CenSSIS algorithms that can be more generally specified to extend applications to multiple research domains  Utilize Software Engineering principles to produce reusable and extensible software toolboxes Object-oriented design principles used Exploratory Software Development Model Library and revision control to support infrastructure  Identify classes of CenSSIS algorithms that can be more generally specified to extend applications to multiple research domains  Utilize Software Engineering principles to produce reusable and extensible software toolboxes Object-oriented design principles used Exploratory Software Development Model Library and revision control to support infrastructure

12. Inverse Modeling Framework OO-MATLAB The CenSSIS Multi-View Tomography (MVT) Toolbox Matlab 6 Electrical Impedance Tomography Electrical Resistance Tomography Diffuse Optical Tomography MVT Frequency-Domain Diffusion Library OO-C++ and IML++ www.censsis.neu.edu/MVT/mvt.html

13. Current Toolbox Development  Development of multi-language toolboxes  Performance tuning  Presently working on three additional toolbox effort  Developing an MPI-2 version of the MVT Toolbox  Development of multi-language toolboxes  Performance tuning  Presently working on three additional toolbox effort  Developing an MPI-2 version of the MVT Toolbox Matlab 6 MSD LPM Modeling

14.  Deliver an web-accessible database for CenSSIS that enables efficient searching and querying of images, sensor data, metadata and image content  Database Characteristics: Relational complex queries (Oracle8i) Data security, reliability and layered user privileges Efficient search and query of image content and metadata Content-based image tagging using XML adopting MPEG-7 standards Easy upload and registration of user images and metadata Indexing algorithms (2D, 3D and 4D) and partitioning of the database for better performance Explore object relational technology to handle collections CenSSIS Image Database System 1 2 3 4 nucleus

15. 9 931175495 2/7/30 F 11/2/1993 Breast II POORLY DIFFERENTIATED UNKNOWN NONE 4/1/1992 NONE p jpeg PAL CD color 332.228........ <Identifier IdOrganization='Clinomics' IdName= 'BreastCancerCell'>BreastCancerCell// image0001 file://D:/Breast/cells/imag0001.jpg Patient239 Human primary breast tumor cells growing in a NASA Bioreactor St. Mary’s Hospital 09/25/2000 Investigate tumor cells behaviour on microcarrier beads Higher magnification of view illustrating breast cancer cells with intercellular boundaries on bead surface General Image Info Image Source Info Image Format Info Image File Location Info Image Donor Info Image Feature Info

16. CenSSIS R3 Research Thrust Summary  Providing both SSI-related computing research expertise and supporting CenSSIS infrastructure needs  Addressing key research barriers in computational efficiency, embedded computing and image/sensor data management  Developing GRID-level tools and an services I/O Partitioning Algorithms Subsurface Toolboxes Image/Sensor Data Databases  Providing both SSI-related computing research expertise and supporting CenSSIS infrastructure needs  Addressing key research barriers in computational efficiency, embedded computing and image/sensor data management  Developing GRID-level tools and an services I/O Partitioning Algorithms Subsurface Toolboxes Image/Sensor Data Databases