"Parallel MATLAB in production supercomputing with applications in signal and image processing" Ashok Krishnamurthy David Hudak John Nehrbass Siddharth Samsi Vijay Gadepally

2 Functions - Scope of Activity Supercomputing. Computation, software, storage, and support services empower Ohio’s scientists, engineers, faculty, students, businesses and other clients. Networking. Ohio’s universities, colleges, K-12 and state government connect to the network. OSC also provides engineering services, video conferencing, and support through a 24x7 service desk. Research. Lead science and engineering projects, assist researchers with custom needs, partner with regional, national, and international researchers in groundbreaking initiatives, and develop new tools. Education. The Ralph Regula School of Computational Science delivers computational science training to students and companies across Ohio.

3 OSC provides a stable computational infrastructure to support scientific research computing IBM 1350 Opteron dual-core w/IBM Cell 4,000+ cores 8 TBytes memory 22+ teraflops Blend of 4 core and 8 core nodes –Large processor count –Large memory SMP jobs Intel P4 Cluster 2.46 TF 512 processors 1 TBytes memory Itanium2 Cluster 2.7 TF 596 processors 3x16 Alt 1 TBytes memory PRODUCTION COMPUTING Mass Storage 470 TBytes disk 80 TBytes tape NFS, PVFS, iSCSI Infiniband or Myrinet Interconnect Gateway to User Science

4 Visualization Cluster AMD Opteron 72 processors 144 GBytes memory nVIDIA Quadro 5600 graphics card(330 GF) Providing agile computational infrastructure to support the research and innovation process Apple G5 Cluster PowerPC G5 64 processors 128 GBytes mem MATLAB/GRI Cluster AMD Opteron 164 processors 328 GBytes memory RESEARCH COMPUTING Mass Storage 470 TBytes disk 80 TBytes tape NFS, PVFS, iSCSI Gateway to User Science BALE Cluster AMD Athlon processors 220 GBytes memory nVIDIA GeForce 6150 GPU

5 OSC Instrumentation and Analytics Services Remote instrumentation uses OSC’s state-wide resources –Networking, Storage, HPC, Analytics (web service) Parallel MATLAB

Parallel MATLAB Choices MathWorks Distributed Computing Engine + Toolbox Interactive Supercomputing’s Star-P MIT Lincoln Labs pMATLAB + OSC bcMPI EMPOWER. PARTNER. LEAD.

MATLAB® Distributed Computing Toolbox Architecture Image source:

EMPOWER. PARTNER. LEAD. Star-P Architecture Image source:

Dedicated Cluster: Desktop client can directly address cluster compute nodes

Typical Shared Production Cluster: Desktop client cannot directly address cluster compute nodes

Parallel MATLAB Computing Configurations Have to map client+engines to a number of computer configurations Local or Remote –Local - single administrative domain Uniform set of user accounts Implicit trust relationship –Remote - multiple administrative domains Authentication required Dedicated or shared –Dedicated - preallocated resources available on demand –Shared - allocation request must complete prior to interactive session

EMPOWER. PARTNER. LEAD. MATLAB Distributed Computing Engine & Toolbox Mathworks implementation of parallel MATLAB Consists of two products : –MATLAB® Distributed Computing Engine (MDCE) –Distributed Computing Toolbox (DCT) MDCE enables users to run MATLAB applications on a cluster DCT provides toolbox migration : Client’s toolbox licenses are available when the parallel job runs on the cluster Can be used in interactive mode as well as in non- interactive batch jobs

EMPOWER. PARTNER. LEAD. MATLAB® Distributed Computing Engine Customization of scripts to run under shared, remote resources which includes –System specific batch scripts must be generated at run time –Authentication and remote connection setup when a job is submitted to the cluster

EMPOWER. PARTNER. LEAD. MATLAB® Distributed Computing Engine Customization Client side : –Need a custom job submission function as defined in the MDCE documentation for job submission –Need to provide a mechanism to copy job data over to cluster –Client must also resolve any file dependencies for the algorithm being run Image source: Solution number : 1-2KEGMH

EMPOWER. PARTNER. LEAD. MATLAB® Distributed Computing Engine Customization Cluster side customization: –Need “decode” function on MATLAB path on the cluster, as specified in the documentation –Some customization of MPI launch script is required Image source: Solution number : 1-2KEGMH

EMPOWER. PARTNER. LEAD. Star-P Star-P is a client-server parallel computing platform available from Interactive Supercomputing Designed to work with high level languages such as MATLAB and Python Designed for interactive usage

EMPOWER. PARTNER. LEAD. Star-P Customization required to run under the Torque workload manager –Custom parameters to be used include command-line options that control job submission to the cluster –Custom MPI launch mechanism developed by Interactive Supercomputing for use on OSC clusters Does not support toolbox migration Monitoring and debugging server backend processes is still a challenge

EMPOWER. PARTNER. LEAD. pMATLAB + bcMPI Runs on UNIX: tested on Linux, NetBSD, MacOS X API with MatlabMPI –If you can use MatlabMPI, you can probably use bcMPI –bcMPI tags are numeric, MatlabMPI alphanumeric Broadcast, barrier, reduce operations bcMPI supports synchronous or asynchronous sends –MPI_Buffer_attach, MPI_Buffer_detatch, MPI_Probe MPI communicator support (new in v1.1) –Supports many MATLAB data types, but no sparse support

EMPOWER. PARTNER. LEAD. bcMPI Advantages : –Extensible - core library makes it easy to add additional MPI functions and interpreter data types –Portable - no dependencies on any machine, or specific MPI library implementations –Scalable - use efficient algorithms; take advantage of native MPI library and communications hardware –Open source software developed at OSC Can be used only in non-interactive batch jobs

20 Example of remote instrumentation application: access to electron microscope Parallel MATLAB Example: Image analysis of Scanning Electron Microscope Images Scanning Electron Microscope at OSU Center for Accelerated Maturation of Materials Demonstrates real-time user control Adding analytics and collaboration services for image analysis and computational modeling

CAMM Portal Interface 21

EMPOWER. PARTNER. LEAD. MATLAB GUI to set image processing parameters

SEGMENTS ADDED TOGETHER

Parallel Processing : MATLAB GUI

Parallel MATLAB : Results 396 images processed on 2, 4, 8, 12 and 16 processors

EMPOWER. PARTNER. LEAD. Parallel MATLAB Application : Acoustics Signal Processing MATLAB used to analyze acoustic signatures used for self-localization of sensors Comparative analysis using multiple algorithms on multiple data sets – embarrassingly parallel

EMPOWER. PARTNER. LEAD. Parallel MATLAB application : Synthetic Aperture Radar Model [1] Develop synthetic aperture radar model for forest clusters MATLAB used to study invertibility of forest model and to fit model parameters to SAR data from several forests Parallel MATLAB used to perform above studies [1] A Model for Generating Synthetic VHF SAR Forest Clutter Images – Julie Ann Jackson, Randolph L. Moses. Paper submitted to IEEE Aerospace and Electronic Systems Journal

Parallel MATLAB Application: Pathology slide image processing

Summary Parallel MATLAB is of interest to many in OSC’s user community Parallel MATLAB is bringing in new users to High Performance Computing Users would like the same experience as desktop MATLAB, but are willing to give up some of it for quicker turn-around time Providing a complete solution is a lot of work – even for embarrassingly parallel cases EMPOWER. PARTNER. LEAD.