Parallel and Distributed Computing for Neuroinformatics Allen D. Malony University of Oregon Professor Department of Computer and Information Science Director NeuroInformatics Center Computational Science Institute

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Outline  Neuroinformatics  Dynamic brain analysis problem  NeuroInformatics Center (NIC) at UO  Neuroinformatics research at the NIC  Dense-array EEG analysis (APECS, HiPerSat, Mc)  Brain image segmentation  Computational head modeling  Ontologies and tool integration (NEMO, GEMINI)  Parallel and distributed computing emphasis  ICONIC Grid HPC system at UO  Cerebral Data Systems  Oregon E-science

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Neuroscience and Neuroinformatics  Application of computer and information science to the understanding of brain organization and function  Integration of information across many levels  Physical and functional  Gene to behavior  Microscopic to macroscopic scales  Challenges in brain observation and modeling  Structure and organization (imaging)  Operational and functional dynamics (temporal/spatial)  Physical, functional, and cognitive operation (models)  How to create and maintain of integrated views of the brain for both scientific and clinical purposes?

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Human Brain Dynamics Analysis Problem  Understand functional operation of the human cortex  Dynamic cortex activation  Link to sensory/motor and cognitive activities  Multiple experimental paradigms and methods  Multiple research, clinical, and medical domains  Need for coupled/integrated modeling and analysis  Multi-modal observation (electromagnetic, MR, optical)  Physical brain models and theoretical cognitive models  Need for robust tools  Complex analysis of large multi-model data  Reasoning and interpretation of brain behavior  Problem solving environment for brain analysis

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Neuroimaging Technology  Alternative techniques for analyzing brain dynamics  Blood flow neuroimaging (PET, fMRI)  Good spatial resolution  allows functional brain mapping  Temporal limitations to tracking of dynamic activities  Electromagnetic measures (EEG/ERP, MEG)  msec temporal resolution  can distinguish fast neurological components  Spatial resolution sub-optimal  requires a mapping to cortical sources  Need integrated neuroimaging technology (How?)  Achieve both good spatial and temporal resolution

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Integrated Dynamic Brain Analysis Individual Brain Analysis Structural / Functional MRI/PET Dense Array EEG / MEG Constraint Analysis Head Analysis Source Analysis Signal Analysis Response Analysis Experiment subject temporal dynamics neural constraints Cortical Activity Model Component Response Model spatial pattern recognition temporal pattern recognition Cortical Activity Knowledge Base Component Response Knowledge Base good spatial poor temporal poor spatial good temporal neuroimaging integration

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Experimental Methodology and Tool Integration source localization constrained to cortical surface processed EEG BrainVoyager BESA CT / MRI EEG segmented tissues 16x256 bits per millisec (30MB/m) mesh generation EMSE Interpolator 3D NetStation

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics NeuroInformatics Center (NIC) at UO  Application of computational science methods to human neuroscience problems  Tools to help understand dynamic brain function  Tools to help diagnosis brain-related disorders  HPC simulation, large-scale data analysis, visualization  Integration of neuroimaging methods and technology  Need for coupled modeling (EEG/ERP, MR analysis)  Apply advanced statistical signal analysis (PCA, ICA)  Develop computational brain models (FDM, FEM)  Build source localization models (dipole, linear inverse)  Optimize temporal and spatial resolution  Internet-based capabilities for brain analysis services, data archiving, and data mining

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics NIC Organization  Allen D. Malony, Director  Don M. Tucker, Associate Director  Sergei Turovets, Computational Physicist  Bob Frank, Mathematician  Dan Keith, Software Engineer (distributed systems grid)  Chris Hoge, Software Engineer (computational)  Ryan Martin / Brad Davidson, Systems administrators  Gwen Frishkoff, Research Associate, Univ. Pittsburgh  Kai Li, Ph.D. student (brain segmentation)  Adnan Salman, Ph.D. student (computational modeling)  Performance Research Lab (my other hat)

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Brainwave Research 101  Electroencephalogram (EEG)  Electrodes (sensors) measure uV  EEG time series analysis  Event-related potentials (ERP)  link brain activity to sensory–motor, cognitive functions  statistical average to increase signal-to-noise ratio (SNR)  Signal cleaning and component decomposition  Localize (map) to neural sources

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Electrical Geodesics Inc. (EGI)  EGI Geodesics Sensor Net  Dense-array sensor technology  64/128/256 channels  256-channel GSN  AgCl plastic electrodes  Net Station  Advanced EEG/ERP data analysis and visualization  Stereotactic EEG sensor registration  State of the art technology  Research / clinical products  EGI/CDS medical services

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Epilepsy  Epilepsy affects more than 5 million people yearly  U.S., Europe, and Japan  EEG in epilepsy diagnosis  Childhood and juvenile absence  Idiopathic (genetic)  Distinguish different types  EEG in presurgical planning  Localize seizure onset  Fast, safe, inexpensive  Dense array improves accuracy  Requires good source modeling

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics EEG Time Series - Progression of Absence Seizure First full spike–wave Pre-spike “buzz’

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Topographic Waveforms – First Full Spike-Wave 350ms interval

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Topographic Mapping of Spike-Wave Progression  Palette scaled for wave-and-spike interval (~350ms) -130 uV (dark blue)  75 uV (dark red)  1 millisecond temporal resolution is required  Spatial density (256) to capture shifts in topography

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics  Dense-array EEG signal analysis and decomposition  Artifact cleaning and component analysis  Automatic brain image segmentation  Brain tissue identification  Cortex extraction  Computational head modeling  Tissue conductivity estimation  Source localization  Statistical analysis to detect brain states  Discriminant analysis  Pattern recognition  Electromagnetic databases and ontologies Neuroinformatic Challenges

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Applying ICA for EEG Blink Removal  Component analysis is used to separate EEG signals  Independent component analysis (ICA)  Blinks are a major source of noise in EEG data  Blink signals are separable from cognitive responses Raw EEG Formatting EEG preprocessing  Event info  Time markers  Blink events  Bad channel removal  Baseline correction  … ICA Analysis  ICA algorithms - Infomax - FastICA - …  ICA components Identify blinks and remove  Blink templates  Reconstitute EEG w/out blink data ERP Analysis

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Independent Component Analysis Mixed sinusoids - raw EEG Original sinusoids ICA EEG waveform

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Tool for EEG Data Decomposition (APECS)  Automated Protocol for Electromagnetic Component Separation (APECS)  Motivation  EEG data cleaning (increases signal-to-noise (SNR))  EEG component separation (addresses superposition)  Data preprocessing prior to source localization  Distinctive Features  Implements different decomposition methods  Multiple metrics for component classification  Quantitative and qualitative criteria for evaluation

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics APECS Evaluation: Qualitative Criteria BadGoodOriginal Spatio-temporal blink profiles Blink-free baseline

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics APECS Evaluation: Quantitative Criteria Covariance between “baseline” (blink-free) and ICA-filtered data: Infomax and FastICA. Infomax gives consistently better results. FastICA results are more variable. ICA decompositions most successful when only one spatial projector is strongly correlated with blink “template” (spatial filter).

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Parallelization of Component Analysis Algorithms  Dense-array EEG increases analysis complexity  Long time measurements require more processing  ICA algorithms are computationally challenging  Processing time and memory requirements  Increase performance through ICA parallelization  High-Performance Signal Analysis Toolkit (HiPerSAT)  Matlab ICA algorithms implemented in C++  Infomax: Matlab ( runica.m )  parallel (OpenMP)  FastICA: Matlab ( fastica.m )  parallel (MPI)  Validate results with Matlab standard algorithms  Evaluate accuracy and compare speedup

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics HiPerSAT Parallel Infomax  Requires multi-threading  Over 3 times faster than Matlab  3-fold increase on 4 processors  Speedup falls after 4 processors  Limits on parallelization of loop matrix operations  May be able to improve with larger blocking

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics HiPerSAT Parallel FastICA  Linear speedup  Over 130 times faster than Matlab  8-fold increase on 32 processors  Performance may allow finer processing  Tradeoff improved accuracy versus more complex processing and execution time

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Matlab Tool Integration  Many neuroimaging tools are based on Matlab  EEGLAB (UC San Diego)  BrainStorm (USC / Los Alamos National Lab)  SPM (University College of London)  Matlab is mostly a closed computational environment  EEG/MEG analysis can overwhelm Matlab  Limited to workstation processing resources  Memory requirements high due to Matlab workspace  Desire to use Matlab as a client in distributed systems  Matlab is not multi-threaded  Complicates building concurrent for external interfaces

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Mc: Matlab Concurrent  Matlab is built on top of a JVM  Use for GUI and graphics  We can leverage JVM for concurrency and interaction  How do we create concurrent tasks in Matlab?  Matlab programming semantics issue  Create task abstraction  Provide Matlab package for constructing tasks  Concurrent tasks interface with runtime layer  Client task manager runs tasks on servers and monitors  Server task executor schedules tasks on resources  Voila!  Mc

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Mc System Architecture

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Mc Task Manager  Single background thread running in the JVM  Coordinate several components:  File transfer  Ganymed SSH2 implementation for Java  Matlab interaction  MatlabControl class  Scripting with Jython  dynamic access to objects in the JVM  Matlab m-scripting through MatlabControl class waits for free cycles when execution Matlab code  Responds to user or timer events  Manages execution of tasks and status reporting

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Task Manager Workflow

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics HiPerSAT, APECS, and Mc  Demonstrated APECS running HiPerSAT tasks  20 simultaneous HiPerSAT servers on a cluster  Compare with sequential EEGLAB processing  ICA processing on 20 1-GB EEG files  Performance speedup  5x for C++ optimization and 2-way parallelism  10x speedup for 20 concurrent HiPerSAT tasks  Also demonstrate for multiple Matlab clients  Recently applied to EEG processing for imagery analysis  100+ simultaneous tasks across 8 different platforms

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Papers  K. Glass, G. Frishkoff, R. Frank, C. Davey, J. Dien, A. Malony, A Framework for Evaluating ICA Methods of Artifact Removal from Multichannel EEG, ICA Conference, Grenada, Spain,  R. Frank and G. Frishkoff, APECS: A Framework for Implementation and Evaluation of Blink Extraction from Multichannel EEG, Journal of Clinical Neurophysiology, to appear,  D. Keith, C. Hoge, A. Malony, and R. Frank, “Parallel ICA Methods for EEG Neuroimaging,” International Parallel and Distributed Processing Symposium (IPDPS 2006), May  C. Hoge, D. Keith, and A. Malony, “Client-side Task Support in Matlab for Concurrent Distributed Execution,” Austrian- Hungarian Workshop on Distributed and Parallel Systems (DAPSYS), September 2006.

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics  Spatial and temporal dynamics are important to observe together  Linked cortical networks  Fronto-thalamic circuit (executive control)  Limbic circuit (episodic memory)  Problem of Superposition  How many sources?  Where are they located?  Can only infer locations using “scalp space” information Animated Topography of Spike–Wave Dynamics

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Addressing Superposition: Brain Electrical Fields  Brain electrical fields are dipolar  Volume conduction  Depth and location indeterminacy  Highly resistive skull (CSF: skull est. from 1:40 to 1:80)  Left-hemisphere scalp field may be generated by a right-hemisphere source  Multiple sources  superposition  Radial source  Tangential sources  one and two sources  varying depths

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Source Localization  Mapping of scalp potentials to cortical generators  Signal decomposition (addressing superposition)  Anatomical source modeling (localization)  Source modeling  Anatomical Constraints  Accurate head model and physics  Computational head model formulation  Mathematical Constraints  Criteria (e.g., “smoothness”) to constrain solution  Current solutions limited by  Simplistic geometry  Assumptions of conductivities

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Brain Sources of Epileptic Seizure  Single time point source solution  Need to identify sources for each msec time sample  Visualize dynamics in “source space”

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Dipole Sources in the Cortex  Scalp EEG is generated in the cortex  Interested in dipole location, orientation, and magnitude  Cortical sheet gives possible dipole locations  Orientation is normal to cortical surface  Need to capture convoluted geometry in 3D mesh  From segmented MRI/CT  Linear superposition

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Advanced Image Segmentation  Native MR gives high gray-to-white matter contrast  Image analysis techniques  Edge detection, edge merger, region growing  Level set methods and hybrid methods  Knowledge-based  After segmentation, color contrasts tissue type  Registered segmented MRI

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Network Flow Based Skull Stripping  Graph construction with LoG  Nodes, edges, edge weights, source terminals, sink terminals  Identify non-brain terminals  Scalp, eyeballs, orbits  Fourth ventricle, pons  Identify brain terminals  Knowledge-based terminal detection  Min-cut (Max-flow)  Separate source from sink  Sum of weights is minimum

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Cortical Surface Extraction Pipeline  Initial segmentation of white matter  Scale-space based analysis of LoG  Incorporation of spatial knowledge  Cortical surface topology correction  Minimize cortical surface are increase  Based on network flow theory  Based on Cauchy-Crofton formula  Cortical surface geometry refinement  White matter shape knowledge  radius function constraints  sheet constraints  Medial surface (skeleton) manipulation

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Topology Correction in Cortex Extraction Before topology correction After topology correction Extracted Cortex

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Building Computational Brain Models  MRI segmentation of brain tissues  Conductivity model  Measure head tissue conductivity  Electrical impedance tomography  small currents are injected between electrode pair  resulting potential measured at remaining electrodes  Finite element forward solution  Source inverse modeling  Explicit and implicit methods  Bayesian methodology

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Conductivity Modeling Governing Equations ICS/BCS Discretization System of Algebraic Equations Equation (Matrix) Solver Approximate Solution Continuous Solutions Finite-Difference Finite-Element Boundary-Element Finite-Volume Spectral Discrete Nodal Values Tridiagonal ADI SOR Gauss-Seidel Gaussian elimination  (x,y,z,t) J (x,y,z,t) B (x,y,z,t)

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Conductivity Optimization and Parallelization  Design as a conductivity search problem  Master launches new inverse problems with guesses  Inverse solvers run iterative forward calculations  Compare solutions with measured results  Parallelization approach  Forward solver  Alterative Direction Implicit (ADI) method  Optimization search  Simplex or simulated annealing (Monte Carlo)  Hybrid

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Alternating Direction Implicit (ADI) Method  Finite difference method  C++ with OpenMP for parallelization  LAPACK for matrix operations

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Conductivity Search Architecture

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Conductivity Optimization Dynamics  Attempting to determine tissue conductivity values  Scalp  Skull  Brain  CSF  Simplex algorithm  Values converge to minimal error  Increase number of features in future work

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Multi-Cluster Search Dynamics

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Papers  K. Li, A. Malony, and D. Tucker, “Automatic Brain MR Image Segmentation with Relative Thresholding and Morphological Image Analysis,” International Conference on Computer Vision Theory and Applications (VISAPP), Setúbal, Portugal, February  K. Ki, A. Malony, and D. Tucker, “A Multiscale Morphological Approach to Topology Correction of Cortical Surfaces,” International Workshop on Medical Imaging and Augmented Reality (MIAR 2006), Sanghai, China, August  A. Salman, S. Turovets, A. Malony, J. Eriksen, and D. Tucker, “Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), best paper, May  A. Salman, S. Turovets, A. Malony, V. Vasilov, “Multi-Cluster, Mixed- Mode Computational Modeling of Human Head Conductivity,” International Workshop on OpenMP (IWOMP), June  S. Turovets, A. Salman, A. Malony, P. Poolman, C. Davey, D. Tucker, “Anatomically Constrained Conductivity Estimation of the Human Head in Vivo: Computational Procedure and Preliminary Experiments,” Electrical Impedance Tomography (EIT), July 2006.

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Neural ElectroMagnetic Ontology (NEMO)  How can brain electromagnetic (EEG and MEG) data be compared and integrated across experiments and laboratories?  Need a system for representation, storage, mining, dissemination  Need standardization of methods for measure generation  Identification and labeling of components  Patterns of interest  General agreement on criteria for component identification  Patterns can be hard to identify  Variability in techniques for measure generation  NEMO will address issue by providing  Spatial and temporal ontology database  Use for large-scale data representation, mining, and meta-analysis  Components in average EEG and MEG (ERPs)

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Electromagnetic Data Spaces and Representation

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics NEMO Architecture  Composed of three modules  Database mining  Inference engine  Query (user) interface  Methods for measure generation  Spatial ontologies and temporal ontologies  Cognitive functional mapping  User interactions  Query formulation  Mapping-rule definitions  Scalable integration system  Online repository for storing metadata  Spatio-temporal ontologies, database schema, mappings

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics NEMO System

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics ICONIC Grid SMP Server IBM p655 Graphics SMP SGI Prism Shared Storage System Gbit Campus Backbone NICCIS Internet 2 Shared Memory IBM p690 Distributed Memory IBM JS20 CNI Distributed Memory Dell Pentium Xeon NIC 4x816 2x82x16 graphics workstationsinteractive, immersive vizother campus clusters 40 Terabytes Tape Backup 112 total processors

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Computational Integrated Neuroimaging System

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics CIS Computational and Visualization Laboratory  Support interdisciplinary computer science  Informatics  Computational science  Resource development  Phase 1 (complete)  NSF MRI grant ($1M)  ICONIC Grid  Phase II (underway)  Visualization Lab ($100K) rear projection »3D stereo and 2x2 tiled 3x4 tiled 24” LCD display  Phase III …

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics GEMINI Project  Grid-enabled Methods for Integrated NeuroImaging  Dynamic neuroimaging algorithms and visualization  Grid-based integration (processing and data sharing)  High-end tool integration and environments  Neuroinformatics data ontologies  GEMINI is  A distributable set of programs and configuration files  A set of web services and WSRF-accessible resources  An internet-accessible service executing on systems  Based on open technologies

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics GEMINI Model  Zones  GEMINI resource servers  Tasks  Job representation  Agents  Execution context  Build on endpoint references  Supports workflow

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics GEMINI Architecture

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Leveraging Internet, HPC, and Grid Computing  Telemedicine imaging and neurology  Distributed EEG and MRI measurement and analysis  Neurological medical services  Shared brain data repositories  Remote and rural imaging capabilities  Build on emerging web services and grid technology  Leverage HPC compute and data centers  Create institutional and industry partnerships  Electrical Geodesics, Inc.  Cerebral Data Systems (UO partnership with EGI)  Other industrial partnerships

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Cerebral Data Systems  Partnership between EGI and University of Oregon  Develop and market neuroinformatics services  Neurological medical data transfer, storage, and analysis  High-performance, expert EEG and MR analysis  Telemedicine and distributed collaboration  Shared brain repositories  Target markets  Research and clinical  Epilepsy diagnosis and pre-surgical planning  MR image segmentation  Technology integration  Internet, computional grids, and HPC

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics CDS Computational Server and Imaging Clients

University of LinzSeptember 2006Parallel and Distributed Computing for Neuroinformatics Oregon E-Science Grid Region 4 Region 1 Region 2 Region 3 Region 5 Internet 2 / National LambdaRail Regional networks Institutional partners Regional points of presence PSI