1. Parallel and Distributed Computing for Neuroinformatics
Allen D. Malony
University of Oregon
Professor
Department of Computer and Information Science
Director
NeuroInformatics Center
Computational Science Institute

2. Outline Who am I? Where is the University of Oregon? Neuroinformatics
Dynamic brain analysis problem
NeuroInformatics Center (NIC) at UO
Neuroinformatics research at the NIC
Dense-array EEG analysis (APECS, HiPerSAT)
Brain image segmentation
Computational head modeling
Ontologies and tool integration (NEMO)
Parallel and distributed computing emphasis
Department of Computer and Information Science

3. Who Am I?
Allen D. Malony
Professor Dept. Computer and Information Science University of Oregon
Ph.D., University of Illinois, Urbana-Champaign
Fulbright Research Scholar (The Netherlands, Austria)
Alexander von Humboldt Research Award
National Science Foundation Young Investigator
Research interests
Parallel performance analysis, high-performance computing, scalable parallel software and tools
Computational science, neuroinformatics

4. Where is Oregon?

5. Neuroscience and Neuroinformatics
Understanding of brain organization and function
Integration of information across many levels
Physical and functional
Gene to behavior
Microscopic to macroscopic scales
Challenges in brain observation and modeling
Structure and organization (imaging)
Operational and functional dynamics (temporal/spatial)
Physical, functional, and cognitive operation (models)
Challenges in interpreting brain states and dynamics
How to create and maintain of integrated views of the brain for both scientific and clinical purposes?

6. Human Brain Dynamics Analysis Problem
Understand functional operation of the human cortex
Dynamic cortex activation
Link to sensory/motor and cognitive activities
Multiple experimental paradigms and methods
Multiple research, clinical, and medical domains
Need for coupled/integrated modeling and analysis
Multi-modal observation (electromagnetic, MR, optical)
Physical brain models and theoretical cognitive models
Need for robust tools
Complex analysis of large multi-model data
Reasoning and interpretation of brain behavior
Problem solving environment for brain analysis

7. NeuroInformatics Center (NIC) at UO
Application of computational science methods to human neuroscience problems
Tools to help understand dynamic brain function
Tools to help diagnosis brain-related disorders
HPC simulation, large-scale data analysis, visualization
Integration of neuroimaging methods and technology
Need for coupled modeling (EEG/ERP, MR analysis)
Apply advanced statistical signal analysis (PCA, ICA)
Develop computational brain models (FDM, FEM)
Build source localization models (dipole, linear inverse)
Optimize temporal and spatial resolution
Internet-based capabilities for brain analysis services, data archiving, and data mining

8. NIC Organization Allen D. Malony, Director
Don M. Tucker, Associate Director
Sergei Turovets, Computational Physicist
Bob Frank, Mathematician
Brad Davidson, Systems administrators
Gwen Frishkoff, Research Associate, Univ. Pittsburgh
Kai Li, Ph.D. student (brain segmentation)
Adnan Salman, Ph.D. student (computational modeling)

9. Observing Dynamic Brain Function
Brain activity occurs in cortex
Observing brain activity requires
high temporal and spatial resolution
Cortex activity generates scalp EEG
EEG data (dense-array, 256 channels)
High temporal (1msec) / poor spatial resolution (2D)
MR imaging (fMRI, PET)
Good spatial (3D) / poor temporal resolution (~1.0 sec)
Want both high temporal and spatial resolution
Need to solve source localization problem!!!
Find cortical sources for measured EEG signals

10. Brainwave Research 101 Electroencephalogram (EEG)
Electrodes (sensors) measure uV
EEG time series analysis
Event-related potentials (ERP)
link brain activity to sensory–motor, cognitive functions
statistical average to increase signal-to-noise ratio (SNR)
Signal cleaning and component decomposition
Localize (map) to neural sources

11. Electrical Geodesics Inc. (EGI)
EGI Geodesics Sensor Net
Dense-array sensor technology
64/128/256 channels
256-channel GSN
AgCl plastic electrodes
Net Station
Advanced EEG/ERP data analysis and visualization
Stereotactic EEG sensor registration
State of the art technology
Research / clinical products
EGI/CDS medical services

12. Brain Electrical Fields 101
Brain electrical fields are dipolar
Volume conduction through head
Depth and location indeterminacy
Highly resistive skull (CSF: skull est. from 1:40 to 1:80)
Left-hemisphere scalp field may be generated by a right-hemisphere source
Multiple sources  superposition
Radial source  Tangential sources
one and two sources  varying depths

13. Epilepsy Epilepsy affects more than 5 million people yearly
U.S., Europe, Japan, …
EEG in epilepsy diagnosis
Childhood and juvenile absence
Idiopathic (genetic)
Distinguish different types
EEG in presurgical planning
Localize seizure onset
Fast, safe, inexpensive
Dense array improves accuracy
Requires good source modeling
Unifying Features of Idiopathic Generalized Epilepsies
•Normal neurologically before onset of seizures
•EEG changes appear generally well organized with otherwise normal background
•Long term outlook for cognitive function is good
Many of the epilepsies previously classified as “generalized” are not truly generalized.
•Idiopathic Generalized Seizures occur in neurologically normal children
•Most seizures or epilepsy syndromes previously classified as “generalized” or “undetermined” should be thought of as Epileptic Encephalopathies
•Epileptic Encephalopathies may have an identifiable mechanism or respond to specific therapies

14. EEG Time Series - Progression of Absence Seizure
First full spike–wave
Pre-spike “buzz’
Figure 1. Topographic waveform plot for the first full spike-wave in one patient’s seizure. Following the clinical EEG convention, negative is up. The 256 channels are arrayed in two dimensions as they would be seen looking down on the top of the head, with the nose at the top of the page, and with the lower channels unwrapped to the sides of the page. This 350 ms epoch captures the first wave and spike of this seizure, plus the initial onset of the second wave. Note that the prominent spike-wave pattern over medial, superior frontal sites inverts over lateral, inferior frontal sites, indicating neural sources in medial frontal cortex.

15. Topographic Waveforms – First Full Spike-Wave
350ms interval
Figure 1. Topographic waveform plot for the first full spike-wave in one patient’s seizure. Following the clinical EEG convention, negative is up. The 256 channels are arrayed in two dimensions as they would be seen looking down on the top of the head, with the nose at the top of the page, and with the lower channels unwrapped to the sides of the page. This 350 ms epoch captures the first wave and spike of this seizure, plus the initial onset of the second wave. Note that the prominent spike-wave pattern over medial, superior frontal sites inverts over lateral, inferior frontal sites, indicating neural sources in medial frontal cortex.

16. Topographic Mapping of Spike-Wave Progression
Palette scaled for wave-and-spike interval (~350ms)
-130 uV (dark blue)  75 uV (dark red)
1 millisecond temporal resolution is required
Spatial density (256) to capture shifts in topography
Figure 2. Selected topographic maps for the spike-wave pattern shown in Figure 3. Palette is scaled for this wave and spike interval, from –130 uV (dark blue) to 75 uV (dark red). The interval characterized by each map is 1 ms, with the selections made to illustrate the major topographic transitions of this subject’s spike-wave pattern.

17. Neuroinformatic Challenges
Dense-array EEG signal analysis and decomposition
Artifact cleaning and component analysis
Automatic brain image segmentation
Brain tissue identification
Cortex extraction
Computational head modeling
Tissue conductivity estimation
Source localization
Statistical analysis to detect brain states
Discriminant analysis
Pattern recognition
Electromagnetic databases and ontologies

18. Applying ICA for EEG Blink Removal
Component analysis is used to separate EEG signals
Independent component analysis (ICA)
Blinks are a major source of noise in EEG data
Blink signals are separable from cognitive responses
Raw EEG
Formatting
EEG
preprocessing
ICA
Analysis
Identify blinks
and remove
Event info
Time markers
Blink events
Bad channel removal
Baseline correction …
ICA algorithms - Infomax - FastICA
ICA components
Blink templates
Reconstitute EEG w/out blink data
ERP
Analysis

19. Independent Component Analysis
EEG waveform
Mixed
sinusoids - raw EEG
ICA
Original
sinusoids - test case

20. Tool for EEG Data Decomposition (APECS)
Automated Protocol for Electromagnetic Component Separation (APECS)
Gwen Frishkoff (Univ. Pittsburgh) and Bob Frank (UO)
Motivation
EEG data cleaning (increases signal-to-noise (SNR))
EEG component separation (addresses superposition)
Data preprocessing prior to source localization
Distinctive Features
Implements different decomposition methods
Multiple metrics for component classification
Quantitative and qualitative criteria for evaluation

21. Parallelization of Component Analysis Algorithms
Dense-array EEG increases analysis complexity
Long time measurements require more processing
ICA algorithms are computationally challenging
Processing time and memory requirements
Increase performance through ICA parallelization
High-Performance Signal Analysis Toolkit (HiPerSAT)
Matlab ICA algorithms implemented in C++
Infomax: Matlab (runica.m)  parallel (OpenMP)
FastICA: Matlab (fastica.m)  parallel (MPI)
Validate results with Matlab standard algorithms
Evaluate accuracy and compare speedup

22. HiPerSAT Parallel Infomax
Requires multi-threading
Over 3 times faster than Matlab
3-fold increase on 4 processors
Speedup falls after 4 processors
Limits on parallelization of loop matrix operations
May be able to improve with larger blocking

23. HiPerSAT Parallel FastICA
Parallelize for distributed memory
Linear speedup
Over 130 times faster than Matlab
8-fold increase on 32 processors
Performance may allow finer processing
Tradeoff improved accuracy versus more complex processing and execution time

24. Matlab Tool Integration
Many neuroimaging tools are based on Matlab
EEGLAB (UC San Diego)
BrainStorm (USC / Los Alamos National Lab)
SPM (University College of London)
Matlab is mostly a closed computational environment
EEG/MEG analysis can overwhelm Matlab
Limited to workstation processing resources
Memory requirements high due to Matlab workspace
Desire to use Matlab as a client in distributed systems
Matlab is not multi-threaded
Complicates building concurrent for external interfaces

25. Matlab Concurrent Runtime Engine (MCORE)
Matlab is built on top of a JVM
Used for GUI and graphics
We can leverage JVM for concurrency and interaction
How do we create concurrent tasks in Matlab?
Matlab programming semantics issue
Create task abstraction
Provide Matlab package for constructing tasks
Concurrent tasks interface with runtime layer
Client task manager runs tasks on servers and monitors
Server task executor schedules tasks on resources

26. MCORE Java-based middleware interfaces with Matlab
Manages workflow between different resources
Automatic task scheduling on remote servers
10x performance improvement
20 concurrent HiPerSAT servers

27. Papers
K. Glass, G. Frishkoff, R. Frank, C. Davey, J. Dien, A. Malony, A Framework for Evaluating ICA Methods of Artifact Removal from Multichannel EEG, ICA Conference, Grenada, Spain, 2004.
R. Frank and G. Frishkoff, APECS: A Framework for Implementation and Evaluation of Blink Extraction from Multichannel EEG, Journal of Clinical Neurophysiology, to appear, 2006.
D. Keith, C. Hoge, A. Malony, and R. Frank, “Parallel ICA Methods for EEG Neuroimaging,” International Parallel and Distributed Processing Symposium (IPDPS 2006), May 2006.
C. Hoge, D. Keith, and A. Malony, “Client-side Task Support in Matlab for Concurrent Distributed Execution,” Austrian-Hungarian Workshop on Distributed and Parallel Systems (DAPSYS), September 2006.

28. Topography of Spike–Wave Dynamics
Must observe spatial and temporal dynamics together
Seizure spike waves involve linked cortical networks
Fronto-thalamic circuit (executive control)
Limbic circuit (episodic memory)
How do we identify cortical networks?
Can only infer locations using “scalp space”
Problem of Superposition
How many sources?
Where are they located?
Must look at the brain “source space”
Must solve source localization problem

29. Brain Sources of Epileptic Seizure
Single time point source solution
Need to identify sources for each msec time sample
Visualize dynamics in “source space”

30. Computational Head Models
Source localization requires modeling
Goal: Full physics modeling of human head electromagnetics
Step 1: Head tissue segmentation
Obtain accurate tissue geometries
Step 2: Numerical forward solution
3D numerical head model
Map current sources to scalp potential
Step 3: Conductivity modeling
Inject currents and measure response
Find accurate tissue conductivities
Step 4: Source optimization

31. Brain Activity Sources in the Cortex
Scalp EEG activity is generated in the cortex
Interested in location, orientation, and magnitude
Cortical sheet gives possible locations
Orientation is normal to cortical surface
Need to capture convoluted geometry in 3D mesh
From segmented MRI/CT
Address linear superposition

32. Source Localization Mapping of scalp potentials to cortical generators
Signal decomposition (addressing superposition)
Anatomical source modeling (localization)
Source modeling
Anatomical Constraints
Accurate head model and physics
Computational head model formulation
Mathematical Constraints
Criteria (e.g., “smoothness”) to constrain solution
Current solutions limited by
Simplistic geometry
Assumptions of conductivities
because the inverse problem
anatomical constraints (head model)
mathematical constraints (mathematical constraints such as “smoothness”)

33. Neuroanatomical Segmentation (Kai Li)
Classify pixels in MR images
Extract neuroanatomical structures
Difficulties
Image artifacts
Convoluted shape of cortex
Inter-subject variations
Exploit robust a priori knowledge
Structural and geometrical
Morphological
Radiological
Develop segmentation pipeline (TAS)

34. TAS Results Outperforms all leading segmentation packages
BrainVisa, SPM5, Freesurfer, FSL
Compare with expert manual MR segmentation
Cerebral cortex and cerebral white matter

35. TAS Comparison Voxel-level comparison
Identify segmentation characteristics based on algorithms
Improvements made in TAS based on results * *

36. Head Modeling: Theoretical/Computational Flow
Continuous Solutions
Governing Equations ICS/BCS 
Finite-Difference Finite-Element Boundary-Element Finite-Volume Spectral
Discretization
System of Algebraic Equations
Discrete Nodal Values
Tridiagonal ADI SOR Gauss-Seidel Gaussian elimination 
Equation (Matrix) Solver
 (x,y,z,t) J (x,y,z,t) B (x,y,z,t)
Approximate Solution

37. Governing Equations Given positions and magnitudes of current sources
Given geometry and head volume conductivity 
Calculate distribution of electrical potential on scalp 
Solve linear Poisson equation on  in  with with no-flux Neumann boundary condition
Inverse problem uses general tomographic structure
Given , current injection configuration, forward model
Predict scalp potential, calculate error (least square norm)
Search for global minimum using non-linear optimization

38. Forward Solution Computation (ICCS 2005)
Finite difference method (FDM) representation
1-to-1 match to MRI resolution, plus air around head
Assign conductivity value to each voxel, zero in air
Alternating Direction Implicit (ADI) Method
IBM p690
C++ and OpenMP
fast matrix libraries
3-sphere approximation of human head tissues
>65% efficiency

39. Conductivity Optimization
Correct source inverse solutions depend on accurate estimates of head tissue conductivities
Scalp, skull, brain, CSF, …
Design as a conductivity search problem
Estimate conductivity values
Calculate forward solution and compare to measured
Iterate until error threshold is obtained (global minimum)
Use electrical impedance tomography methods
Multiple current injection pairs (source, sink)
Conductivity search can be parallelized
Simplex method (ICCS 2005)
Simulated annealing (ICCS 2007)
current source
current sink
measurement electrodes

40. Conductivity Search Architecture

41. Conductivity Optimization Dynamics (ICCS 2005)
Simplex search
Four tissues:
scalp
CSF
skull
brain
4-way search parallelism
8-way ADI parallelism
1mm3 resolution

42. Observation Bad conductivity solutions lead to bad source solutions
Must include skull inhomogeneity
Differences in skull anatomy (dense, marrow, sutures, ...)
Differences in conductivity
Parcel up the skull into several homogeneous parts
Assign separate conductivity to each part
Eleven major bones in skull

43. Complexity and Simplex Performance Problems
Mores tissues to model increases search complexity
Number of parameters in the inverse solver increase
Simplex search reaches performance limits quickly
5 tissues: fails to converge in a practical time
6 tissues: fails to converge at all! 1
0.8
0.6
0.4
0.2
Fraction of successful search
Number of forward soultions

44. Simulated Annealing for Conductivity Search
Allow greater number of conductivity parameters
Include skull inhomogeneity in forward/inverse solution
SA starts with control point (X0) and temperature (T0)
SA has four nested loops
Temperature reduction
Step length (Nt)
Search radius (Ns)
Control point perturbation (N = # tissues)
X = X0
T = T0
Xk = Perturb(X,k)
Fk = Cost (Xk)
SA accept/reject
k = 0..N
Update optimal
X = Xk
F = Fk
j = 0..Ns
i = 0..Nt
Adjust max step length
T = rT
check termination

45. SA Details Algorithm SA accept/reject criteria SA stopping criteria
Search around control point
Accept/reject and update
Check termination
Reduce temperature and repeat until termination
SA accept/reject criteria
Always accept moves that lower the cost function
Metropolis criteria: accept moves that have higher cost with probability based on temperature and cost size
SA stopping criteria
Epsilon (error) unchanges K temperature reductions
Number of function evaluations > maximum
Optimal solution < tolerance

46. Serial/Parallel Simulated Annealing Pseudo Code
Serial search around control point
Parallelization along the search radius
Inner loop perturbs control point dimension by dimension
Forward solution always parallelized (occurs in cost() calculation)

47. Serial Algorithm Validation and Dynamics
Preset 13 conductivity values
11 skull, CSF, scalp, brain
Validate for 11 head tissues
31 hours on IBM p655
8-way forward calculations
SA is robust for large number of tissue

48. Parallelize Intermediate Loop (search radius)
Distribute search radius loop across nodes
Up to Ns random searches in parallel (Higginson et al.)
Each task executes the inner loop to find best point
Tasks communicate to adjust the maximum step length and update the optimal point
Theoretical speedup by a factor of Ns (< 14 generally)

49. Results: Intermediate Loop Parallelization
Validate conductivity optimization for 11 head tissues
Performance results for 5 tissue types
San Diego Supercomputing Center DataStar
IBM p655 cluster
One task per 8-way node
Parallel forward calculation
Linear speedup to 96 processors

50. Papers
K. Li, A. Malony, and D. Tucker, “Automatic Brain MR Image Segmentation with Relative Thresholding and Morphological Image Analysis,” International Conference on Computer Vision Theory and Applications (VISAPP), Setúbal, Portugal, February 2006.
K. Ki, A. Malony, and D. Tucker, “A Multiscale Morphological Approach to Topology Correction of Cortical Surfaces,” International Workshop on Medical Imaging and Augmented Reality (MIAR 2006), August 2006.
A. Salman, S. Turovets, A. Malony, J. Eriksen, and D. Tucker, “Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), May (Best paper)
A. Salman, S. Turovets, A. Malony, V. Vasilov, “Multi-Cluster, Mixed-Mode Computational Modeling of Human Head Conductivity,” International Workshop on OpenMP (IWOMP), June 2005.
S. Turovets, A. Salman, A. Malony, P. Poolman, C. Davey, D. Tucker, “Anatomically Constrained Conductivity Estimation of the Human Head in Vivo: Computational Procedure and Preliminary Experiments,” Electrical Impedance Tomography (EIT), July 2006.
A. Salman, A. Malony, S. Turovets, and D. Tucker, “Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity,” International Conference on Computational Science (ICCS), May (Best paper)

51. Computational Integrated Neuroimaging System…
raw
storage
resources
virtual
services
compute resources

52. NIC Computational Services Architecture
clients

54. CIS Faculty Research Areas

55. Assistive Technology and Brain Injury Research
Technology for people with cognitive impairments
Navigation
Trimet
Multi-disciplinary research
Prof. Steve Fickas, CIS
Wearable Computing Lab
Prof. McKay Sohlberg, Education
NSF grants
CogLink, Inc.
Startup company

56. Genomics and Bioinformatics
Research in comparative genomics analyzes similarities and differences between orthologous genes
ortholog = “same word”
Zebrafish, salmon, and other teleost fish often have two orthologs of a single human gene
UO software to scan human chromosomes, identify co-orthologs in zebrafish
Studying co-orthologs improves our ability to understand functions of genes, potential medical applications
Salmon calcitonin is up to 50 times more effective than human calcitonin in treating osteoporosis

57. Computational Paleontology
Dinosaur 3D modeling
DinoMorph modeling engine
Paleontology-based
Reconstructs true dimensions, poses, flexibility, movements
Dinosaur species
Other domestic, wild, and fanciful animals
Kaibridge, Inc.
Startup company
Interactive museum exhibits
Dinosaur educational software
BBC online mystery game
Photo by Rick Edwards, AMNH, 2006

58. Neural ElectroMagnetic Ontologies (NEMO)
How can EEG data be compared across laboratories?
Need a system for representation, storage, mining, and dissemination of electromagnetic information
Need standardization of methods for measure generation and classification of information
Identification and labeling of components
Patterns of interest
NEMO will address issue by providing
Spatial and temporal ontology database
Use for data representation, mining, and meta-analysis
Components in average EEG and MEG (ERPs)
Dejing Dou (UO) and Gwen Frishkoff (Univ. Pittsburgh)

59. NEMO System Composed of three modules Methods for measure generation
Database mining
Inference engine
Query (user) interface
Methods for measure generation
Spatial & temporal ontologies
Cognitive functional mapping
User interactions
Query formulation
Mapping-rule definitions
Scalable integration system
Online repository for storing metadata
Spatio-temporal ontologies, database schema, mappings

60. Computer Science Visualization Laboratory
Support interdisciplinary computer science
Informatics
Computational science
Resource development
Phase 1 (complete)
NSF MRI grant ($1M)
ICONIC HPC Grid
Phase II
Visualization Lab ($100K)
rear projection
3D stereo and 2x2 tiled
3x4 tiled 24” LCD display
Phase III …