Parallel Simulated Anealing for Computational Modeling of Human Head Conductivity A. Salman 1, Allen D. Malony 1,2, S. Turovets 1, D. Tucker 3 Department of Computer & Information Science 1 Neuroinformatics Center 2 University of Oregon Electrical Geodesics, Inc. 3

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity2 Background: Observing Dynamic Brain Function  Brain activity occurs in cortex  Observing brain activity requires  high temporal and spatial resolution  Cortex activity generates scalp EEG  EEG data (dense-array, 256 channels)  High temporal (1msec) / poor spatial resolution (2D)  MR imaging (fMRI, PET)  Good spatial (3D) / poor temporal resolution (~1.0 sec)  Want both high temporal and spatial resolution  Need to solve source localization problem!!!  Find cortical sources for measured EEG signals

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity3 Computational Head Models  Source localization requires modeling  Goal: Full physics modeling of human head electromagnetics  Step 1: Head tissue segmentation  Obtain accurate tissue geometries  Step 2: Numerical forward solution  3D numerical head model  Map current sources to scalp potential  Step 3: Conductivity modeling  Inject currents and measure response  Find accurate tissue conductivities  Step 4: Source optimization

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity4 Brain Activity Sources in the Cortex  Scalp EEG activity is generated in the cortex  Interested in location, orientation, and magnitude  Cortical sheet gives possible locations  Orientation is normal to cortical surface  Need to capture convoluted geometry in 3D mesh  From segmented MRI/CT  Address linear superposition

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity5 Addressing Superposition: Brain Electrical Fields  Brain electrical fields are dipolar  Volume conduction through head  Depth and location indeterminacy  Highly resistive skull (CSF: skull est. from 1:40 to 1:80)  Left-hemisphere scalp field may be generated by a right-hemisphere source  Multiple sources  superposition  Radial source  Tangential sources  one and two sources  varying depths

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity6 Source Localization  Mapping of scalp potentials to cortical generators  Signal decomposition (addressing superposition)  Anatomical source modeling (localization)  Source modeling  Anatomical Constraints  Accurate head model and physics  Computational head model formulation  Mathematical Constraints  Criteria (e.g., “smoothness”) to constrain solution  Current solutions limited by  Simplistic geometry  Assumptions of conductivities

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity7 Theoretical/Computational Flow Governing Equations ICS/BCS Discretization System of Algebraic Equations Equation (Matrix) Solver Approximate Solution Continuous Solutions Finite-Difference Finite-Element Boundary-Element Finite-Volume Spectral Discrete Nodal Values Tridiagonal ADI SOR Gauss-Seidel Gaussian elimination  (x,y,z,t) J (x,y,z,t) B (x,y,z,t)

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity8 Governing Equations  Given positions and magnitudes of current sources  Given geometry and head volume conductivity   Calculate distribution of electrical potential on scalp    Solve linear Poisson equation on   in  with with no-flux Neumann boundary condition  Inverse problem uses general tomographic structure  Given , current injection configuration, forward model  Predict scalp potential, calculate error (least square norm)  Search for global minimum using non-linear optimization

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity9 Forward Solution Computation (ICCS 2005)  Finite difference method (FDM) representation  1-to-1 match to MRI resolution, plus air around head  Assign conductivity value to each voxel, zero in air  Alternating Direction Implicit (ADI) Method 3-sphere approximation of human head tissues IBM p690 C++ and OpenMP fast matrix libraries >65% efficiency

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity10 Conductivity Optimization  Correct source inverse solutions depend on accurate estimates of head tissue conductivities  Scalp, skull, brain, CSF, …  Design as a conductivity search problem  Estimate conductivity values  Calculate forward solution and compare to measured  Iterate until error threshold is obtained (global minimum)  Use electrical impedance tomography methods  Multiple current injection pairs (source, sink)  Conductivity search can be parallelized  Simplex method (ICCS 2005)  Simulated annealing (ICCS 2007) current source current sink measurement electrodes

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity11 Conductivity Optimization Dynamics (ICCS 2005)  Simplex search  Four tissues:  scalp  CSF  skull  brain  4-way search parallelism  8-way ADI parallelism 1mm 3 resolution

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity12 Observation  Bad conductivity solutions lead to bad source solutions  Must include skull inhomogeneity  Differences in skull anatomy (dense, marrow, sutures,...)  Differences in conductivity  Parcel up the skull into several homogeneous parts  Assign separate conductivity to each part  Eleven major bones in skull

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity13 Complexity and Simplex Performance Problems  Mores tissues to model increases search complexity  Number of parameters in the inverse solver increase  Simplex search reaches performance limits quickly  5 tissues: fails to converge in a practical time  6 tissues: fails to converge at all! Fraction of successful search Number of forward soultions

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity14 Simulated Annealing for Conductivity Search  Allow greater number of conductivity parameters  Include skull inhomogeneity in forward/inverse solution  SA starts with control point (X 0 ) and temperature (T 0 )  SA has four nested loops  Temperature reduction  Step length (N t )  Search radius (N s )  Control point perturbation (N = # tissues) X = X 0 T = T 0 X k = Perturb(X,k) F k = Cost (X k ) SA accept/reject Update optimal X = X k F = F k Adjust max step length T = rT check termination i = 0..N t j = 0..N s k = 0..N

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity15 SA Details  Algorithm  Search around control point  Accept/reject and update  Check termination  Reduce temperature and repeat until termination  SA accept/reject criteria  Always accept moves that lower the cost function  Metropolis criteria: accept moves that have higher cost with probability based on temperature and cost size  SA stopping criteria  Epsilon (error) unchanges K temperature reductions  Number of function evaluations > maximum  Optimal solution < tolerance

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity16 Serial/Parallel Simulated Annealing Pseudo Code Serial search around control point Parallelization along the search radius Forward solution always parallelized (occurs in cost() calculation) Inner loop perturbs control point dimension by dimension

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity17 Serial Algorithm Validation and Dynamics  Preset 13 conductivity values  11 skull, CSF, scalp, brain  Validate for 11 head tissues  31 hours on IBM p655  8-way forward calculations  SA is robust for large number of tissue

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity18 Parallelize Intermediate Loop (search radius)  Distribute search radius loop across nodes  Up to N s random searches in parallel (Higginson et al.)  Each task executes the inner loop to find best point  Tasks communicate to adjust the maximum step length and update the optimal point  Theoretical speedup by a factor of N s (< 14 generally)

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity19 Results: Intermediate Loop Parallelization  Validate conductivity optimization for 11 head tissues  Performance results for 5 tissue types  San Diego Supercomputing Center DataStar  IBM p655 cluster  One task per 8-way node  Parallel forward calculation  Linear speedup to 96 processors

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity20 Parallelize Intermediate and Inner loop  Check all possible control point perturbations  Generate all possible points and compute cost function  Apply the SA criteria when selecting a possible path  Select best point from all acceptable paths  Theoretical speedup by factor of N for 2 N CPUs  Potential convergence speedup with no quality effect

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity21 Results: Inner Loop Parallelization  Experimental setup:  Three tissues (N=3)  One intermediate (search radius) SA task (N S = 1)  Two CPUs for the forward calculations  Execute serial and parallel algorithms with same parameters  Serial (2 CPUs for forward calculation)  Execution time: sec  Convergence : 600 (inner loop calculations)  Parallel inner loop (2 N – 1 CPUs per perturbation, x2 forward)  Exec Time: sec  Convergence: 552 (inner loop calculations)  Speedup:  Due to convergence: %  Due to speeding the calculations: (N=3 predicted)  Total Speedup: 1.93

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity22 Summary  Simulated annealing algorithm proved to be stable in extracting up to 13 tissues  Parallel SA is robust and achieves good speedups  We were able to extract 5 tissues in 40 minutes  The overall potential speedup is: S = S s * S n * S c * S f  S s – intermediate loop speedup  S n – inner loop speedup  S c – convergence rate speedup  S f – forward calculation speedup  For 5 tissues with 96*32 (3072) CPUs  ~ 12 * 5 * 1.08 * 8 ~ 500 times

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity23 Future Work  Conductivity modeling accuracy depends on:  Head tissues geometry model  MRI/CT quality and segmentation  Head tissues conductivity model  skull inhomogeneity, skull anisotropy, sutures, …  Experimental measurements  Numerical solvers and parameters  Study the solution sensitivity to the models and parameters?  Multi-disciplinary computational environment  Compare with other approaches  DTMRI, MEG, in vivo measurements  Interdisciplinary integration

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity24 NeuroInformatics Center (NIC) at UO  Computational science applied to human neuroscience  Tools to help understand dynamic brain function  Tools to help diagnosis brain-related disorders  HPC simulation, large-scale data analysis, visualization  Integrated neuroimaging methods and technology  Coupled analysis and modeling  Advanced statistical signal analysis  Computational head/brain models  Source localization methods  Optimize temporal and spatial resolution  Internet telemedine and medical services  brain analysis services, data archiving, and data mining

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity25 NIC Scientists and Projects  Sergei Turovets, Ph.D., Physics Physics models for head electromagnetics / optics  Adnan Salman, Ph.D. student, Computer Science Computational modeling of head conductivity  Kai Li, Ph.D., Computer Science MRI segmentation and cortical surface extraction  Bob Frank, M.S., Mathematics Physiological signal and brain EEG analysis  Middleware for distributed tasks workflows

ICCS 2007Parallel Simulated Annealing for Computational Modeling of Human Head Conductivity26 Electrical Geodesics Inc. (EGI)  EGI Geodesics Sensor Net  Dense-array sensor technology  64/128/256 channels  256-channel GSN  AgCl plastic electrodes  Net Station  Advanced EEG/ERP data analysis and visualization  Stereotactic EEG sensor registration  State of the art technology  Research / clinical products  EGI/CDS medical services