Distributed Computational Architectures for Integrated Time-Dynamic Neuroimaging Dr. Allen D. Malony malony@cs.uoregon.edu Computer & Information Science Department Computational Science Institute CIBER University of Oregon

Who Am I Associate Professor, CIS Department, UO Computer Science specialties / interests parallel performance analysis (primary) environments computational science (secondary) software development environments distributed and parallel computing environment Cognitive Neuroscience interests two-year association with Don Tucker (Psychology, UO) Carmel Neuroinformatics workshop (2000, presentation) HBP Neuroinformatics Review Panel (2000, 2001) HBP Annual Meeting (2000, presentation) September 21, 2018 Hill Center

Talk Outline Computational science and cognitive neuroscience Brain dynamics analysis problem (my view) integrated electromagnetic analysis system Motivating case studies observations: computation and informatics Computational architectures models and technology key ideas Opportunities and the Neural Informatics Center Final Thoughts September 21, 2018 Hill Center

Computational Science & Cognitive Neuroscience Computational methods applied to scientific research high-performance simulation of complex phenomena large-scale data analysis and visualization Understand functional activity of the human cortex multiple cognitive domains multiple experimental paradigms and methods Need for coupled/integrated modeling and analysis electrical and magnetic, cortical and theoretical Need for robust tools: computational & informatic Problem solving environment for brain analysis September 21, 2018 Hill Center

Brain Dynamics Analysis Problem (My View) Identify functional components in cognitive contexts Interpret with respect to cognitive theoretical models Requirements: spatial (structure), temporal (activity) Imaging techniques for analyzing brain dynamics blood flow neuroimaging (PET, fMRI) good spatial resolution  functional brain mapping temporal limitations to tracking of dynamic activities electromagnetic measures (EEG/ERP, MEG) msec temporal resolution to distinguish components spatial resolution sub-optimal (source localization) potential to map electrical activity to cortex surface September 21, 2018 Hill Center

Electromagnetic Analysis Methodology Multi-trial analysis signal analysis and response analysis averaging across subjects and trials (S/N ratio) distortion (smearing) of estimated source response noise artifacts, signal variation (individuals, trials) improvements: artifact removal, selective averaging create component response models ERP identification factor analysis: PCA, ICA, … error in source factors: variability, statistics Multi-subject and single-subject analysis quantify differences of individual from population September 21, 2018 Hill Center

Single-Trial Analysis Capability Improve fidelity of single-subject response model higher information content than multi-trial/subject reduce analysis error from trial/subject variability knowledge of subject population, stimulus deviations Diagnosis (identification) of cognitive state known stimulus blind stimulus match response to known component response model Problems greater noise greater complexity September 21, 2018 Hill Center

Single-Trial Analysis Methodology Integrate methods for analyzing brain dynamics Improve resolution and robustness of techniques increase measurement density (128 to 256 channels) Coupled modeling: constraints and cross-validation component response model  cortical activity model tuned models for single individual Build models in experimental paradigm context Match single-trial measurements to models known stimulus  multiple trial models blind stimulus  multiple stimulus/trial models Training and learning September 21, 2018 Hill Center

Integrated Electromagnetic Brain Analysis Cortical Activity Knowledge Base Head Analysis Source Analysis Structural / Functional MRI spatial pattern recognition temporal dynamics Cortical Activity Model Experiment subject Constraint Analysis Single-trial Analysis EEG MEG Component Response Model neural constraints Dense Array EEG / MEG temporal pattern recognition Signal Analysis Response Analysis Component Response Knowledge Base September 21, 2018 Hill Center

Integrated Electromagnetic Analysis System Carmel Workshop

Case Study: Readiness Potential Self-paced button pressing task slow negative shifts in potential contralateral to hand Single subject examination multi-trial (150 trials) averaged ERP analysis Dense-array scalp electrical measurement 129 electrode array (EGI Geodesic Sensor Net) Modeling of brain electrical activity MRI and CT data analysis with tissue segmentation realistic boundary element meshes (2K ’s for brain) source localization with dipole modeling Past studies (Deeke & Kornhuber 1969, Libet 1985, etc.) have shown that self-paced button pressing is preceded by slow negative shifts in potential (the bereitschafts or readiness potential) recorded by scalp electrodes contralateral to the hand that is pushing the button. In this study, we examine this phenomenon in a single subject, modeling the electrical activity of the brain and scalp. The electrical activity of the scalp was recorded using a dense array of 129 electrodes, and modeled using realistic boundary element meshes based on the MRI and CT data of the subject. Can ERP analysis accurately localize cortical activity? September 21, 2018 Hill Center

Experimental Methodology 16x256 bits per microsec (30MB/m) CT / MRI EEG segmented tissues NetStation processed EEG BrainVoyager mesh generation, source localization constrained to cortical surface Constraining the source solutions to the cortical surface is a major advantage for analyzing EEG and ERP effects. A working assumption is that sources are likely to be oriented normal to the cortical surface. Interpolator 3D EMSE September 21, 2018 Hill Center

Electrical Activity of Scalp and Brain Expected brain activity Correlated with fMRI experimental studies Topographic and cortex mapped spatial analysis Lateralize Readiness Potential (LRP) -404 ms -56 ms 0 ms 160 ms September 21, 2018 Hill Center

Optimizing Spatial Resolution for ERP Adequate spatial sampling Accurate head surface mapping Accurate sensor registration Measured skull conductivity Convergence with MEG  MEG-compatible EEG Convergence with fMRI  fMRI-compatible EEG Test spatial resolution with know pathological sources EEG as link for converging analysis? What problems exist? • Adequate spatial sampling requires enough channels to cover the head surface with a density equal to the spatial Nyquist. This is the discretization rate required to capture the highest spatial frequency or variation in electrical potential over distance. Current estimates suggest 256 to 512 sensors will be necessary. • Accurate sensor registration with structural MR is accomplished by 3D localization, such as with a magnetic (Polhemus) digitizer. Because this method is tedious and subject to interference, EGI is developing a geodesic photogrammetry method. • Head surface maps, which must be animations to capture time variation, only become accurate with 64 and more channels evenly distributed over the head surface, because this is the number required to estimate the zero surface integral to produce an unbiased reference. • Conductivity varies widely over the skull, such that estimates from thickness are misleading. EGI is developing a scanning current injection method. Convergence with MEG allows an independent test of this measurement. • Functionally important electrical effects may not produce BOLD or blood flow changes. Conversely, far field electrical effects may not be created by important events of brain activation. September 21, 2018 Hill Center

Electrical Impedance Tomography Small (10µA) currents are injected between electrode pair Resulting potential is measured from all remaining electrodes Measures used to estimate conductivity of each tissue compartment Boundary element forward solution 4-shell polyhedron model (1280 faces) direct (31244 sec) and iterative approaches (933 sec) Finite element forward solution greater computational requirements September 21, 2018 Hill Center

Case Study: Self-Monitored Motivated Action Learning task with feedback (Gehring et al. (1993)) left- or right-hand button press response "incorrect" feedback on error "OK" or “late” feedback if correct timed expectancy and motivated response Error-Related Negativity (ERN) large medial negative response on error self-monitoring when motivated action goes wrong What is the nature and complexity of the ERN with respect to dynamic components of brain activity? September 21, 2018 Hill Center

Cognitive Experiments and Brain Dynamics Visualize the dynamic operations of brain Example: fMRI blood flow response to reading a word Dense-array EEG / MEG frontal lobe activity (ERN) significant changes in milliseconds frontal oscillations and separate time courses BrainVoyager September 21, 2018 Hill Center

ERN Analysis using ICA (Makeig, Salk Institute) Average analysis smears temporal/spatial dynamics Single-trial analysis may expose greater detail Independent Components Analysis (ICA) find independent EEG component contributors temporal and spatial components accounting for artifacts components accounting for functional sources (ERN) analysis over single trials Two components account for averaged ERN response-locked ERN difference wave dominated show temporal and functional independence September 21, 2018 Hill Center

ERP and Component Envelopes (Left/Correct) Complementary behavior Both active at strongest ERN channels September 21, 2018 Hill Center

ERPs averaged across response hand Neither C2 nor C7 explain the waveforms Component sum does explain the waveforms and shows ERN response September 21, 2018 Hill Center

Topographic Imaging and Dipole Modeling Component #2 Component #7 Averaged ERN Brain Electrical Source Analysis (BESA) September 21, 2018 Hill Center

ICA Component #2 Dynamics Stimulus locked Memory of deadline September 21, 2018 Hill Center

ICA Component #7 Dynamics Phase reset by response, largest after incorrect September 21, 2018 Hill Center

Optimize Temporal Information Inherent problem – both electrical and magnetic Trial averaging methodologies can mask dynamics Techniques to boost signal to noise ratio Selective averaging Stimulus and response locking Techniques to estimate time function fMRI timing models EEG/MEG time function for fMRI signal extraction Single trial analysis with individual modeling What problems exist? September 21, 2018 Hill Center

Case Study Observations Diverse set of tools function and implementation separate tools (monolithic) and not integrated incompatibilities and limitations for interoperation Complex analysis processes multiple processes applied (process pipeline) high-level, hierarchical process methodology scientific discovery through integrated techniques heterogeneous, flexible, extensible capabilities increasingly high computational demands Multiple, interdisciplinary scientific domains September 21, 2018 Hill Center

High-Performance Computational Environments Integrated database, analysis, and visualization Distributed tool infrastructure diverse tools across multiple platforms interoperation requirements user interaction requirements support portability, flexibility, extensibility Scalable, high-performance parallel computing increase data resolution minimize solution time High-level access to tools web-based access September 21, 2018 Hill Center

Computational Systems: Models and Technology Domain-specific, problem-specific environments (PSE) TIERRA Scientific “workbench” SCIRun Programming environments numerical frameworks POOMA application coupling PVM / MPI CUMULVS PAWS SILOON / PDT Metacomputing / GRID Legion Globus Heterogeneous distributed computing / coupling NetSolve INTERLACE HARNESS Web-based environments ViNE PUNCH VNC September 21, 2018 Hill Center

TIERRA (Computational Science Institute, UO) Tomographic Imaging Environment for Ridge Research and Analysis High-performance, domain-specific environment for seismis tomography parallelized tomography code runtime distributed array access computational steering via MatLab frontend full problem solving process for seismic tomography Led to new discoveries for three-dimensional melt migration beneath the East Pacific Rise September 21, 2018 Hill Center

KEY IDEAS TIERRA Architecture Domain specific Support for the entire process September 21, 2018 Hill Center

SCIRun (Johnson, University of Utah) Scientific programming environment large-scale simulations “computational workbench” visual programming interface dataflow model of computing modules: operation or algorithm with I/O ports network: set of modules and their interconnections widgets: 3D user interaction data types: Mesh, Surface, Matrix, Field, Geometry extensible module library computational steering September 21, 2018 Hill Center

SCIRun User Interface Visual programming lets users select, arrange, and connect modules into a desired network Interactive steering of design, computation, and visualization allows more rapid convergence September 21, 2018 Hill Center

ICA for EEG Source Localization with SCIRun PCA decomposition for EEG signal/noise subspaces ICA activity map separation on signal subspace Solution to a single dipole source forward problem underlying model is shown in the MRI planes dipole source is indicated by red and blue spheres electric field visualized by cropped scalp potential map and wire-frame equipotential isosurface KEY IDEAS Integrated application development environment “Component-based” application programming High-level data objects September 21, 2018 Hill Center

POOMA (Advanced Computing Lab, LANL) Parallel Object-Oriented Methods and Applications Goals use object-oriented programming to help manage complexity of modern scientific simulation codes extract physics content of simulations from details of parallel, high-performance computing framework approach: allows flexible code structure, object reuse across problem domains build upon standards to maintain code portability An object-oriented framework for scientific computing applications on parallel computers September 21, 2018 Hill Center

POOMA Approach C++ class library Generic programming high-level, generally data-parallel API Generic programming classes modeled after STL style heavy use of C++ templates Parallelism encapsulated message-passing for distributed memory machines multi-threaded shared memory (POOMA II) Cross platform code development and scalable parallelism September 21, 2018 Hill Center

KEY IDEAS POOMA Framework Numerical programming framework Compile-time Polymorphism Computer Science Physics Application Algorithm Local Parallel Global STL Expression Templates Domain Decomposition Message Passing Load Balancing Fields Meshes Particles Interpolators FFT Differential Operators MC++ NTTP LINAC KEY IDEAS Numerical programming framework Encapsulated parallelism High-level API’s / data support September 21, 2018 Hill Center

PDT (Malony, University of Oregon) Program Database Toolkit Program analysis multi-language (Fortran, C, C++, Java) commercial- grade parsers IL to program database (PDB) API for PDB access / query Tools: instrumentation, code wrapping, documentation September 21, 2018 Hill Center

SILOON (Advanced Computing Lab, LANL; UO) Scripting Interface Language for OO Numerics Toolkit and run-time support for building easy-to-use external interfaces to existing numerical codes Scripting language to “glue” components together KEY IDEAS Support for application interaction control Support for application code wrapping Application / tool coupling Data exchange support September 21, 2018 Hill Center

Metasystems and Metacomputing Many resources accessible on the internet computers, data, devices, people Extend single system model to internet domain wide-area (department, campus, region, country) scalable, transparent access to resources hides network complexity (“as if on your machine”) Extend computing model to internet domain shared persistent space of objects (data, execution) heterogeneous distributed and parallel processing meta-applications (multi-component, hierarchical) Deal with complex environment / primitive tools September 21, 2018 Hill Center

“The GRID” New applications based on high-speed coupling of people, computers, databases, instruments, ... computer-enhanced instruments collaborative engineering browsing of remote datasets use of remote software data-intensive computing very large-scale simulation large-scale parameter studies September 21, 2018 Hill Center

GRID Architectural Picture KEY IDEAS Metasystems infrastructure / services Metacomputing applications programming GRID resources September 21, 2018 Hill Center

NetSolve (Dongarra, University of Tennessee) Client-server system to access distributed computational / DB HW/SW resources Distributed computing: resources, processes, data, users Load-balancing policy for efficiency / performance Integration with arbitrary software components C, Fortran, Java, MatLab, Mathematica, Excel BLAS, (Sca)LAPACK, MINPACK, FFTPACK September 21, 2018 Hill Center

NetSolve Usage “Blue collar” GRID-based computing Scenarios users can set things up (without “su” privileges) no deep network programming knowledge required Scenarios clients, servers, and agents anywhere on Internet clients, servers, and agents on an Intranet clients, servers, and agent on the same machine Focus on MATLAB users OO-style language (objects are matrices) one of most popular desktop systems for numerical computing (> 400K users) September 21, 2018 Hill Center

NetSolve – The Client NetSolve API hides complexity of numerical software Computation is location transparent Provides access to virtual libraries: Component GRID-based framework Central management of library resources User not concerned with most up-to-date versions Automatic tie to Netlib repository Synchronous or asynchronous calls User-level parallelism September 21, 2018 Hill Center

NetSolve – The Agent and Server gateway to computational services performs load balancing and resource management Server various software installed on various hardware configurable and extendable framework to easily add software many numerical libraries being integrated supports parallel computing September 21, 2018 Hill Center

MCell (Bartol, Salk Institute; Salpeter, Cornell) Monte Carlo simulator of cellular microphysiology Study how neurotransmitters diffuse and activate receptors in synapses between different cells NetSolve distributes processing workload and allows access to computational resources Simultaneous evaluation of large number of different parameter combinations September 21, 2018 Hill Center

INTERLACE (Malony, University of Oregon) INTERoperation and Linking Architecture for Computational Engines Goals framework for building high-performance computing environments from existing tools reusable components in heterogeneous environment abstract connection mechanisms for control/data flow resource management for dynamic operation use standard software technologies parallel and distributed computational environments http://www.cs.uoregon.edu/research/paracomp/proj/interlace/ September 21, 2018 Hill Center

KEY IDEAS INTERLACE Components Computational engines: libraries or programs providing specific functions Computational server: program interfacing multiple engines with middleware Wrappers: server-engine interface for data/control Middleware: server-to-server interoperation software KEY IDEAS High-level numeric computational services Access to metasystem resources Wrapping/linking of computational engines Dynamic, adaptable, extensible High-level metasystems programming support September 21, 2018 Hill Center

ViNE (Malony, University of Oregon) Virtual Notebook Environment High-level, shared notebooks, data, and tools in distributed, heterogenous system Architecture leaves: notebook functions and data stems: notebook communication Web-based access September 21, 2018 Hill Center

ViNE Experiment Builder List of available, named data, tools, and experiments Visual dataflow model of experiment process Wrapped tools and databases wrapped MATLAB “tool” September 21, 2018 Hill Center

Brain Electrophysiology Lab Notebook Dense array EEG datasets Commercial of the shelf statistical and numerical packages Multiple machines types Notebook content automatically generated from experiment results September 21, 2018 Hill Center

PUNCH Purdue University Network-Computing Hubs Educational and research computing “portals” across the Purdue “enterprise” with affiliated institutions Resource sharing by Purdue users computers, software, laboratory equipment educational materials Distance education allows sharing of courses and instructors Collaborative research September 21, 2018 Hill Center

PUNCH – User’s and Developer’s View Set of network-based laboratories that provide software tools for various fields Specialized WWW-server interfaces WWW-browsers access software and download data run tools and view results Tool specification Virtual laboratory development environment September 21, 2018 Hill Center

PUNCH Web Page Hubs September 21, 2018 Hill Center

PUNCH Tool Display Support via VNC X Windows display KEY IDEAS Web-based access to tools Web-based applications development Web-based data, results, process management MATLAB command window MATLAB interactive window MATLAB graphics window September 21, 2018 Hill Center

Opportunities and the Neural Informatics Center Integrated time-dynamic neuroimaging poses methodological, computational, and informatic challenges Apply computer science technology to create problem solving environment for brain analysis neuroscientist defines methods and processes add value to environment through its use Neural Informatics Center (NIC) within BBMI focus on single trial analysis problem advanced EEG/ERP analysis and integrated fMRI BEM/FEM brain models (EEG, CT, MRI) September 21, 2018 Hill Center

Final Thoughts Enable high-level problem solving environments Tools to enable scientists to compose solutions from a set of building blocks Seamless access to local and remote resources Enabling infrastructure framework standards and interfaces implementations of reusable components Collaboration environments Future Neural Informatics Grid September 21, 2018 Hill Center