1. Distributed Computational Architectures for Integrated Time-Dynamic Neuroimaging
Dr. Allen D. Malony
Computer & Information Science Department
Computational Science Institute
CIBER
University of Oregon

2. Who Am I Associate Professor, CIS Department, UO
Computer Science specialties / interests
parallel performance analysis (primary)
environments computational science (secondary)
software development environments
distributed and parallel computing environment
Cognitive Neuroscience interests
two-year association with Don Tucker (Psychology, UO)
Carmel Neuroinformatics workshop (2000, presentation)
HBP Neuroinformatics Review Panel (2000, 2001)
HBP Annual Meeting (2000, presentation)
September 21, 2018
Hill Center

3. Talk Outline Computational science and cognitive neuroscience
Brain dynamics analysis problem (my view)
integrated electromagnetic analysis system
Motivating case studies
observations: computation and informatics
Computational architectures
models and technology
key ideas
Opportunities and the Neural Informatics Center
Final Thoughts
September 21, 2018
Hill Center

4. Computational Science & Cognitive Neuroscience
Computational methods applied to scientific research
high-performance simulation of complex phenomena
large-scale data analysis and visualization
Understand functional activity of the human cortex
multiple cognitive domains
multiple experimental paradigms and methods
Need for coupled/integrated modeling and analysis
electrical and magnetic, cortical and theoretical
Need for robust tools: computational & informatic
Problem solving environment for brain analysis
September 21, 2018
Hill Center

5. Brain Dynamics Analysis Problem (My View)
Identify functional components in cognitive contexts
Interpret with respect to cognitive theoretical models
Requirements: spatial (structure), temporal (activity)
Imaging techniques for analyzing brain dynamics
blood flow neuroimaging (PET, fMRI)
good spatial resolution  functional brain mapping
temporal limitations to tracking of dynamic activities
electromagnetic measures (EEG/ERP, MEG)
msec temporal resolution to distinguish components
spatial resolution sub-optimal (source localization)
potential to map electrical activity to cortex surface
September 21, 2018
Hill Center

6. Electromagnetic Analysis Methodology
Multi-trial analysis
signal analysis and response analysis
averaging across subjects and trials (S/N ratio)
distortion (smearing) of estimated source response
noise artifacts, signal variation (individuals, trials)
improvements: artifact removal, selective averaging
create component response models
ERP identification
factor analysis: PCA, ICA, …
error in source factors: variability, statistics
Multi-subject and single-subject analysis
quantify differences of individual from population
September 21, 2018
Hill Center

7. Single-Trial Analysis Capability
Improve fidelity of single-subject response model
higher information content than multi-trial/subject
reduce analysis error from trial/subject variability
knowledge of subject population, stimulus deviations
Diagnosis (identification) of cognitive state
known stimulus
blind stimulus
match response to known component response model
Problems
greater noise
greater complexity
September 21, 2018
Hill Center

8. Single-Trial Analysis Methodology
Integrate methods for analyzing brain dynamics
Improve resolution and robustness of techniques
increase measurement density (128 to 256 channels)
Coupled modeling: constraints and cross-validation
component response model  cortical activity model
tuned models for single individual
Build models in experimental paradigm context
Match single-trial measurements to models
known stimulus  multiple trial models
blind stimulus  multiple stimulus/trial models
Training and learning
September 21, 2018
Hill Center

9. Integrated Electromagnetic Brain Analysis
Cortical Activity
Knowledge Base
Head Analysis
Source Analysis
Structural /
Functional
MRI
spatial pattern
recognition
temporal
dynamics
Cortical
Activity Model
Experiment
subject
Constraint
Analysis
Single-trial
Analysis
EEG
MEG
Component
Response Model
neural
constraints
Dense
Array EEG / MEG
temporal pattern
recognition
Signal Analysis
Response Analysis
Component Response
Knowledge Base
September 21, 2018
Hill Center

10. Integrated Electromagnetic Analysis System
Carmel Workshop

11. Case Study: Readiness Potential
Self-paced button pressing task
slow negative shifts in potential contralateral to hand
Single subject examination
multi-trial (150 trials) averaged ERP analysis
Dense-array scalp electrical measurement
129 electrode array (EGI Geodesic Sensor Net)
Modeling of brain electrical activity
MRI and CT data analysis with tissue segmentation
realistic boundary element meshes (2K ’s for brain)
source localization with dipole modeling
Past studies (Deeke & Kornhuber 1969, Libet 1985, etc.) have shown that self-paced button pressing is preceded by slow negative shifts in potential (the bereitschafts or readiness potential) recorded by scalp electrodes contralateral to the hand that is pushing the button. In this study, we examine this phenomenon in a single subject, modeling the electrical activity of the brain and scalp. The electrical activity of the scalp was recorded using a dense array of 129 electrodes, and modeled using realistic boundary element meshes based on the MRI and CT data of the subject.
Can ERP analysis accurately localize cortical activity?
September 21, 2018
Hill Center

12. Experimental Methodology
16x256 bits per microsec
(30MB/m)
CT / MRI
EEG
segmented tissues
NetStation
processed EEG
BrainVoyager
mesh generation,
source localization
constrained to
cortical surface
Constraining the source solutions to the cortical surface is a major advantage for analyzing EEG and ERP effects. A working assumption is that sources are likely to be oriented normal to the cortical surface.
Interpolator 3D
EMSE
September 21, 2018
Hill Center

13. Electrical Activity of Scalp and Brain
Expected brain activity
Correlated with fMRI experimental studies
Topographic and cortex mapped spatial analysis
Lateralize Readiness Potential (LRP)
-404 ms
-56 ms
0 ms
160 ms
September 21, 2018
Hill Center

14. Optimizing Spatial Resolution for ERP
Adequate spatial sampling
Accurate head surface mapping
Accurate sensor registration
Measured skull conductivity
Convergence with MEG  MEG-compatible EEG
Convergence with fMRI  fMRI-compatible EEG
Test spatial resolution with know pathological sources
EEG as link for converging analysis?
What problems exist?
• Adequate spatial sampling requires enough channels to cover the head surface with a density equal to the spatial Nyquist. This is the discretization rate required to capture the highest spatial frequency or variation in electrical potential over distance. Current estimates suggest 256 to 512 sensors will be necessary.
• Accurate sensor registration with structural MR is accomplished by 3D localization, such as with a magnetic (Polhemus) digitizer. Because this method is tedious and subject to interference, EGI is developing a geodesic photogrammetry method.
• Head surface maps, which must be animations to capture time variation, only become accurate with 64 and more channels evenly distributed over the head surface, because this is the number required to estimate the zero surface integral to produce an unbiased reference.
• Conductivity varies widely over the skull, such that estimates from thickness are misleading. EGI is developing a scanning current injection method. Convergence with MEG allows an independent test of this measurement.
• Functionally important electrical effects may not produce BOLD or blood flow changes. Conversely, far field electrical effects may not be created by important events of brain activation.
September 21, 2018
Hill Center

15. Electrical Impedance Tomography
Small (10µA) currents are injected between electrode pair
Resulting potential is measured from all remaining electrodes
Measures used to estimate conductivity of each tissue compartment
Boundary element forward solution
4-shell polyhedron model (1280 faces)
direct (31244 sec) and iterative approaches (933 sec)
Finite element forward solution
greater computational requirements
September 21, 2018
Hill Center

16. Case Study: Self-Monitored Motivated Action
Learning task with feedback (Gehring et al. (1993))
left- or right-hand button press response
"incorrect" feedback on error
"OK" or “late” feedback if correct
timed expectancy and motivated response
Error-Related Negativity (ERN)
large medial negative response on error
self-monitoring when motivated action goes wrong
What is the nature and complexity of the ERN with respect to dynamic components of brain activity?
September 21, 2018
Hill Center

17. Cognitive Experiments and Brain Dynamics
Visualize the dynamic operations of brain
Example: fMRI blood flow response to reading a word
Dense-array EEG / MEG frontal lobe activity (ERN)
significant changes in milliseconds
frontal oscillations and separate time courses
BrainVoyager
September 21, 2018
Hill Center

18. ERN Analysis using ICA (Makeig, Salk Institute)
Average analysis smears temporal/spatial dynamics
Single-trial analysis may expose greater detail
Independent Components Analysis (ICA)
find independent EEG component contributors
temporal and spatial
components accounting for artifacts
components accounting for functional sources (ERN)
analysis over single trials
Two components account for averaged ERN
response-locked ERN difference wave dominated
show temporal and functional independence
September 21, 2018
Hill Center

19. ERP and Component Envelopes (Left/Correct)
Complementary
behavior
Both active at strongest ERN channels
September 21, 2018
Hill Center

20. ERPs averaged across response hand
Neither C2
nor C7 explain
the waveforms
Component sum
does explain the waveforms and shows ERN response
September 21, 2018
Hill Center

21. Topographic Imaging and Dipole Modeling
Component #2
Component #7
Averaged ERN
Brain Electrical
Source Analysis
(BESA)
September 21, 2018
Hill Center

22. ICA Component #2 Dynamics
Stimulus locked
Memory of deadline
September 21, 2018
Hill Center

23. ICA Component #7 Dynamics
Phase reset by response, largest after incorrect
September 21, 2018
Hill Center

24. Optimize Temporal Information
Inherent problem – both electrical and magnetic
Trial averaging methodologies can mask dynamics
Techniques to boost signal to noise ratio
Selective averaging
Stimulus and response locking
Techniques to estimate time function
fMRI timing models
EEG/MEG time function for fMRI signal extraction
Single trial analysis with individual modeling
What problems exist?
September 21, 2018
Hill Center

25. Case Study Observations
Diverse set of tools
function and implementation
separate tools (monolithic) and not integrated
incompatibilities and limitations for interoperation
Complex analysis processes
multiple processes applied (process pipeline)
high-level, hierarchical process methodology
scientific discovery through integrated techniques
heterogeneous, flexible, extensible capabilities
increasingly high computational demands
Multiple, interdisciplinary scientific domains
September 21, 2018
Hill Center

26. High-Performance Computational Environments
Integrated database, analysis, and visualization
Distributed tool infrastructure
diverse tools across multiple platforms
interoperation requirements
user interaction requirements
support portability, flexibility, extensibility
Scalable, high-performance parallel computing
increase data resolution
minimize solution time
High-level access to tools
web-based access
September 21, 2018
Hill Center

27. Computational Systems: Models and Technology
Domain-specific, problem-specific environments (PSE)
TIERRA
Scientific “workbench”
SCIRun
Programming environments
numerical frameworks
POOMA
application coupling
PVM / MPI
CUMULVS
PAWS
SILOON / PDT
Metacomputing / GRID
Legion
Globus
Heterogeneous distributed computing / coupling
NetSolve
INTERLACE
HARNESS
Web-based environments
ViNE
PUNCH
VNC
September 21, 2018
Hill Center

28. TIERRA (Computational Science Institute, UO)
Tomographic Imaging Environment for Ridge Research and Analysis
High-performance, domain-specific environment for seismis tomography
parallelized tomography code
runtime distributed array access
computational steering via MatLab frontend
full problem solving process for seismic tomography
Led to new discoveries for three-dimensional melt migration beneath the East Pacific Rise
September 21, 2018
Hill Center

29. KEY IDEAS TIERRA Architecture Domain specific
Support for the entire process
September 21, 2018
Hill Center

30. SCIRun (Johnson, University of Utah)
Scientific programming environment
large-scale simulations
“computational workbench”
visual programming interface
dataflow model of computing
modules: operation or algorithm with I/O ports
network: set of modules and their interconnections
widgets: 3D user interaction
data types: Mesh, Surface, Matrix, Field, Geometry
extensible module library
computational steering
September 21, 2018
Hill Center

31. SCIRun User Interface
Visual programming lets users select, arrange, and connect modules into a desired network
Interactive steering of design, computation, and visualization allows more rapid convergence
September 21, 2018
Hill Center

32. ICA for EEG Source Localization with SCIRun
PCA decomposition for EEG signal/noise subspaces
ICA activity map separation on signal subspace
Solution to a single dipole source forward problem
underlying model is shown in the MRI planes
dipole source is indicated by red and blue spheres
electric field visualized by cropped scalp potential map and wire-frame equipotential isosurface
KEY IDEAS
Integrated application development environment
“Component-based” application programming
High-level data objects
September 21, 2018
Hill Center

33. POOMA (Advanced Computing Lab, LANL)
Parallel Object-Oriented Methods and Applications
Goals
use object-oriented programming to help manage complexity of modern scientific simulation codes
extract physics content of simulations from details of parallel, high-performance computing
framework approach: allows flexible code structure, object reuse across problem domains
build upon standards to maintain code portability
An object-oriented framework for scientific computing applications on parallel computers
September 21, 2018
Hill Center

34. POOMA Approach C++ class library Generic programming
high-level, generally data-parallel API
Generic programming
classes modeled after STL style
heavy use of C++ templates
Parallelism encapsulated
message-passing for distributed memory machines
multi-threaded shared memory (POOMA II)
Cross platform code development and scalable parallelism
September 21, 2018
Hill Center

35. KEY IDEAS POOMA Framework Numerical programming framework
Compile-time
Polymorphism
Computer
Science
Physics
Application
Algorithm
Local
Parallel
Global
STL
Expression
Templates
Domain
Decomposition
Message
Passing
Load
Balancing
Fields
Meshes
Particles
Interpolators
FFT
Differential
Operators
MC++
NTTP
LINAC
KEY IDEAS
Numerical programming framework
Encapsulated parallelism
High-level API’s / data support
September 21, 2018
Hill Center

36. PDT (Malony, University of Oregon)
Program Database Toolkit
Program analysis
multi-language (Fortran, C, C++, Java)
commercial- grade parsers
IL to program database (PDB)
API for PDB access / query
Tools: instrumentation, code wrapping, documentation
September 21, 2018
Hill Center

37. SILOON (Advanced Computing Lab, LANL; UO)
Scripting Interface Language for OO Numerics
Toolkit and run-time support for building easy-to-use external interfaces to existing numerical codes
Scripting language to “glue” components together
KEY IDEAS
Support for application interaction control
Support for application code wrapping
Application / tool coupling
Data exchange support
September 21, 2018
Hill Center

38. Metasystems and Metacomputing
Many resources accessible on the internet
computers, data, devices, people
Extend single system model to internet domain
wide-area (department, campus, region, country)
scalable, transparent access to resources
hides network complexity (“as if on your machine”)
Extend computing model to internet domain
shared persistent space of objects (data, execution)
heterogeneous distributed and parallel processing
meta-applications (multi-component, hierarchical)
Deal with complex environment / primitive tools
September 21, 2018
Hill Center

39. “The GRID”
New applications based on high-speed coupling of people, computers, databases, instruments, ...
computer-enhanced instruments
collaborative engineering
browsing of remote datasets
use of remote software
data-intensive computing
very large-scale simulation
large-scale parameter studies
September 21, 2018
Hill Center

40. GRID Architectural Picture
KEY IDEAS
Metasystems infrastructure / services
Metacomputing applications programming
GRID resources
September 21, 2018
Hill Center

41. NetSolve (Dongarra, University of Tennessee)
Client-server system to access distributed computational / DB HW/SW resources
Distributed computing: resources, processes, data, users
Load-balancing policy for efficiency / performance
Integration with arbitrary software components
C, Fortran, Java, MatLab, Mathematica, Excel
BLAS, (Sca)LAPACK, MINPACK, FFTPACK
September 21, 2018
Hill Center

42. NetSolve Usage “Blue collar” GRID-based computing Scenarios
users can set things up (without “su” privileges)
no deep network programming knowledge required
Scenarios
clients, servers, and agents anywhere on Internet
clients, servers, and agents on an Intranet
clients, servers, and agent on the same machine
Focus on MATLAB users
OO-style language (objects are matrices)
one of most popular desktop systems for numerical computing (> 400K users)
September 21, 2018
Hill Center

43. NetSolve – The Client
NetSolve API hides complexity of numerical software
Computation is location transparent
Provides access to virtual libraries:
Component GRID-based framework
Central management of library resources
User not concerned with most up-to-date versions
Automatic tie to Netlib repository
Synchronous or asynchronous calls
User-level parallelism
September 21, 2018
Hill Center

44. NetSolve – The Agent and Server
gateway to computational services
performs load balancing and resource management
Server
various software installed on various hardware
configurable and extendable
framework to easily add software
many numerical libraries being integrated
supports parallel computing
September 21, 2018
Hill Center

45. MCell (Bartol, Salk Institute; Salpeter, Cornell)
Monte Carlo simulator of cellular microphysiology
Study how neurotransmitters diffuse and activate receptors in synapses between different cells
NetSolve distributes processing workload and allows access to computational resources
Simultaneous evaluation of large number of different parameter combinations
September 21, 2018
Hill Center

46. INTERLACE (Malony, University of Oregon)
INTERoperation and Linking Architecture for Computational Engines
Goals
framework for building high-performance computing environments from existing tools
reusable components in heterogeneous environment
abstract connection mechanisms for control/data flow
resource management for dynamic operation
use standard software technologies
parallel and distributed computational environments
September 21, 2018
Hill Center

47. KEY IDEAS INTERLACE Components
Computational engines: libraries or programs providing specific functions
Computational server: program interfacing multiple engines with middleware
Wrappers: server-engine interface for data/control
Middleware: server-to-server interoperation software
KEY IDEAS
High-level numeric computational services
Access to metasystem resources
Wrapping/linking of computational engines
Dynamic, adaptable, extensible
High-level metasystems programming support
September 21, 2018
Hill Center

48. ViNE (Malony, University of Oregon)
Virtual Notebook Environment
High-level, shared notebooks, data, and tools in distributed, heterogenous system
Architecture
leaves: notebook functions and data
stems: notebook communication
Web-based access
September 21, 2018
Hill Center

49. ViNE Experiment Builder
List of available, named data, tools, and experiments
Visual dataflow model of experiment process
Wrapped tools and databases
wrapped
MATLAB
“tool”
September 21, 2018
Hill Center

50. Brain Electrophysiology Lab Notebook
Dense array EEG datasets
Commercial of the shelf statistical and numerical packages
Multiple machines types
Notebook content automatically generated from experiment results
September 21, 2018
Hill Center

51. PUNCH Purdue University Network-Computing Hubs
Educational and research computing “portals”
across the Purdue “enterprise”
with affiliated institutions
Resource sharing by Purdue users
computers, software, laboratory equipment
educational materials
Distance education
allows sharing of courses and instructors
Collaborative research
September 21, 2018
Hill Center

52. PUNCH – User’s and Developer’s View
Set of network-based laboratories that provide software tools for various fields
Specialized WWW-server interfaces WWW-browsers
access software and download data
run tools and view results
Tool specification
Virtual laboratory development environment
September 21, 2018
Hill Center

53. PUNCH Web Page
Hubs
September 21, 2018
Hill Center

54. PUNCH Tool Display Support via VNC
X Windows
display
KEY IDEAS
Web-based access to tools
Web-based applications development
Web-based data, results, process management
MATLAB command
window
MATLAB interactive
window
MATLAB graphics
window
September 21, 2018
Hill Center

55. Opportunities and the Neural Informatics Center
Integrated time-dynamic neuroimaging poses methodological, computational, and informatic challenges
Apply computer science technology to create problem solving environment for brain analysis
neuroscientist defines methods and processes
add value to environment through its use
Neural Informatics Center (NIC) within BBMI
focus on single trial analysis problem
advanced EEG/ERP analysis and integrated fMRI
BEM/FEM brain models (EEG, CT, MRI)
September 21, 2018
Hill Center

56. Final Thoughts Enable high-level problem solving environments
Tools to enable scientists to compose solutions from a set of building blocks
Seamless access to local and remote resources
Enabling infrastructure
framework standards and interfaces
implementations of reusable components
Collaboration environments
Future Neural Informatics Grid
September 21, 2018
Hill Center