1. Indexing and Retrieving Dynamic Brain Images
Kickoff Meeting
March 28, 2003 SCILS, Rutgers
Paul B. Kantor, PI
Stephen J. Hanson , Co-PI

2. Overall Model
A “brain event” is characterized by an “activation” a(t) which is an (imaginary?) pointer that moves through the brain as neurons act sequentially in some “mental process”.
The set of points {t,a(t)} characterizes the “mental process”

3. Model (2)
The set of points can be conceived of as a graph in 4-dimensional space with coordinates x,t.
The graph can be concisely represented by graph indexing techniques
Those indexing techniques can be used to support retrieval of related “mental processes” even if the associated circumstances are not apparently similar.

4. Experimental-Analytic Paradigm
Do {until funds are exhausted}
Experiments produce data
Analysts reduce it to graphs
Study of graphs suggests new experiments
loop
Funds presently:
Rutgers ISTC Pilot funds 25K
NSF 2. Millions/3years
McDonnell Foundation – 500K/
NIH R01 (planning)

5. Roles of Key Personnel J. Cohen - design, conduct experiments
C. Hanson - design, conduct expts; manage data
S. Hanson - co-PI design expts; fusion time resolution
P. Kantor - co-PI design indexing and retrieval
D. Silver - reduce activations to centroids
S. Dickinson - index and retrieve graphs
L. Shepp - improve time resolution -- physics level
B. Bly - RUMBA software

6. Additional Key Personnel
Graduate Students
Ulukbek Ibraev -- graph finding and indexing
Xiaosong Yuan -- correlation and image analysis
Arnav Sheth -- diffusion of signal around activation
Yaroslav Halchenko --Fusion EEG/fMRI
Adi Zaimi –Data Collection and Design, Simulations, Modeling
Other Personnel
Donovan Rebbechi--Programming, RUMBA architecture
Mike Edwards--Programming, Archive Maintenance
Barak Pearlmutter--Theory and Algorithm Development
Toshi Matsuka-- Cognitive Modeling and Neural Computation

7. Threats to the Undertaking
A. Time resolution of fMRI is very poor
B. Hemodynamic response wrt to Neural spike firing is poorly understood
C. Signals may leap across brain on long neurons
D. All brain activation may be connectionist/distributed
E. All other threats (Tell the Wigner story)
Today’s focus:
Improving time resolution & Dynamic Activity Modeling

8. Improving time resolution
New basis functions for selected regions (today)
Shepp et al
Fusion (at a later meeting) - Hanson et al
Ingenious spacing of stimuli and collection
Kantor

9. Dynamic brain activity modeling (BOLD)
Wavelets (today)
--Daubechies
Cluster following (at a later meeting)
--Kantor & Silver
Skeletonization (at a later meeting)
--Dickinson

10. TOOLS and ARCHIVE RUMBA tools (at another meeting) --Ben Bly
Archive—RUMBA sharing
--Donovan Rebbechi
AIR- RUMBA

11. Data Collection & Cognitive Theory
Continuous fMRI paradigms: Event Perception
(at later meeting)
--Catherine Hanson
Similarity based fMRI paradigms: Flanker task
(at a later meeting)
--Jonathan Cohen
Allegra User Group--(set up by Cohen—www. Fill in)

12. Administrivia Although we are all hard at work
we have just completed the contract Rutgers-Princeton
the NSF would like to know what we are accomplishing, to decide whether to give us the next part of the funding
Materials so far submitted are at the web site:
Please send material to me.

13. Project Details: Improved Detection methods Neural Networks and Object Recognition (Matsuka, Hanson & Haxby)
Problem: Determining the sensitivities of voxels in ventral temporal lobe for object recognition.
Method:
1) Neural Networks as a nonlinear classifier—no contrast or baseline reference!
2) sensitivity analysis with noisei ~N(0, SDi)
Results: 
(a) individual voxels are sensitive in recognition of multiple categories.
(b) patterns of voxels’ sensitivities are somewhat similar for many categories.

14. Object Recognition (cont.) (Matsuka & Hanson)
Face House Cat Bottle
Results (figure)
Use: can be used for detection of complex signals and input into dynamic indexing algorithms
Scissor Shoe Chair Random

15. Project Details:RUMBA tools
D. Rebbechi, B. Bly, S. Hanson-Newark
C++ Library
Python scripting/python environment support
Command line tools
Work in progress: GUI

16. RUMBA sharing—Napster Brain.
Archive
RUMBA sharing—Napster Brain.
D. Rebbechi, B. Bly, S. Hanson
Encrypted data archive
Data sharing requires the downloader to obtain the owners consent
A data sharing agreement is represented in a cryptographically signed `contract'
XML repository summarising data is publically accesible but data is hidden.

17. AIR- RUMBA (automatic image registraion)
D. Rebbechi, B. Bly, S. Hanson
Rigid body motion correction
Affine inter-modality registration using AIR-inspired cost function.
Polynomial warp transformation for template based image registration
Future directions: stochastic gradient method, using mutual information

18. Project Details:FMRI/EEG FUSION Hanson, Halchenklo & Zaimi
Preprocessing
EEG noise/artifacts(eyeblinks) removal – ICA
fMRI baseline preprocessing
Initialization
Merge inverse solutions for EEG or fMRI
Use LP to get 'worst-ever' approximate solution
Optimization to concurrently reconstruct both signals F and E while satisfying constraints on fused modality S: smoothness in time/space. Each signal has some temporal (fMRI) or spatial (EEG) influence on the other through forward equations.

20. Project details: Data Collection: Data Flow
(Hanson, C. Edwards, Rebbechi)

21. Data Archive
(Hanson, C. Edwards, Rebbechi)

22. Data Sets So far (Hanson, C, Edwards, Rebbechi)
Oddball task (C Hanson, Rutgers) – subject responds when an “oddball” (a nonconforming stimulus) appears in a series of identical stimuli
Event perception task (C Hanson, Rutgers) – subject asked to parse action sequences into meaningful units (events)
Incremental stimulus recognition task (C Hanson, Rutgers) – subject probed for identity of occluded objects that are incrementally revealed
Noise and motor and auditory tasks( (Bly, C. Hanson Rutgers)- subject either at rest or doing simple motor task (finger tapping) or listening to auditory input
1-back task (Haxby, Princeton) – subject is presented with a series of stimuli and periodically asked to decide if current stimulus was presented in the previous trial
Morality task (Cohen, Princeton) – subject asked to make moral decisions about fictitious situations
Flanker task (Cohen, Princeton) – subject is asked to report the directionality of an arrow when flanking stimuli are consistent or inconsistent with target