1. Realtime FMRI: Robert W Cox, PhD Medical College of Wisconsin Why Who, What, When, Where, How, and © 2000 RWCox

2. What is Realtime?  Realtime  computation of results or response within a fixed time of external triggering event  True realtime computing requires a guarantee that the calculations will be completed within the specified fixed time  Don’t often give this guarantee in FMRI

3. Definitions of Realtime FMRI  Display of results/maps within fixed time after end of scanning run, or  Display of intermediate results/maps within a fixed time after end of each image acquisition, or  Calculation of some parameters from the FMRI results/maps (e.g., to make decision about what to do next)

4. Where & Who ( and a little When )  UIUC/NCSA [Potter et al.] proposed the “Neuroscope” at 1994 SMRM meeting  MCW [Cox et al.] implemented recursive single-slice RT-FMRI in 1994  3D in 1997; 3D registration in late 1998  Software available from MCW (me): AFNI  Other groups:  UPMC [Voydovic et al.]  Jülich [Posse et al.]  Mayo Clinic [Riederer et al.]  Scanner makers have ongoing efforts

5. How [a lot of slides]

6. Pulse Sequence Subject Scanner StimulusResponse Image Reconstruction Image Registration/ Motion Detection Time Series Analyses Compute & Display Neuropsychological Stuff Investigator! (Every presentation needs a block diagram)

7. Keeping up with the Data  Must keep pace with raw data acquisition  2D EPI: just a few FFTs — no big deal on Pentium II-III class systems  2D Spiral: regridding too slow for single CPU — must send out to CPU “farm”  Methods that use B 0 field map will need to get it before the FMRI time series, or only use it later for offline (higher accuracy) reconstruction  Best idea: get images from scanner computer in realtime Image Reconstruction

8. The Biggest Practical Problem in FMRI: Dealing with Subject Head Motion [several slides] Image Registration/ Motion Detection

9. Effects of Subject Motion on Brain Activation Maps  Case 1: Motions are “random”  Signal changes look like extra noise  Areas of activation shrink  Case 2: Motions occur with stimulus  Signal changes look like expected time course  False areas of activation appear  Both cases can occur together  Best solution: don’t let subjects move

10. The Need for Image Resampling + Have data at horizontal- vertical grid intersections Need data on slanted grid, at solid dots Regions around pink dots show original grid areas we might use to compute resampled image values

11.  Easy to implement and executes quickly  Will smear image (a lot if done repeatedly)  Noise: will introduce spatial correlation between neighboring voxels (Mis)Features of Linear Resampling Original image After 12 30  rotations with linear resampling

12. Registration Goals for Online FMRI  Estimate 3D (6 DOF) movement parameters as fast as volume acquisition happens  Realign each volume to a “target” volume during scanning  Display updating graphs of estimated motions to investigator  Feedback movements to slice selection?

13. Technique used in AFNI: 3D Image Rotation via Shears  Factor rotation matrix into products of shear matrices:

14. Shear in the x-direction x new = x + a·y + c ·z y new = y z new = z x y x y  Must resample row from old grid to a shifted grid with the same spacing

15. 3D Rotation as 4 1D Operations  Each matrix is just a 1D row operation:  S x leaves y and z unchanged: x new = x + a ·y + b  z  3D rotation matrix can be written as a product of 4 such shear matrices: U = S z S x S y S z  May need to use a different shearing order  Can do row resamplings with 1D FFTs:  Total of 8 N 2 1D FFTs (2N 2 per shear)

16. CPU Times for 3D Registration 128  128  30  80 volume acquired in 264 s: Resampling Method Computer SystemFourierHeptic Quintic SGI R10000 175 MHz 335.0 * 227.5 213.3 HP PA-8000 200 MHz 276.2 * 127.6 115.6 Pentium II 400 MHz 162.1 100.7 90.0 * Slower than realtime

17. Other Approaches  Use navigator echoes to detect slice position+orientation prior to each acquisition; alter slice selection before 2D image acquisition  Good: effectively like keeping subject’s head still  Bad: much MRI time spent in navigator echoes  Alternative: track head with lasers+mirrors  Register volumes as before, but feed back to pulse program for next image  Bad: can’t track subject movements < 1  TR  Good: no MRI time penalty

18. Analysis of Voxel Time Series in Realtime [a few slides] Time Series Analyses

19. Temporal Pattern Matching  Used when form of FMR signal is known  v(t) = measured signal in a single voxel   h(t) + a + b  t + noise  h(t) = known function of time (“reference”)  Small number of unknown parameters:  = amplitude of response a,b = mean and trend (background stuff)  Finding parameters via linear least squares is equivalent to the correlation method

20. Multiple References  v(t)   1  h 1 (t ) +  2  h 2 (t ) +    1,  2 are amplitudes (unknown)  h 1, h 2 are known reference responses  Used for experiments with more than 1 stimulus condition: rest task A rest task B rest task A  h 1  0 h 2  0 h 1  0  Widely used for event-related FMRI

21. Linear Deconvolution  v(t)  a + b  t +  j h(t  j ) + noise   j = stimulation times [known]  h(t) =  k  k  u k (t) = response function  u k (t) = basis functions [known]   k = amplitudes [unknown]  Goal is to find shape and amplitude of response function in each voxel  Unlike previous analyses, form of response is not completely bound to hemodynamic assumptions

22. Recursive Linear Regression  All methods above can be cast into form of linear regression:  Solution of linear equations to get estimated fit parameters  Estimation of significance from noise model [i.e., using what’s left after regression fit]  Recursive regression:  With each new time point, add one equation  Given previous solution, can re-compute new fit with relatively little work (much less than starting over)  Method used in AFNI for realtime analysis

23.  v(t)  a + b  t +  j h(t  j ) + noise  h(t) = h(t ; ) = nonlinearly dependent on  = vector of unknown parameters  Example: h(t) = A (t  ) r exp(  (t  )/c) Nonlinear Regression or Deconvolution r = 8.6 c = 0.54  = 0

24. Extended Kalman Filter  Extends ideas of recursive linear fitting to nonlinear regression  At each step, must evaluate nonlinear model and its partial derivatives w.r.t. parameters  Update formula is similar to recursive linear regression methods  Can iterate updates to adjust to nonlinear model better  Should be able to do this in realtime  I don’t think that this has been tried yet

25. Single Event FMRI: Use Nonlinear Regression?  In this type of experiment, the stimulus and its consequences last a long time  Only have one stimulus/response event per imaging run  Administration of a drug  Presentation of an affect altering video  Know when stimulus started, but don’t know exactly what response should be  Nonlinear curve fitting seems appropriate

27. Signal  R2 * time rCBV Measuring rCBV: Gd-DTPA Passage

28. Displaying Something [a few slides] Compute & Display Neuropsychological Stuff

29. Things to Look At (à la AFNI ) Graphing voxel time series data Displaying EP images from time series Control Panel

30. FIM overlaid on SPGR, in Talairach coords Multislice layouts

31. Estimatedsubjectmovementparameters

32. The End < 1 s to render Blocked trials: 20 s on/20 s off 8 blocks Color shows through brain Correlation > 0.45 1 Blocks: 12345678

33. Too Much to See  Things to monitor include:  Images (2D and 3D)  Graphs of voxel data, movement parameters, subject responses, subject physiology,....  It’s too hard to juggle multiple windows on a single screen in realtime  Idea: develop a multi-head display platform with 6 or 8 1280  1024 LCD panels  Should be able to show everything the investigator might want, all evolving in realtime

34. Why? [several slides]

35. Reasons for Realtime FMRI  The purpose of this part of the talk is to start some synapses  Don’t just listen and wish it weren’t so early in the morning!  Make suggestions  Propose ideas What should go  What should go in this box? in this box?  And how should the system boxes be connectable in realtime? system boxes be connectable in realtime? Compute & Display Neuropsychological Stuff }

36. Pulse Sequence Subject Scanner StimulusResponse Image Reconstruction Image Registration/ Motion Detection Time Series Analyses Compute & Display Neuropsychological Stuff Investigator! An N-back Memory Task

37. The Obvious Reason: Quality Assurance  Make sure you have good data before subject leaves the scanner  Repeat bad imaging runs  Provide feedback to subject (e.g., “stop nodding your head!”)  Most important when dealing with patient populations  when FMRI is used for pre-surgical planning  when patients in study are hard to come by

38. Are There Any Further Reasons?  Adjustment of stimulus level to reach a desired response magnitude  ½ of peak response  mapping of voxel stimulus-response curves  Carrying out experiment until some statistically significant result has been reached  Must pay attention to statisticians for this!  e.g., do tasks until language lateralization has been established [Besides fun, I mean] S R

39. More Research Reasons?  Rapid prototyping of new experiments  Adjustment of stimulus until are getting some results (e.g., reliably activating brain region X)  Followed by usual series of experiments on multiple subjects, with fixed stimulus protocol  Mind reading MRI  Map activation pattern and compare to a previously established library of patterns  e.g., library = subject’s retinotopic mapping then could find what part of visual field subject was attending to [Brefczynski et al.]

40. More Clinical Reasons?  Carry out some sort of neuropsycho- logical exam in the scanner  What would it entail? What would it be good for?  “Branching protocols” — when one sees a certain kind of result, protocol specifies the next stimulus/task to apply  The “ultimate biofeedback device”?  Training for control of phantom limb pain?  Rehabilitation/neural reorganization therapy?

41. Reasons Not to do Realtime FMRI  Scientific experiments should be carried out with protocols/procedures that are defined ahead of time  Information rate is slow  Hemodynamic response is sluggish  Noise is high, implying it takes a long time (minutes) to get strongly believable results  What kinds of decision-making could be useful on that lengthy time scale?

42. Almost Done (Just a Loose Ends)

43. Realtime AFNI  AFNI software package has a realtime plugin, distributed with every copy  Price: USD$0 [except for time & effort]  Runs on Unix / Linux  Requires input of reconstructed images and geometrical information about them  For more information and download: www.biophysics.mcw.edu  AFNI page

44. Thanks To MCW Faculty JR Binder, MD EA DeYoe, PhD KM Donahue, PhD JS Hyde, PhD A Jesmanowicz, PhD B Kalyanaraman, PhD MM K osek, PhD SM Rao, PhD EA Stein, PhD MCW Students MCW Students (past & present) RM Birn, PhD V Roopchansingh R Tong R Zhang, PhD Et Alii PA Bandettini, PhD MS Beauchamp, PhD BD Ward, MS

45. And now, any more thoughts, suggestions, or crazy ideas from the audience?