1. 3-D Visualization of Functional Brain Map Data A.V. Poliakov; E.B. Moore; J.F. Brinkley, Structural Informatics Group Department of Biological Structure University of Washington

2. UW Human Brain Project Structural Informatics Group, Dr. Brinkley Collaborators Neurosurgery, Dr. Ojemann Psychiatry, Dr. Corina Radiology, Dr. Maravilla Medical Procedure Neurosurgery for intractable epilepsy Objective Remove epileptic foci while sparing the patient’s language function Method Language mapping using electrical stimulation Information System Help researches organize and analyze the data Unique data Study language Evaluate non-invasive imaging techniques for surgical planning

3. Data acquired during surgery Electrical stimulation map Electrical stimulation map Phonograms Phonograms Data acquired prior to surgery MR scans of cortical anatomy, veins, arteries MR scans of cortical anatomy, veins, arteries Functional MRI Functional MRI Information System for Language Mapping Primary Sources of Data

4. Data acquired during surgery Electrical stimulation map Electrical stimulation map Phonograms Phonograms Single unit recording Single unit recording Data acquired prior to surgery MR scans of cortical anatomy, veins, arteries MR scans of cortical anatomy, veins, arteries Functional MRI Functional MRI EEG in grid patients EEG in grid patients Information System for Language Mapping Primary Sources of Data

5. Brain Mapper Alignment Masking Reconstructing 3D models Mapping CSM Sites Information System for Language Mapping

7. In development fMRI DB and Automated Data Analysis Single Unit DB Data Management Talairach Daemon Query Query the Brain Mapping Repository using XQuery Information System for Language Mapping SPM

8. Web Interface

17. 3D Visualization System Architecture –Client-server –Server-based rendering and data storage –Web-based client (e.g. Java or forms-based) Web Interface Server Client #1 CGI script Client #2 Java Applet Client #3... Internet Graphics Server snapshot Stimulation Sites 3-D Models 3-D Image Volumes Client Data Experiment Management System Relational Database 2-D Images Brain Mapper

18. 3D Visualization Requests New Users Different data modalities Various software packages Spectroscopy ERP Source Signal fMRI Transcranial Magnetic Stimulation Collaborators New features Remote collaboration

19. 3D Visualization Limitations of current system Relatively slow software rendering Non-portable architecture SGI Irix and Linux only Large code base Needs to be maintained and developed Reliance on outdated software tools

20. 3D Visualization Other possible approaches OpenGL, DirectX (Windows) VTK, Java3D Higher level toolkits Java3D Cross Platform Supports hardware acceleration Off-screen rendering Java Pilot studies Performance Performance Off-screen rendering Off-screen rendering Client-server Client-server Proof of concept BrainJ3D – Next Generation

21. Live Demo BrainJ3D

22. BrainJ3D Features Seen in Demo Cutaway Cutaway Mapping Mapping fMRI Overlays fMRI Overlays Workspace Workspace

23. BrainJ3D Other Features Supports common formats and data types (Analyze, Minc, data created with SPM, FSL, etc.) Supports common formats and data types (Analyze, Minc, data created with SPM, FSL, etc.) –Leverages functionality of other programs –Increases user base –Use Java utilities 3D model loaders 3D model loaders NetCDF library NetCDF library 2D image API 2D image API Configuration and internal files in XML Configuration and internal files in XML –More portable –Easy to customize

24. BrainJ3D Other Features Client/Server Client/Server –Client/Server mode looks and behaves exactly like standalone mode –Reduced performance Sending the image is a bottleneck Sending the image is a bottleneck –Takes advantage of built in Java security –Thin client only requires installation of Java Java3D and accelerated graphics card not required Java3D and accelerated graphics card not required

25. BrainJ3D Architecture

26. BrainJ3D BrainJ3D Java and Java3D provide viable development environment for scientific visualization applications + Cross Platform ( Linux, Windows and Solaris) + Supports hardware acceleration and scene optimization + Supports hardware accelerated off-screen rendering + Mature software package that provides needed functionality + Java Modern object-oriented language Numerous libraries and utilities Web-based and remote technologies are well supported

27. BrainJ3D BrainJ3D Future Development Different way to access program Java Web Start HTML GUI Advanced Visualization Techniques Time series (movie) mode for Volumes Show functional activation on Surface model Other Features Analysis and visualization features requested by users

28. Current and Future Work Integration with Information System for Language Mapping Improving Interoperability Support for other data modalities Spectroscopy, ERP source signal, Transcranial Magnetic Stimulation Importing data from other Brain Imaging software packages FreeSurfer and SureFit * Interoperability is a fast moving target, often beyond our control Refining Client/Server mode SecurityPerformance

29. UW Human Brain Project Structural Informatics Group http://sig.biostr.washington.edu/ Acknowledgements Human Brain Project grant DC02310 National Institute of Deafness and Other Communication Disorders and National Institute for Mental Health