1. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Medical Robotics and Computer-Integrated Surgery Russell H. Taylor Dept. of Computer Science/Radiology/Mechanical Engineering Director, Center for Computer-Integrated Surgical Systems and Technology The Johns Hopkins University 3400 N. Charles Street; Baltimore, Md. 21218 rht@cs.jhu.edu NSF Engineering Research Center for Computer- Integrated Surgical Systems and Technology

2. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Disclosures Russell Taylor is a member of the Scientific Advisory Board of Integrated Surgical Systems and of Image Guide, Inc., and he consequently has (an exceedingly small) financial interest in these companies, and he may join other Advisory Boards. Some of Dr. Taylor’s patents have been licensed by Image Guide, Incorporated, and he consequently has a small financial interest in IG, Other of Dr. Taylor’s patents have been licensed by Intuitive Surgical Systems. ISS, IG, GE, Intuitive Surgical, Siemens, Medtronic, Burdette Medical, and several other companies whose systems were discussed in this talk are affiliates of the NSF CISST ERC, of which Dr. Taylor is the Director. Dr. Taylor also enjoys fishing

3. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Prediction The impact of computer- integrated surgical systems and technology on medical care in the next 20 years will be as great as the impact of computer-integrated manufacturing systems and technology on industrial production over the past 20 years. Engineering Research Center for Computer Integrated Surgical Systems and Technology

4. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Basic means for fulfilling prediction: systems that integrate information to action Provide new capabilities that transcend human limitations in surgery Increase consistency and quality of surgical treatments Promote better outcomes and more cost-effective processes in surgical practice

5. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Imagers and other sensors Robotic devices Human- machine interfaces Interface technology Computers and networks Image & sensor processing, anatomical modeling, surgical planning and control, … Patient data bases Anatomic atlases and surgical task models patient surgeon Images & other sensor data Command & control Decision support & commands

6. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Computer- assisted planning Patient-specific Model Update Model Computer- Assisted Execution Update Plan Computer- Assisted Assessment Preoperative Intraoperative Atlas Postoperative Patient

7. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Computer- assisted planning Patient-specific Model Update Model Computer- Assisted Execution Update Plan Computer- Assisted Assessment Preoperative Intraoperative Atlas Postoperative Patient Surgical “CAD” Surgical “TQM” Surgical “CAM”

8. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Computer- assisted planning Patient-specific Model Update Model Computer- Assisted Execution Update Plan Computer- Assisted Assessment Preoperative Intraoperative Atlas Postoperative Patient Surgical Assistants

9. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Computer- assisted simulation & planning Patient-specific Model Update Model Simulate and replan in real time Computer- Assisted Assessment Preoperative Intraoperative Atlas Postoperative Patient The grand unified system* * From Turf Valley Workshop (March 2003)

10. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Drivers for adoption Error-free surgery –Designs & systems to make things safer –Integration of imaging, computer assistance, etc. Enablers for otherwise infeasible interventions –Superhuman capabilities (access, precision, …) –Delivery systems for novel image-guided therapies Enablers for research & development –Consistency, capability & data gathering Capture of information about cases Use of robots as training tools –Mentoring –Simulation integrated with robotics * From Turf Valley Workshop (March 2003)

11. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Research and technology barriers Modeling & analysis Segmentation Registration Atlases Optimization Visualization Task characterization etc. Interface Technology Sensing Robotics Human-machine interfaces Systems Safety & verifiability Usability & maintainability Performance and validation

12. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Building anatomic models Data fusion Reasoning about uncertainty Managing complexity Computational algorithms Limited data Modeling & Analysis Core Challenge: Develop computationally effective methods for patient-specific modeling and analysis Before After

13. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Modeling and Analysis Highlights Explosion of 3D imaging modalities –MRI, CT, Ultrasound, etc. Patient-specific models based on images Statistical atlases and deformable models are emerging as crucial enabling technologies within CIS Beginning to see application of real-time computer vision methods

14. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Interface Technology Imaging and sensing Robotics Human-machine interfaces Core Challenge: Fundamentally extend the sensory, motor, and human-adaptation abilities of computer-based systems in an unusually demanding and constrained environment

15. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Interface Technology Highlight Robotic devices for surgery and image-guided interventions

16. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Systems Science Modularity & Standards Robustness & Safety Customization Validation Information management Core Challenge: Develop architectures, building blocks, and analysis techniques that facilitate rapid development and validation of versatile CIS systems & processes with predictable performance Engineering Research Center for Computer Integrated Surgical Systems and Technology

17. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Systems Highlights Commercial systems –CAD/CAM: Robodoc, Arthrobot, Neuromate, Accuray, Image-Guide, … –Assistants: daVinci, Zeus, AESOP, … Research systems being clinically evaluated –E.g., JHU in-MRI prostate robot, US guided robots, … Recognized need for testbed architectures &shared components New interest in systems as enablers for biomedical research Intuitive Surgical daVinci

18. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Surgical CAD/CAM Goals: to develop and evaluate systems that will incorporate surgical plans into surgical execution using computers and robotics. Research areas: –Percutaneous therapy systems (needle placement) –Patient specific models (image segmentation and registration) –Statistical atlases (prostate, femur, pelvis) –Robotics (remote center of motion, needle drivers)

19. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Image-Guided Needle-Based Interventions Potentially significant impact on medical practice Minimally invasive (compared to open surgery) Faster recovery Less morbidity Fewer complications Lower cost Repeatable in many indications Sharply increasing number of procedures Challenging but also doable Constrained process – formally describableConstrained process – formally describable Major challenges (in addition to open/lap surgery): no visibility no access no room to maneuver no room to recover

20. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Current clinical scope Spine/Bone 70% of population affected in lifetime 400,000 metastatic cancer /year Liver Metastasis from colorectal cancer 130,000 new /year 60,000 death /year Hepatitis worldwide Prostate 200,000 cancers/year 1M biopsies /year 10M BPH currently 25% of men affected in lifetime Why these? Significant health problems Right mix of challenge and doability Clinical buy-in Experience of investigators Funding opportunities United States numbers Eye ~100k/y retinal occlusions, >100k/y age- related macular degeneration (AMD) Ear Hearing loss of 30-35% of 65- 75 yo 40-50% over 75 yo MACRO SCALEMICRO SCALE

21. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Why these organ systems? Humanitarian reasons Significant health problems in US and worldwide Opportunistic reasons Right mix of challenge and doability Clinical buy-in Funding opportunities Experience of investigators Publishing opportunities

22. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Engineering Goals 1.“One-Stop Shopping” Plan, Do, and Verify – in one sitting 2.“Plug & Play Surgery” Rapidly configure for target organ, imaging modality, and therapy Predictable performance 3.Critical Mass of Components

23. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Surgical Assistants Goals: to develop and evaluate systems that augment the physical and sensory performance of a surgeon during surgery Research areas: –Microsurgery (tremor reduction, manipulation on small scale) –Video augmentation (tool tracking mosaicing) –Surgical enhancement (tracking surgical actions, virtual fixtures)

24. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Handle force KvKv Steady Hand Guiding for Microsurgery R. Taylor & R. Kumar Free hand motion Steady hand motion

25. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Stable platform for interventions R. Kumar, 2001

26. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Biomanipulation with a steady hand robot Rajesh Kumar (1), Ankur Kapoor (2), Russ Taylor (2) Cooperatively controlled robot for single-cell scale biomanipulation tasks Multiple control modes Simple compliant guiding “Augmented” compliant guiding Shared/supervised autonomy Future work includes: Next generation system Visual “virtual fixtures” (1) Foster-Miller; (2) CISST ERC, JHU

27. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Evolution to human-machine partnership Situation assessment Task strategy & decisions Sensory-motor coordination Augmentation System Sensor processing Model interpretation Display atlases Manipulation enhancement Online references & decision support Cooperative control and “macros” atlases libraries R. Taylor

28. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Computer- assisted simulation & planning Patient-specific Model Update Model Simulate and replan in real time Computer- Assisted Assessment Preoperative Intraoperative Atlas Postoperative Patient The grand unified system* * From Turf Valley Workshop (March 2003)

29. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology System barriers Safety & verifiability “The usual” No fly zones Information-based supports Usability & maintainability “The usual” Surgical performance analysis Recording information Performance and validation “The usual” Accuracy propogation Analysis-based generation of test plans Standards

30. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Key need: Standards Interoperability System/OR architectures Information Interchange Control Human interfaces Performance measurement Accuracy Task performance Outcomes Use/Practices Information archiving

31. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology A few “grand challenge” areas Grand Unified System –Real-time patient-specific models –Surgical total information management –Active human-machine partnerships –Correlate performance to outcome “Fanstastic Voyage”* –Micro-invasive interventions on tiny structures Surgery on cells & microstructures

32. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Surgical Total Information Management Real time sensing & modeling of surgical situation –Patient-specific models –Surgical tools & tool-tissue interations –Surgeon movements –Etc. Application drivers –Assistance (robots & information support) –Outcomes assessment Technology drivers –Imaging, computer vision, image analysis –Sensors, sensor networks –Biomechanics, statistical modeling –Human-machine interfaces, haptics –Performance/task modeling

33. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology How can we get there? Strong and committed teams –Surgeons –Engineers –Industry Focus on systems that address important needs Rapid iteration with measurable goals Have fun!

34. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology The real bottom line: patient care Provide new capabilities that transcend human limitations in surgery Increase consistency and quality of surgical treatments Promote better outcomes and more cost-effective processes in surgical practice

35. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Questions Where are we now, and where should we be in 5, 10, and 20 years? Why can software for future medical devices not be developed using existing design and productivity technologies? What are emerging technical and scientific issues in medical devices? What are key obstacles to the development and certification of medical device software and systems? What are the major obstacles in innovative design and accelerated certification? How can information and software technology help?

36. Copyright © CISST ERC, 2004 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology Bio Russell H. Taylor has a B.E.S. from Johns Hopkins, and a Ph.D. in Computer Science from Stanford. He joined IBM Research, where he developed the Agent Markup Language (AML) robot language. Following an assignment in Boca Raton, he managed robotics and automation technology research activities at IBM Research until returning to full-time technical work, when he became a manager of Computer Assisted Surgery. Taylor then moved to Johns Hopkins University as a Professor of Computer Science. He is currently a Professor of Computer Science, with joint appointments in Radiology and Mechanical Engineering, and is the Director of the NSF Engineering Research Center for Computer-Integrated Surgical Systems and Technology at Johns Hopkins. Research interests include robot systems, programming languages, model-based planning, and the use of imaging, model-based planning, and robotic systems to augment human performance in surgical procedures. He is Editor Emeritus of the IEEE Transactions on Robotics and Automation, Fellow of the IEEE, and a member of various honorary societies, panels, editorial boards, and program committees.