1. www.haptica.com | tel. 617 342 7270 | email. info@haptica.com Efficient rigid body motion tracking with applications to human psychomotor performance assessment “Surgeons who trained on simulations performed the operation 29% faster and made 7 times fewer errors”. Annals of Surgery 2002 Gerard Lacey Computer Vision and Robotics Group Dept. of Computer Science Trinity College Dublin

2. Haptica ProMIS May2006 Page 2 Overview of Talk Context for research – Haptica Ltd Laparoscopic Surgery Intro Solutions to Surgical Training Computer vision tracking Live multimedia annotation of training video The results The current position The Future

3. Haptica ProMIS May2006 Page 3 Inspiration for ProMIS Previous Research Mobile Robotics for Frail Visually Impaired Elderly Bayesian Belief Networks for multi-sensor classification / context sensitive reasoning Haptica Ltd formed in 2000 Focus on human performance enhancement in critical tasks Guidance for mobility Computer Assisted Docking Fraud detection in credit card payments Semantic Web search of patent databases Laparoscopic Surgery Training

4. Haptica ProMIS May2006 Page 4 Laparoscopic Surgery Laparoscopy – Minimally Invasive Surgery Surgeons operate with long narrow instruments Watch their movements on TV monitors Rapidly growing due to patient demand and cost savings in reduced patient stays Key Challenges Loss of dexterity Fulcrum effect Loss of sense of touch Slower than conventional surgery Environment for learning Apprentice model See one, do one, teach one Patients used for learning No objective assessment

5. Haptica ProMIS May2006 Page 5 Current Training Methods Current Training Tools Patients Physical simulators - “Box Trainers” Virtual Reality and Full Haptic Simulators Physical SimVR Sim System behaves like abdomen Realistic tactile feedback Realistic visual feedback Use the real tools Team approach to procedure Simulate specific surgical procedure Assess Trainee ’ s performance Use with computer based training Playback a video of their operation Simulate affect on of errors Simulate unusual pathologies

6. Haptica ProMIS May2006 Page 6 Objective Skills Measurement in Medicine Human observation of bench tests Resnick – U Toronto (OSATS) Track hand movements to measure surgical skills Darzi – Imperial College (ICSAD) Linking skills to patient outcomes VR to OR trials Key understanding of the skills to be learned Cognitive X% + Affective Y% + Psychomotor Skill Z% Surgery mainly 75% X & 25% Z with Z higher in minimally invasive surgery

7. Haptica ProMIS May2006 Page 7 Key Skills in surgery Key indicators of skill acquisition The objective is Proficiency The time proxy – How fast can it be done? The repetitions proxy – How many times has it been done? The errors proxy – have any of these listed errors occurred while the task was being completed? Surgical Finesse – smooth movement / economy of movement Path of surgical instruments in space Novice Expert

8. Haptica ProMIS May2006 Page 8 Measuring Economy of Movement Stereo Cameras Track Instruments

9. Haptica ProMIS May2006 Page 9 Design of the Stereo system Background simplification Camera calibration – essential step Tracking algorithm Top 1/3 of the image Instrument identification Image de-warping Icon tracking Complications Multiple instruments – Occlusion Correspondence disambiguation

10. Haptica ProMIS May2006 Page 10 Vision Systems Design Off the shelf web cams (firewire) Calibration of cameras to sub pixel accuracy High frame rate (15fps or higher) Task trays manufactured to high accuracy Task trays rigidly mounted to high accuracy Key design factor in enabling low cost Augmented Reality Bile duct AR

11. Haptica ProMIS May2006 Page 11 The Augmented Reality system Aligning parallel 3D Universe’s Vision System reference frame Graphics Engine reference frame Mechanical mounting reference frame Registration Optical Distortion De-warping of images Occlusion masks (instruments and objects) Real-time tracking and AR display (on a laptop) down and dirty in direct x / direct show

12. Haptica ProMIS May2006 Page 12 Augmented Reality Concept Annotate live video with Computer Mediated Learning content Teaching Instructions Feedback on Progress Hybrid tasks Adverse Events AR Demo

13. Haptica ProMIS May2006 Page 13 Results ProMIS Construct Validity Trials RCSI / AMNCH Emory University Imperial College Very high ability to discriminate novice from expert using economy of movement Papers published in surgical fora Other Validations European IST prize Irish Software Association Technical Innovation Prize Society of Laparoendoscopic Surgeons Innovation of the Year award Irish Design Institute “Highly Commended Design”

14. Haptica ProMIS May2006 Page 14 Idea refinement – The importance of key reference customers Concept to product March 02 to December 03 Patent May 02 Basic concept – cardboard box Stage Gate™ type process Multiple Prototypes with RCSI, Emory (US), Imperial (UK) Production prototypes June 03 Production process set-up Oct 03 First sales to RCSI December 03

15. Haptica ProMIS May2006 Page 15 Current Position ProMIS 2 Launched October 2005 Hand Tracking for “hand assisted” Procedures Model abdominal organs Full procedure training for Colectomy 8 cameras, tilt-sensor, animatronics Still runs on a laptop… 7 Papers validating ProMIS published at most recent surgical congress

16. Haptica ProMIS May2006 Page 16 ProMIS 2.0

17. Haptica ProMIS May2006 Page 17

18. Haptica ProMIS May2006 Page 18

19. Haptica ProMIS May2006 Page 19

20. Haptica ProMIS May2006 Page 20

21. Haptica ProMIS May2006 Page 21

22. Haptica ProMIS May2006 Page 22 Future of Augmented Reality and Video Indexation Current Projects Illicit Content detection Colonoscopy: Automated Reporting tools - tagging Colonoscopy: Automated cancer detection Hand washing quality assessment Ground Collision Warning System for Aircraft