SSGRR - 2000 International Conference on Advances in Infrastructure for Electronic Business, Science and Education on the Internet, July 31 - August 6, L’Aquila, Italy

Designing Virtual Environments for Critical Transactions and Collaborative Interventions: the VERTEX / APIA Framework for Networked, Physics-Compliant Objects Denis Poussart Denis Laurendeau François Bernier Martin Simoneau Nathalie Harrison Denis Ouellet Christian Moisan poussart@gel.ulaval.ca Computer Vision and Systems Laboratory and Interventional MRI Unit, CHUQ, Université Laval Québec, CANADA www.gel.ulaval.ca/~vision/ Supported, in part, by grants from NSERC, FCAR, the Institute for Robotics and Intelligent Systems and the Canadian Foundation for Innovation

Critical Interventions represent cases where errors, delays, lack of optimization} may have very negative consequences for safety for the environment for health for costs … In the future, as more and more complex situations arise, we may anticipate that operational support from Virtual Environments will become paramount and prevalent in the planning training execution phases of delicate tasks

The inspection, maintenance and repair of hydroelectric facilities is just one example

Virtualizing “reality” Human Machine Interaction Computer Graphics Computer Vision Automated generation of geometry of objects and scenes

But something is missing ... For critical tasks, visual illusion is not sufficient. There is more to “real things” than just shape, or forms, even if they are augmented with some “behaviors”. Reality includes PHYSICS! Accurate physical modeling, laws, and simulation capabilities must be integrated within the virtual environment.

Virtualizing “reality” Human Machine Interaction Simulation Physics actual world Computer Graphics Computer Vision

This opens up a huge question space! what is relevant to be physically modeled? what would be appropriate forms of models? what level of detail? perhaps multiple levels of detail, depending ... how to develop scenarios ? and brings about many integration issues ...

We are exploring this approach in: VERTEX: Virtual Environments: from 3D Representations to Task planning and EXecution A project of Phase 3 of the Institute of Robotics and Intelligent Systems (IRIS) of the Network of Centers of Excellence program of Canada. Objective is to optimize the execution of delicate tasks by combining the accurate simulation of actual scenes, tools and processes with advanced human machine interfaces.

VERTEX On site acquisition

Geometric modeling, 3D and photometric VERTEX On site acquisition Geometric modeling, 3D and photometric Modeling Models of behaviors Models of augmented scenes Tools, materials, processes

VERTEX Planning Modeling Optimized scenarios Acquisition sur le site Planning Task simulation in “VR” mode Reactive Interaction Predictive evaluation Task decomposition Optimized scenarios Geometric modeling, 3D and photometric Modeling Models of behaviors Models of augmented scenes Tools, materials, processes

VERTEX Training Planning Modeling Simulated scenarios Acquisition sur le site Planning Optimized scenarios Task simulation in “VR” mode Reactive Interaction Predictive evaluation Task decomposition Geometric modeling, 3D and photometric Modeling Models of behaviors Models of augmented scenes Tools, materials, processes

VERTEX Execution Training Planning Modeling Task supervision & Teleoperation in augmented VR mode VERTEX Execution Real time control of robot and tools Training Simulated scenarios Planning Optimized scenarios Task simulation in “VR” mode Reactive Interaction Predictive evaluation Task decomposition On site acquisition Geometric modeling, 3D and photometric Modeling Models of behaviors Models of augmented scenes Tools, materials, processes

VERTEX Execution Training Planning Modeling Task supervision & Teleoperation in augmented VR mode VERTEX Execution Real time control of robot and tools Training Simulated scenarios Planning Optimized scenarios Task simulation in “VR” mode Reactive Interaction Predictive evaluation Task decomposition On site acquisition Geometric modeling, 3D and photometric Modeling Models of behaviors Models of augmented scenes Tools, materials, processes

Who is the user? Design issue: What are his / her needs? Actually, complex interventions typically involve several users, of various types, with different needs, perspectives and internal models These users, acting cooperatively, might very well be in different locations User-centric design Hix, D., Swan, E., Gabbard, J., McGee, M., Durbin, J., King, T. (1999) User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment. In Proceedings of IEEE Virtual Reality '99

AR VR VERTEX Execution: hard real-time Planning: interactive, soft real-time VR On site Acquisition Modeling: off-line

A key aspect of physics relates to the handling of time. Real time???? Design issue: In (critical) VE’s, time has many different flavors: it might just flow out of the action loop, it may relate to factors which impact upon the user’s sense of interactivity, such as latency jitter, during strategic planning activities, it blends with predictive evaluation, and during the direct, immediate {supervision. control, execution} of the task (Augmented Reality), timing accuracy is mandatory: this is the realm of hard real time. But different physical components may require different time resolution: run time optimization requires fine grain control of time.

dynamic extensibility (non - stop) The (physics) simulation engine is at the core of the system Design objectives: Beside its predictable real-time behavior, the engine should be capable of supporting: dynamic extensibility (non - stop) internal coherence, robustness precisely known degradation multi-resolution behavioral modeling

Design objectives (cont): From an implementation point of view, the engine should seek modularity, reusability networked deployment (multiple users, geographica extent of tasks) capability to operate from heterogeneouscomponents with run-time binding close match to current and foreseeable trends - Moore’s law - high-speed networking

Implementation choice: Use the Common Object Request Broker Architecture (CORBA) as the software bus - the glue - in assembling the Vertex system. Why? CORBA is “heavy”, with significant overhead ... True, but as time will unfold complexity of relevant problems CPU, network performance will both and and the benefits of a robust architecture  a more and more significant asset. Silicon, bandwidth  free!

* APIA Actors * Properties * Interactions Architecture network To insure time accuracy and conformity to physics, we locate the main driving loop in the simulation engine, “away” from the HMI component. *

Physics engine APIA network Maintains an on-going representation of the “world” A Lego-like approach, with hierarchical capabilities Implemented on a cluster of COTS (à la Beowulf) Runs (preferably) on hard real-time OS (OS’s) May include heterogeneous components

Controller APIA network Overall management Scenario authoring Repository of model objects …

Sensors & APIA Actuators network I/O links to the actual physical world

HMI’s APIA network Multiple and different views / interactions easily implemented To suit the representation levels required by different types of users

APIA network Provides the physical glue between the components Geographically - distributed computing, users RT-CORBA provides the logical glue Designed to fully exploit high performance networking QOS, CaNet3

CA*net3 IPv6

Implementation choice Vertex uses ACE™ and TAO™, a CORBA implementation under development at the Center for Distributed Object Computing, Washington University. ACE / TAO is designed to support real-time networked applications, with rigorous control of task priorities and QOS. Douglas C. Schmidt, Center for Distributed Object Computing,Washington University

minimally invasive surgery shares many aspects of telerobotics, A generic approach ... VERTEX / APIA is currently being deployed in other areas, such as breast and liver cancer treatment through cryosurgery. minimally invasive surgery shares many aspects of telerobotics, a collaborative project with the Imaging Research Unit of Hopital St-François d’Assise (Dr. C. Moisan) and the Finite Element Research Group at Laval.

VERTEX Execution Training Planning Modeling Task Supervision & Teleoperation in AR mode Training Real-Time Cryogenic Probe Control Simulated scenarios Planning Optimized Scenarios Task Simulation in VR mode Reactive Interaction & Predictive Evaluation Task Decomposition MRI Acquisition 3D Geometrical and Tissue Modeling Models of Augmented Scenes Behavior Modeling Modeling Tools, Materials and Processes

Real-time display of (simulated) cold front spatial distribution On-going 3D visualization of cryoprobe location and orientation

Current status and future work Video