1. SSGRR International Conference on Advances in Infrastructure for Electronic Business, Science and Education on the Internet,
July 31 - August 6, L’Aquila, Italy

2. Designing Virtual Environments for Critical Transactions and Collaborative Interventions: the VERTEX / APIA Framework for Networked, Physics-Compliant Objects
Denis Poussart
Denis Laurendeau
François Bernier
Martin Simoneau
Nathalie Harrison
Denis Ouellet
Christian Moisan
Computer Vision and Systems Laboratory and
Interventional MRI Unit, CHUQ, Université Laval
Québec, CANADA
Supported, in part, by grants from NSERC, FCAR, the Institute for Robotics and Intelligent Systems and the Canadian Foundation for Innovation

3. Critical Interventions represent cases where
errors, delays, lack of optimization}
may have very negative consequences
for safety
for the environment
for health
for costs …
In the future, as more and more complex situations arise, we may anticipate that operational support from Virtual Environments will become paramount and prevalent in the
planning
training
execution phases of delicate tasks

4. The inspection, maintenance and repair of hydroelectric facilities is just one example

5. Virtualizing “reality”
Human
Machine
Interaction
Computer
Graphics
Computer
Vision
Automated
generation of
geometry of objects
and scenes

6. But something is missing ...
For critical tasks, visual illusion is not sufficient.
There is more to “real things” than just shape, or forms, even if they are augmented with some “behaviors”.
Reality includes PHYSICS!
Accurate physical modeling, laws, and simulation capabilities must be integrated within the virtual environment.

7. Virtualizing “reality”
Human
Machine
Interaction
Simulation
Physics
actual world
Computer
Graphics
Computer
Vision

8. This opens up a huge question space!
what is relevant to be physically modeled?
what would be appropriate forms of models?
what level of detail?
perhaps multiple levels of detail, depending ...
how to develop scenarios ?
and brings about many integration issues ...

9. We are exploring this approach in:
VERTEX: Virtual Environments: from 3D Representations to Task planning and EXecution
A project of Phase 3 of the Institute of Robotics and Intelligent Systems (IRIS) of the Network of Centers of Excellence program of Canada.
Objective is to optimize the execution of delicate tasks by combining the accurate simulation of actual scenes, tools and processes with advanced human machine interfaces.

10. VERTEX
On site acquisition

11. Geometric modeling, 3D and photometric
VERTEX
On site acquisition
Geometric modeling, 3D and photometric
Modeling
Models of behaviors
Models of augmented scenes
Tools, materials, processes

12. VERTEX Planning Modeling Optimized scenarios
Acquisition sur le site
Planning
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Optimized scenarios
Geometric modeling, 3D and photometric
Modeling
Models of behaviors
Models of augmented scenes
Tools, materials, processes

13. VERTEX Training Planning Modeling Simulated scenarios
Acquisition sur le site
Planning
Optimized scenarios
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
Geometric modeling, 3D and photometric
Modeling
Models of behaviors
Models of augmented scenes
Tools, materials, processes

14. VERTEX Execution Training Planning Modeling
Task supervision & Teleoperation in augmented VR mode
VERTEX
Execution
Real time control of robot and tools
Training
Simulated scenarios
Planning
Optimized scenarios
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
On site acquisition
Geometric modeling, 3D and photometric
Modeling
Models of behaviors
Models of augmented scenes
Tools, materials, processes

15. VERTEX Execution Training Planning Modeling
Task supervision & Teleoperation in augmented VR mode
VERTEX
Execution
Real time control of robot and tools
Training
Simulated scenarios
Planning
Optimized scenarios
Task simulation in “VR” mode
Reactive Interaction Predictive evaluation
Task decomposition
On site acquisition
Geometric modeling, 3D and photometric
Modeling
Models of behaviors
Models of augmented scenes
Tools, materials, processes

16. Who is the user? Design issue: What are his / her needs?
Actually, complex interventions typically involve several users, of various types, with different needs, perspectives and internal models
These users, acting cooperatively, might very well be in different locations
User-centric design
Hix, D., Swan, E., Gabbard, J., McGee, M., Durbin, J., King, T. (1999) User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment. In Proceedings of IEEE Virtual Reality '99

17. AR VR VERTEX Execution: hard real-time
Planning: interactive, soft real-time VR
On site Acquisition
Modeling: off-line

18. A key aspect of physics relates to the handling of time. Real time????
Design issue:
In (critical) VE’s, time has many different flavors:
it might just flow out of the action loop,
it may relate to factors which impact upon the user’s sense of interactivity, such as latency jitter,
during strategic planning activities, it blends with predictive evaluation,
and during the direct, immediate {supervision. control, execution} of the task (Augmented Reality), timing accuracy is mandatory: this is the realm of hard real time.
But different physical components may require different time resolution: run time optimization requires fine grain control of time.

19. dynamic extensibility (non - stop)
The (physics) simulation engine is at the core of the system
Design objectives:
Beside its predictable real-time behavior, the engine should be capable of supporting:
dynamic extensibility (non - stop)
internal coherence, robustness
precisely known degradation
multi-resolution behavioral modeling

20. Design objectives (cont):
From an implementation point of view, the engine should seek
modularity, reusability
networked deployment (multiple users, geographica extent of tasks)
capability to operate from heterogeneouscomponents with run-time binding
close match to current and foreseeable trends
- Moore’s law
- high-speed networking

21. Implementation choice:
Use the Common Object Request Broker Architecture (CORBA) as the software bus - the glue - in assembling the Vertex system.
Why? CORBA is “heavy”, with significant overhead ...
True, but as time will unfold
complexity of relevant problems
CPU, network performance
will both
and
and the benefits of a robust architecture  a more and more significant asset.
Silicon, bandwidth  free!

22. * APIA Actors * Properties * Interactions Architecture network
To insure time accuracy and conformity to physics, we locate the main driving loop in the simulation engine, “away” from the HMI component. *

23. Physics engine APIA network
Maintains an on-going representation of the “world”
A Lego-like approach, with hierarchical capabilities
Implemented on a cluster of COTS (à la Beowulf)
Runs (preferably) on hard real-time OS (OS’s)
May include heterogeneous components

24. Controller APIA network Overall management Scenario authoring
Repository of model objects …

25. Sensors & APIA Actuators network
I/O links to the actual physical world

26. HMI’s
APIA
network
Multiple and different views / interactions easily implemented
To suit the representation levels required by different types of users

27. APIA network Provides the physical glue between the components
Geographically - distributed computing, users
RT-CORBA provides the logical glue
Designed to fully exploit high performance networking QOS, CaNet3

28. CA*net3 IPv6

29. Implementation choice
Vertex uses ACE™ and TAO™, a CORBA
implementation under development at the Center for Distributed Object Computing,
Washington University.
ACE / TAO is designed to support real-time networked applications, with rigorous control of task priorities and QOS.
Douglas C. Schmidt, Center for
Distributed Object Computing,Washington University

30. minimally invasive surgery shares many aspects of telerobotics,
A generic approach ...
VERTEX / APIA is currently being deployed in other areas, such as breast and liver cancer treatment through cryosurgery.
minimally invasive surgery shares many aspects of telerobotics,
a collaborative project with the Imaging Research Unit of Hopital St-François d’Assise (Dr. C. Moisan) and the Finite Element Research Group at Laval.

31. VERTEX Execution Training Planning Modeling
Task Supervision & Teleoperation in AR mode
Training
Real-Time Cryogenic Probe Control
Simulated scenarios
Planning
Optimized Scenarios
Task Simulation in VR mode
Reactive Interaction & Predictive Evaluation
Task Decomposition
MRI Acquisition
3D Geometrical and Tissue Modeling
Models of Augmented Scenes
Behavior Modeling
Modeling
Tools, Materials and Processes

32. Real-time display of (simulated) cold front spatial distribution
On-going 3D visualization of cryoprobe location and orientation

33. Current status and future work
Video