1. PhD Defense Emanuele Ruffaldi
7 July 2006
Multirate and Perceptual Techniques for Haptic Rendering in Virtual Environments
PhD Defense
Emanuele Ruffaldi
I am going to present part of my research activity performed in the last four months in the BioRobotics Lab at Stanford. In particular I am going to present the aspects related to the Benchmarking of Haptic Rendering Algorithms in its current status and the forthcoming application for the haptic rendering of voxel based models.
Scuola Superiore S.Anna, PERCRO

2. Haptic Interaction

3. Haptic Interfaces Robotic systems that provide
kinesthetic and tactile stimuli

4. Haptics in Human Machine Interaction
Classic Robot
Teleoperated System
Networked Teleoperation
Haptics in Virtual Environment
Haptic Collaborative
Virtual Environment
Augmented Reality with Haptics

5. Haptics in Virtual Environments
Realism of Interaction is not a requirement
Integration with advanced visualization techniques and technologies
Haptics fits in the Multimodal approach to VE
Visual
Haptic
Physical
Action
Spatial

6. Kinesthetic Interaction
Generate a force feedback depending on the interaction of the user in the Virtual Environment
The user handles the device with a stylus or constraints its fingers
The interaction in the Virtual Environment is tool mediated, we call this tool the Haptic Handle, usually a sphere
2 point 3-DOF
device
3-DOF
device
How compute the force
depending on the position of the haptic contact point?
It is the objective of Haptic Rendering

7. 3-DOF Haptic Rendering
In 3-DOF Haptic Rendering we represent the contact point as a small sphere and we suppose it is able to move around in space
The force is obtained by computing a proxy point that is always outside the surface (Zilles95, Ruspini97)
Surface rendering can be enhanced by friction, force shading and textures
The performance of a 3-DOF rendering depends on the way the algorithm computes the intersection with the geometry
3-DOF rendering can be used for exploring surfaces, pushing objects, perceiving force fields
A device with Two 3-DOF points can be used for grasping objects

8. 6-DOF Haptic Rendering
3-DOF is not adequate when the Haptic Handle has a complex shape and in specific tasks as Virtual Prototyping
6-DOF rendering can be used with 3-DOF devices for simulating complex interactions
Otaduy 2003
Haption 2005
Higher complexity in the Collision Detection
Compute force and torque at the contacting points
Involves a Dynamic Simulation

9. Virtual Coupling
The force computed by the 6-DOF Collision Response could be applied directly to the body (Direct Rendering)
The varying number of contact points would produce a varying stiffness
Performance depends on the speed of the Collision Detection
Virtual Coupling overcomes these problems by smoothing the interaction (Colgate95, McNeeily99)
In VC the body and the haptic handle are connected by a damped 6-DOF spring
Haptic Handle
Virtual Body

10. Decomposing Haptic Systems
Haptic Systems can be decomposed using a layered approach similar to the OSI stack for networked applications. Research topics can be described relative to this layering
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Data
Transport
Network
Session
Presentation
Application
Physical
Haptic Stack
OSI Stack

11. Haptic Research New Devices Haptic Rendering for 6-DOF
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
New Devices
Working Tactile device
Haptic Rendering for 6-DOF
What is missing? High resolution
meshes handling
Multi-finger Haptics
Realistic models for grasping
Deformable Haptics
Domain specific deformable models
Integration with applications
Medical simulators and games
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Esperimenti Johannson, Lederman, Choi => potere aggiunto multinger > 2
Albero tutti I temi di ricerca storico, riorganizzato
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware

12. Motivations of the Work
Improve the realism of Haptic interaction for Object Manipulation
6-DOF handling independent of the geometry
3+3-DOF realistic grasping of objects
Evaluating the results using benchmarking

13. Claims
A complete solution for the manipulation of objects using Haptic interfaces
A Soft-Finger Proxy for grasping objects using two finger 3-DOF
A 6-DOF Volume Based algorithm for Collision Detection and Response
Technology integration for easy development of Haptic applications

14. Soft-finger Grasping
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Concept: create an object grasping model based on the concept of human finger compliance. Provide such representation by introducing in the VE the fingertip deformability (soft-finger)
Objective: provide and validate a realistic grasping of virtual objects using the Haptic interface based on the soft-finger model
modeling
Robotic Grasping
Haptic Grasping
Human Grasping
Sistemare mano e reference
perception
The perceptual work starts from early research in cognitive sciences.
The mathematical models come from the
problem of design robotic hands
Friction assessment of the human fingertip grasping, Johansson 99
DLR 2005

15. Haptic Soft-finger proxy
The finger interaction is simulated using two 3-DOF proxy enhanced with linear and rotational friction
The force feedback computed on the proxy is sent to the Dynamic Simulator for balancing the linear and rotational sliding due to gravity
The device used is a 3-DOF device with two arms (GRAB) but the algorithm is ready for an evolution of the device providing torques

16. Rotational Friction Cone
The classic friction cone algorithm uses the proxy position for simulating stick-slip friction
The algorithm has been enhanced with the rotational component
The linear condition on the position is a coupled condition on the relative position and rotation
It allows to take into account the real coupling between the two types of friction in human grasping

17. Simulating Object Lifting in VE
An experiment in which users has been asked to lift object with different weights and stiffness coefficients
The Haptic interface allows a simulation of the operation that is coherent with the physical parameters
But the security margin of the forces applied by the user are much higher than in the real case
Because the current haptic interface do not transmit information about the contact area
Smaller stiffness generates forces that are easier to be controlled by the Human, but the surface is less realistic
A possible future

18. Improvement respect classic algorithm
The coupled algorithm
allows the combined
behavior of
sliding and rotation in the
range L < L0
It allows the simulation of a real behavior of an object sliding and rotating between the fingers
Without the algorithm the objects just slides
rotation
rot-slide
L0 is the contact area

19. Simulation of the Grasping
The algorithm has been implemented in a real system and tested also using Simulink
The figure shows how the coupled solution allows the simulation of the rotation during the sliding

20. Multirate architecture for the Soft-Finger
The system is able to provide stable interaction for the grasping with multi-finger computation of the forces.
This Haptic library for Virtual Environments written in C++ has been named EHAP

21. Example Experiment

22. Point Rendering Limitations
Point rendering is useful for exploration of surfaces (PureForm 01)
Point rendering is fast (AABB tree or Local model)
But it is limited when the interaction tool is more complicated (McNeeily99)
Also certain applications, like Virtual Prototyping are not possible with point rendering

23. 6-DOF Rendering aspects

24. 6-DOF Architecture Collision Detection Collision detection
Contact Management
300 kHz
Haptic
device
Haptic Rendering
Control
Algorithms
Virtual
Coupling
X, R
Fr, Tr
Simulation
Contact
response
Rigid Body
X,R
Fc, Tc
F,T
-Fc, -Tc
1 kHz C

25. Motivations of the Work
Provide 6-DOF simulation of teeth interaction in a planning tool for craniofacial surgical operations
Given an imaging dataset made of X-rays
Only one algorithm is in literature the Voxel Point Shell by McNeely, with some limitations
Voxel Point Shell
The point shell is a set of surface point with normals for the Haptic Tool
The Collision Detection is obtained testing each point of the shell against a static voxel world
The Collision Response uses a single voxel penetration
The number of Contact points is managed just by a mean
The algorithm quality is limited by number of samples

26. New Volume Based Collision Detection and Response Algorithm
Collision Detection based on a Voxel Volume and Implicit Sphere Tree
Global information on the collision (in VPS is local)
No dependency on point shell sampling
Any object can be tested
Collision Response for the reduction of number of points and fast computation of the dynamic simulation of the body
No mean applied
No need for braking
Final Haptic Rendering based on Virtual Coupling
Same as in VPS for smoothing the interaction
The voxel volume has been obtained from X-rays scans of teeth and voxelized by segmentation using Amira
The voxelization of mesh models uses a simple flood filling algorithm
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware

27. Implicit Sphere Tree Octree Node Occupancy optimization
The Voxel Volume is stored in a Octree
Information per Voxel and per Node can be selected at compile time depending on tradeoff between memory and speed:
Voxel Type (free, full, surface, proximity)
Distance to surface
Normal/Gradient
Normal Cone (additional Collision Detection optimization)
The CD Bounding Volume Hierarchy Is computed implicitly from the Octree
No memory usage
Additional optimizations provided by octree storage
Octree
Node Occupancy
optimization

28. Collision Response p1 n δ p0 p1 n δ p0 1 2 3 4
The Collision Response is based on an impulsive force approach that selects the best contact point, in simultaneous mode without requiring another CD
Contact points are excluded if separating
This approach removes the need for clustering
The impulse is applied using the formula (frictionless in this case): 1 2 3 4 p1 n δ p0 p1 n δ p0
Resolution of contacts by two impulses
(not) separating contact points

29. Collision Detection and Response

30. Visualization by Q-Splat
For the completeness of the system the visualization as been addressed
Possible solution s:
Mesh from the isosurface using marching cubes (~ 1M triangles for the 2563 teeth)
Volumetric rendering based on GPU (3D texture and advanced use of shaders)
Q-Splat (introduced for rendering meshes of > 100M triangles)
Q-Splat is a point based technique that replaces every voxel with a single screen aligned quad or ellipse.
It provided level of detail rendering using the distance of the octree cube from the viewer and replace all the octree node with a single splat
It has been chosen because it uses the Octree and the Normal cones of the Voxel Volume geometry
Inserire riferimento paper qsplat
Coupled rendering using CPU and some GPU
It can be used to visualize the current status of each voxel/node during the collision

31. Benchmarking Algorithms should be measured and evaluated
How this can be done with Haptics?
Proposing a Benchmarking Framework for Haptic Rendering algorithm based on real measurements
In this first phase the algorithm addresses 3-DOF rendering

32. Benchmarking Structure
3D model
out trajectory
traj
scan
(pos + force)
in trajectory
result
timings
Haptic renderer
comparer
extract

33. Results of Benchmarking
Evaluation of relative performance of algorithms
Quality of rendering
Computational performance
Identification of errors

34. Developing with Haptics
Requires skills on device and algorithms
Integration with existing software is challenging
Sometimes the developer is just a psychologist that want to perform an experiment
Conceptually should be just little harder than 3D graphics
Objective - Improve Haptics from the point of view of the Platform and Development Tools

35. Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
HapticWeb
A system for developing Haptic applications on the Web or in VR installations
Based on the eXtreme Virtual Reality (XVR) technology
Providing low level Haptic Rendering and Dynamic Simulating
Plus higher level interface for integrating 3D graphics and haptics

36. EHAP HapticWeb applications
ENACTIVE 06
Pool for ENACTIVE 06
EHAP HapticWeb applications
Next slide, HapticWeb. Dove e’ stato usato: CREATE, VICOM, Enactive
Architectural contribution to
PERCRO projects
CREATE 2004
VICOM 2006

37. Scripting the Haptic Rendering in Applications
position
graphics loop
native loop 1kHz
force
position
graphics loop
force script
loop ~1kHz
force
position
graphics loop
force script
loop ~1kHz
native
loop 1kHz
plane
force

38. Example of Pool Simulator
Device Driver
Device Abstraction
Haptic Rendering
Dynamic Simulation
GUI and Effects
Application
Hardware
Example of Pool Simulator

39. Realistic Physics
Based on the Novodex Dynamic Engine

40. Future challenges Enhance the proposed algorithm to deformable objects
Enhance the technological solution using a GPU based approaches
Semantic Haptic Interaction, moving from low level Haptic Rendering to semantic manipulation of objects in VE
Visual Programming with Haptic behavior, taking into accounts the perceptual properties of Haptics

41. Conclusions
This work has addressed one of the typical aspect of Human interaction. Manipulation of objects.
A new realistic soft-finger proxy for rendering the grasping based on coupled linear and rotational friction
The introduction of a new Voxel Volume Collision Detection and Response algorithm for 6-DOF rendering, with high performance
A framework for the Benchmarking of 3-DOF haptic rendering algorithm based on real measurements
Finally HapticWeb, a system for the development of haptic enabled application, for VR and desktop systems. Provided to PERCRO and to external developers

42. Acknowledgments Thanks to all PERCRO people
and nothing could be possible and meaningful without Elisabetta
This Thesis is dedicated to my Parents