PhD Defense Emanuele Ruffaldi 7 July 2006 Multirate and Perceptual Techniques for Haptic Rendering in Virtual Environments PhD Defense Emanuele Ruffaldi I am going to present part of my research activity performed in the last four months in the BioRobotics Lab at Stanford. In particular I am going to present the aspects related to the Benchmarking of Haptic Rendering Algorithms in its current status and the forthcoming application for the haptic rendering of voxel based models. Scuola Superiore S.Anna, PERCRO

Haptic Interaction

Haptic Interfaces Robotic systems that provide kinesthetic and tactile stimuli

Haptics in Human Machine Interaction Classic Robot Teleoperated System Networked Teleoperation Haptics in Virtual Environment Haptic Collaborative Virtual Environment Augmented Reality with Haptics

Haptics in Virtual Environments Realism of Interaction is not a requirement Integration with advanced visualization techniques and technologies Haptics fits in the Multimodal approach to VE Visual Haptic Physical Action Spatial

Kinesthetic Interaction Generate a force feedback depending on the interaction of the user in the Virtual Environment The user handles the device with a stylus or constraints its fingers The interaction in the Virtual Environment is tool mediated, we call this tool the Haptic Handle, usually a sphere 2 point 3-DOF device 3-DOF device How compute the force depending on the position of the haptic contact point? It is the objective of Haptic Rendering

3-DOF Haptic Rendering In 3-DOF Haptic Rendering we represent the contact point as a small sphere and we suppose it is able to move around in space The force is obtained by computing a proxy point that is always outside the surface (Zilles95, Ruspini97) Surface rendering can be enhanced by friction, force shading and textures The performance of a 3-DOF rendering depends on the way the algorithm computes the intersection with the geometry 3-DOF rendering can be used for exploring surfaces, pushing objects, perceiving force fields A device with Two 3-DOF points can be used for grasping objects

6-DOF Haptic Rendering 3-DOF is not adequate when the Haptic Handle has a complex shape and in specific tasks as Virtual Prototyping 6-DOF rendering can be used with 3-DOF devices for simulating complex interactions Otaduy 2003 Haption 2005 Higher complexity in the Collision Detection Compute force and torque at the contacting points Involves a Dynamic Simulation

Virtual Coupling The force computed by the 6-DOF Collision Response could be applied directly to the body (Direct Rendering) The varying number of contact points would produce a varying stiffness Performance depends on the speed of the Collision Detection Virtual Coupling overcomes these problems by smoothing the interaction (Colgate95, McNeeily99) In VC the body and the haptic handle are connected by a damped 6-DOF spring Haptic Handle Virtual Body

Decomposing Haptic Systems Haptic Systems can be decomposed using a layered approach similar to the OSI stack for networked applications. Research topics can be described relative to this layering Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Data Transport Network Session Presentation Application Physical Haptic Stack OSI Stack

Haptic Research New Devices Haptic Rendering for 6-DOF Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware New Devices Working Tactile device Haptic Rendering for 6-DOF What is missing? High resolution meshes handling Multi-finger Haptics Realistic models for grasping Deformable Haptics Domain specific deformable models Integration with applications Medical simulators and games Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Esperimenti Johannson, Lederman, Choi => potere aggiunto multinger > 2 Albero tutti I temi di ricerca storico, riorganizzato Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware

Motivations of the Work Improve the realism of Haptic interaction for Object Manipulation 6-DOF handling independent of the geometry 3+3-DOF realistic grasping of objects Evaluating the results using benchmarking

Claims A complete solution for the manipulation of objects using Haptic interfaces A Soft-Finger Proxy for grasping objects using two finger 3-DOF A 6-DOF Volume Based algorithm for Collision Detection and Response Technology integration for easy development of Haptic applications

Soft-finger Grasping Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Concept: create an object grasping model based on the concept of human finger compliance. Provide such representation by introducing in the VE the fingertip deformability (soft-finger) Objective: provide and validate a realistic grasping of virtual objects using the Haptic interface based on the soft-finger model modeling Robotic Grasping Haptic Grasping Human Grasping Sistemare mano e reference perception The perceptual work starts from early research in cognitive sciences. The mathematical models come from the problem of design robotic hands Friction assessment of the human fingertip grasping, Johansson 99 DLR 2005

Haptic Soft-finger proxy The finger interaction is simulated using two 3-DOF proxy enhanced with linear and rotational friction The force feedback computed on the proxy is sent to the Dynamic Simulator for balancing the linear and rotational sliding due to gravity The device used is a 3-DOF device with two arms (GRAB) but the algorithm is ready for an evolution of the device providing torques

Rotational Friction Cone The classic friction cone algorithm uses the proxy position for simulating stick-slip friction The algorithm has been enhanced with the rotational component The linear condition on the position is a coupled condition on the relative position and rotation It allows to take into account the real coupling between the two types of friction in human grasping

Simulating Object Lifting in VE An experiment in which users has been asked to lift object with different weights and stiffness coefficients The Haptic interface allows a simulation of the operation that is coherent with the physical parameters But the security margin of the forces applied by the user are much higher than in the real case Because the current haptic interface do not transmit information about the contact area Smaller stiffness generates forces that are easier to be controlled by the Human, but the surface is less realistic A possible future

Improvement respect classic algorithm The coupled algorithm allows the combined behavior of sliding and rotation in the range L < L0 It allows the simulation of a real behavior of an object sliding and rotating between the fingers Without the algorithm the objects just slides rotation rot-slide L0 is the contact area

Simulation of the Grasping The algorithm has been implemented in a real system and tested also using Simulink The figure shows how the coupled solution allows the simulation of the rotation during the sliding

Multirate architecture for the Soft-Finger The system is able to provide stable interaction for the grasping with multi-finger computation of the forces. This Haptic library for Virtual Environments written in C++ has been named EHAP

Example Experiment

Point Rendering Limitations Point rendering is useful for exploration of surfaces (PureForm 01) Point rendering is fast (AABB tree or Local model) But it is limited when the interaction tool is more complicated (McNeeily99) Also certain applications, like Virtual Prototyping are not possible with point rendering

6-DOF Rendering aspects

6-DOF Architecture Collision Detection Collision detection Contact Management 300 kHz Haptic device Haptic Rendering Control Algorithms Virtual Coupling X, R Fr, Tr Simulation Contact response Rigid Body X,R Fc, Tc F,T -Fc, -Tc 1 kHz C

Motivations of the Work Provide 6-DOF simulation of teeth interaction in a planning tool for craniofacial surgical operations Given an imaging dataset made of X-rays Only one algorithm is in literature the Voxel Point Shell by McNeely, with some limitations Voxel Point Shell The point shell is a set of surface point with normals for the Haptic Tool The Collision Detection is obtained testing each point of the shell against a static voxel world The Collision Response uses a single voxel penetration The number of Contact points is managed just by a mean The algorithm quality is limited by number of samples

New Volume Based Collision Detection and Response Algorithm Collision Detection based on a Voxel Volume and Implicit Sphere Tree Global information on the collision (in VPS is local) No dependency on point shell sampling Any object can be tested Collision Response for the reduction of number of points and fast computation of the dynamic simulation of the body No mean applied No need for braking Final Haptic Rendering based on Virtual Coupling Same as in VPS for smoothing the interaction The voxel volume has been obtained from X-rays scans of teeth and voxelized by segmentation using Amira The voxelization of mesh models uses a simple flood filling algorithm Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware

Implicit Sphere Tree Octree Node Occupancy optimization The Voxel Volume is stored in a Octree Information per Voxel and per Node can be selected at compile time depending on tradeoff between memory and speed: Voxel Type (free, full, surface, proximity) Distance to surface Normal/Gradient Normal Cone (additional Collision Detection optimization) The CD Bounding Volume Hierarchy Is computed implicitly from the Octree No memory usage Additional optimizations provided by octree storage Octree Node Occupancy optimization

Collision Response p1 n δ p0 p1 n δ p0 1 2 3 4 The Collision Response is based on an impulsive force approach that selects the best contact point, in simultaneous mode without requiring another CD Contact points are excluded if separating This approach removes the need for clustering The impulse is applied using the formula (frictionless in this case): 1 2 3 4 p1 n δ p0 p1 n δ p0 Resolution of contacts by two impulses (not) separating contact points

Collision Detection and Response

Visualization by Q-Splat For the completeness of the system the visualization as been addressed Possible solution s: Mesh from the isosurface using marching cubes (~ 1M triangles for the 2563 teeth) Volumetric rendering based on GPU (3D texture and advanced use of shaders) Q-Splat (introduced for rendering meshes of > 100M triangles) Q-Splat is a point based technique that replaces every voxel with a single screen aligned quad or ellipse. It provided level of detail rendering using the distance of the octree cube from the viewer and replace all the octree node with a single splat It has been chosen because it uses the Octree and the Normal cones of the Voxel Volume geometry Inserire riferimento paper qsplat Coupled rendering using CPU and some GPU It can be used to visualize the current status of each voxel/node during the collision

Benchmarking Algorithms should be measured and evaluated How this can be done with Haptics? Proposing a Benchmarking Framework for Haptic Rendering algorithm based on real measurements In this first phase the algorithm addresses 3-DOF rendering

Benchmarking Structure 3D model out trajectory traj scan (pos + force) in trajectory result timings Haptic renderer comparer extract

Results of Benchmarking Evaluation of relative performance of algorithms Quality of rendering Computational performance Identification of errors

Developing with Haptics Requires skills on device and algorithms Integration with existing software is challenging Sometimes the developer is just a psychologist that want to perform an experiment Conceptually should be just little harder than 3D graphics Objective - Improve Haptics from the point of view of the Platform and Development Tools

Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware HapticWeb A system for developing Haptic applications on the Web or in VR installations Based on the eXtreme Virtual Reality (XVR) technology Providing low level Haptic Rendering and Dynamic Simulating Plus higher level interface for integrating 3D graphics and haptics

EHAP HapticWeb applications ENACTIVE 06 Pool for ENACTIVE 06 EHAP HapticWeb applications Next slide, HapticWeb. Dove e’ stato usato: CREATE, VICOM, Enactive Architectural contribution to PERCRO projects CREATE 2004 VICOM 2006

Scripting the Haptic Rendering in Applications position graphics loop native loop 1kHz force position graphics loop force script loop ~1kHz force position graphics loop force script loop ~1kHz native loop 1kHz plane force

Example of Pool Simulator Device Driver Device Abstraction Haptic Rendering Dynamic Simulation GUI and Effects Application Hardware Example of Pool Simulator

Realistic Physics Based on the Novodex Dynamic Engine

Future challenges Enhance the proposed algorithm to deformable objects Enhance the technological solution using a GPU based approaches Semantic Haptic Interaction, moving from low level Haptic Rendering to semantic manipulation of objects in VE Visual Programming with Haptic behavior, taking into accounts the perceptual properties of Haptics

Conclusions This work has addressed one of the typical aspect of Human interaction. Manipulation of objects. A new realistic soft-finger proxy for rendering the grasping based on coupled linear and rotational friction The introduction of a new Voxel Volume Collision Detection and Response algorithm for 6-DOF rendering, with high performance A framework for the Benchmarking of 3-DOF haptic rendering algorithm based on real measurements Finally HapticWeb, a system for the development of haptic enabled application, for VR and desktop systems. Provided to PERCRO and to external developers

Acknowledgments Thanks to all PERCRO people and nothing could be possible and meaningful without Elisabetta This Thesis is dedicated to my Parents