1. Introduction to Haptic Rendering Ming C. Lin lin@cs.unc.edu http://gamma.cs.unc.edu/interactive

2. UNC Chapel Hill M. C. Lin What Is Haptic Rendering? Simulation Haptic Device Robot Virtual Reality Master-Slave Systems Human Force Feedback Tactile Feedback Human-in-the-Loop Pictures: http://haptic.mech.nwu.edu/intro/gallery/

3. UNC Chapel Hill M. C. Lin Why Haptics? Natural 3D interaction with a simulated environment. Not constrained by a 2D projection. Additional sensorial input. Visual is dominant. Exploit extra dimensions, ‘cheat the user through the eyes’… In case of inconsistencies, visual wins.

4. UNC Chapel Hill M. C. Lin Inter-disciplinary Research Computer Science Electrical Engineering Mechanical Engineering Haptic RenderingControl and actuators Mechanical design Computation of the forces output by the device

5. UNC Chapel Hill M. C. Lin To be covered Haptic rendering for pen-type devices: 3DOF Haptics (3D force output) 6DOF Haptics (3D force + torque output) Point-Object interaction Object-Object interaction DOF = Degree of Freedom

6. UNC Chapel Hill M. C. Lin Some History Argonne ’54, first master-slave systems. Master=slave. Salisbury ’80, independent master and slave. Cartesian control of robot arm. Later, interaction with computer simulated slave (VR). GROPE project UNC ’67-’90, molecular docking. Minsky ’90, the Sandpaper. Massie & Salisbury ’94, the Phantom. Early 90’s to ’97, 3DOF haptics. Late 90’s to today, 6DOF haptics.

7. UNC Chapel Hill M. C. Lin Control of Haptic Devices Impedance devices Admittance devices User Actuators SimulationUser Actuators Simulation F x*x x, v F*F Collision detection + contact response model More popular Collision detection + position constraints Bulkier and more expensive Very good for virtual walls

8. UNC Chapel Hill M. C. Lin Control of Haptic Devices Impedance Devices Admittance devices 6-DOF PhantomCOBOTs

9. UNC Chapel Hill M. C. Lin 1KHz Performance Requirement The user becomes part of the simulation loop. 1KHz is necessary so that the whole system doesn’t suffer from disturbing oscillations. Think of the analogy with numerical integration of a system with spring, mass and damper, where the frequency of the haptic loop sets the integration step. The Phantom haptic devices run their control loop at 1KHz. Consequence: we are very limited on the amount of computation that we can do.

10. UNC Chapel Hill M. C. Lin 3DOF Haptics: Intro Output: 3D force  3DOF haptics Limited to applications where point- object interaction is enough. –Haptic visualization of data –Painting and sculpting –Some medical applications Object-object Point-object

11. UNC Chapel Hill M. C. Lin 3DOF Haptics: Basic approach Check if point penetrates an object. Find closest point on the surface. Penalty-based force. x F

12. UNC Chapel Hill M. C. Lin 3DOF Haptics: The problems Force discontinuities when crossing boundaries of internal Voronoi cells. F1F1 F2F2 Unexpected force discontinuities (both in magnitude and direction) are very disturbing!

13. UNC Chapel Hill M. C. Lin 3DOF Haptics: The problems Pop-through thin objects. After the mid line is crossed, the force helps popping through. motion

14. UNC Chapel Hill M. C. Lin 3DOF Haptics: Point-to-plane Mark et al., 1996. For distributed applications. The simulator sends the equation of a plane to the haptic loop. The update of the plane is asynchronous. Forces are computed between the haptic point and the plane. Possible stiffness is limited, because too stiff would bring a jerky behavior at low update rates.

15. UNC Chapel Hill M. C. Lin 3DOF Haptics: God-object Zilles and Salisbury, 1995. Use the position of the haptic interface point (HIP) and a set of local constraint surfaces to compute the position of god- object (GO). Constraint surfaces defined using heuristics. Compute GO as the point that minimizes the distance from HIP to the constraint surfaces. Lagrange multipliers.

16. UNC Chapel Hill M. C. Lin 3DOF Haptics: God-object Constraint surfaces: –Surfaces impeding motion –GO is outside (orientation test) and in the extension of the surface. –The HIP is inside the surface. motion edge/vertex Concavity: 2/3 constraint planes in 3D motion

17. UNC Chapel Hill M. C. Lin 3DOF Haptics: God-object Constraint plane equations: Energy function that will account for the distance. Define cost function using Lagrange multipliers. Minimize, solving for x, y, z, 1, 2 and 3. 3 planes at most

18. UNC Chapel Hill M. C. Lin 3DOF Haptics: God-object Partial derivatives on x, y, z, 1, 2 and 3 yield 6 linear equations: In case of less than 3 planes, the problem has a lower dimension.

19. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy Ruspini et al., 1997. Based on god-object. Virtual proxy is a small sphere, instead of a point. Use configuration-space obstacles (C-obstacles), from robotics. More formal definition of constraint planes. Implementation of additional features, based on relocation of the virtual proxy.

20. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy C-obstacles: for a spherical object, is reduced to computing offset surfaces at a distance equal to the radius of the sphere. Check the HIP against the offset surface. This is done to avoid problems with small gaps in the mesh.

21. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy Finding the virtual proxy is based on an iterative search. Basically, find subgoals based on the same distance minimization as for the god-object. At each subgoal, all the planes that go through that point are potential constraints. The minimum set of active constraints is selected. If the subgoal is in free space, set as new subgoal the HIP. The path might intersect the C-obstacles. Add the first plane intersected as a constraint and the intersection point as the current subgoal. The process ends when the virtual proxy becomes stable.

22. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) Perform collision detection between the path of the HIP and the C-obstacles HIP(t+  t)

23. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) Set the subgoal and the constraint plane(s) HIP(t+  t)

24. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) HIP(t+  t) Find a new subgoal using the active planes and the minimization based on Lagrange multipliers

25. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) Since the subgoal is in free space, drop the constraints, set the HIP as the new subgoal and perform collision detection between the path and the C-obstacles HIP(t+  t)

26. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) Recompute subgoal with new constraints HIP(t+  t)

27. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) The path to the new subgoal intersects another plane, so this is added to the set of constraints HIP(t+  t)

28. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy HIP(t) Compute active constraints (in 2D there are only 2) and find subgoal HIP(t+  t) For this example, this is the final position of the virtual proxy

29. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy Quadratic programming approach: –The constraint planes define an open convex region (bounded by the plane at infinity). –The function to minimize is the distance from the haptic device (HIP) to the new subgoal (VP i+1 ): –The translation from the current location to the new subgoal cannot intersect the constraint planes. Define linear constraints based on the normals of the planes. Quadratic function Linear constraints

30. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy Special case: simplified approach. –Let’s look at the convex region defined by the constraint planes Plane at  Vertex Edge Face (all surrounded by the plane at  )

31. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy –Normals (n) become points (-n) in a dual space (Gauss map). –The plane at infinity is the origin (o). –Points (-n) are joint together if the associated planes intersect. –The position of the virtual proxy also has a dual: –The vertices of the closest feature to P are the duals of the active constraint planes that are used for the minimization with Lagrange multipliers.

32. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy

33. UNC Chapel Hill M. C. Lin 3DOF Haptics: Virtual proxy Force output: PD (proportional- derivative) control. Produces a force that will try to keep the VP and the HIP at the same position. User Simulation VP PD controlActuators F*F HIP It’s like a spring+damper, but the authors look at it from a control engineering approach

34. UNC Chapel Hill M. C. Lin 3DOF Haptics: Additional features Force shading by Basdogan et al., 1997. –Interpolate per-vertex normals using baricentric coordinates (same as Gouraud shading) –Effect: edges and vertices seem ‘rounded’ A1A1 A2A2 A3A3 v2v2 v1v1 v3v3

35. UNC Chapel Hill M. C. Lin 3DOF Haptics: Additional features Force shading by Ruspini et al., 1997. –Modify the position of the VP. –A subgoal (HIP’) is computed changing the normal of the plane. –This subgoal replaces the HIP, and the final VP is computed. HIP HIP’ VP Original plane Force-shaded plane

36. UNC Chapel Hill M. C. Lin 3DOF Haptics: Additional features Other effects by Ruspini et al., 1997. –Friction. –Textures. –Deformation. All of them are achieved performing operations that modify the position of the VP.

37. UNC Chapel Hill M. C. Lin 3DOF Haptics: Additional features Friction by Hayward and Armstrong, 2000. –Model friction as an elasticity between the actual contact point (x) and a fictitious point, called adhesion point (w). –The adhesion point remains static or follows the actual contact depending on the ‘state’ (sliding, static…) xw

38. UNC Chapel Hill M. C. Lin 3DOF Haptics: H-Collide What about the collision detection? –Spatial uniform decomposition of space. Locate the path traversed by the probe in that grid, using a hash table. –Test the path against an OBBTree. Optimized test. –Use frame-to-frame coherence, caching the previous contact, and perform incremental computation. –When the leaf intersects, compute the surface contact point (SCP). Pass this to GHOST (haptic rendering library used by the Phantoms).

39. UNC Chapel Hill M. C. Lin 3DOF Haptics: H-Collide Spatial partitioning with hash table Ray Vs. OBBTree test SCP computation SCP

40. UNC Chapel Hill M. C. Lin 6DOF Haptics: Intro Output: 3D force + 3D torque For applications related to manipulation. –Assembly and maintenance oriented design. Removal of parts from complex structures. Typical problem: peg-in-the-hole. There is a net torque

41. UNC Chapel Hill M. C. Lin 6DOF Haptics: Intro Formulate the problem as a rigid body simulation problem and output to the user the forces exerted on the object. Why not? –Too expensive –Constantly facing a situation of resting contacts. –Too much time performing collision detection to find non-penetrating situations. –Solve the linear complementarity problem. –Theoretically, it seems like the natural extension of the virtual proxy, though.

42. UNC Chapel Hill M. C. Lin 6DOF Haptics: Intro The position of the center of mass (com) and the orientation cannot be governed directly by the haptic device. Interpenetrations would be unavoidable. com Simulated object (x s, q s ) com Haptic device (x h, q h ) com

43. UNC Chapel Hill M. C. Lin 6DOF Haptics: Contacts We cannot afford to compute ‘exact’ contacts. Allow tolerances. Consider as valid contact whatever is inside a distance tolerance. Outward offset surfaces are better. Distance is cheaper to compute than penetration. Contact volumeContact areaContact points (e.g. local minima of distance function)

44. UNC Chapel Hill M. C. Lin 6DOF Haptics: Major approaches All the approaches suffer from the fact that it’s an output-dependant problem. –Point cloud Vs. voxelized model. It can be linear on the number of points. The number of points grows quadratically with the resolution. –Polygonal models. At least linear on the number of convex pairs inside the tolerance. –NURBS. Very complex exact distance computation between NURBS surfaces.

45. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling McNeely et al., 1999. Voxelize all the models in the scene. For the haptic object, represent each voxel by a point  Point Shell. Test points against voxels of the scene. Compute force at each point that penetrates an object. Add up all the force and torque, and apply them to the user through virtual coupling.

46. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling All the objects are voxelized. For the haptic object, store points at the center of the voxels, with inward normals determined by the polygonal model. For the objects in the scene, create the voxmap, a 512-tree with 3 levels (the 512-tree is an extension of the octree, with 512 children per node, cube of 8*8*8). The 512-tree optimizes memory. The initial implementation is only for static objects.

47. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Original objectVoxelsPointshell

48. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling All the voxels in the voxmap store info about their position in the object: interior, surface, proximity or free space. interior surface proximity free space

49. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling To start the collision query, all the pointshell has to be transformed to the local coordinates of the voxelized objects. Points that are in the surface or proximity voxels are considered as contacts.

50. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Forces at each contact are computed according to the tangent-plane force model. Contacts are valid if d<0 (penetration). If penetration to the interior happens, the contact is missed. One might think of computing a distance field in the interior, but the distance depends on the normal N. d Take the plane with normal N that goes through the center of the voxel. Compute d as the distance from the point to that plane. N

51. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling The force is continuous in direction, but discontinuous in magnitude. The virtual coupling (see later) is supposed to filter these discontinuities. d2d2 d1  d2 The contact might even show up or disappear suddenly N d1d1 motion

52. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Force per contact: Add all the forces together. Several contacts act as a single contact with higher stiffness. Limit the total stiffness: There can still be problems of irregular distribution of stiffness. If n<10If n>10

53. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Irregular distribution of stiffness. Geometry-driven instabilities F 2 =F 1  d 2 =5*d 1 Risk of penetration! F 1 =5*k*d 1 F 2 =k*d 2 1 contact 5 contacts Object bounces from one side to the other. More common due to torque on long objects.

54. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Penetration has to be avoided. Pre-contact braking forces. Strong viscous force applied in the proximity layer. The viscosity is adjusted so that all the kinetic energy is dissipated in a distance equal to the width of the layer.

55. UNC Chapel Hill M. C. Lin 6DOF Haptics: Voxel sampling Virtual coupling, see Adams and Hannaford, 1998. Spring-damper between the position and orientation of the haptic device and the position and orientation of the object. Filters discontinuities, to enhance stability. We can run into numerical integration problems, though.

56. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models Gregory et al., 2000; Kim et al., 2002. Convex decomposition of polygonal models. Collision detection (modified SWIFT++), returning all pairs of convex pieces within a tolerance. Handle shallow penetrations. Cluster contacts. Combination of penalty-based restoring forces and impulsive response.

57. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models SWIFT++, by Ehmann and Lin, 2001. Hierarchy of convex pieces. Voronoi marching between pairs of pieces. Travel down if the distance is less than the tolerance.

58. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models Modifications to SWIFT++: –Return all pairs of pieces inside the tolerance (group that contains the set of local minima of the distance function). –Handle shallow penetrations, computing penetration depth between convex pieces. Check for pairs of features with overlapping normals.

59. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models Convex decomposition: –Hard problem. The decomposition might not be efficient. Explosion of pairs O(n 4 ). –Surface decomposition  Convex patches, completed with virtual faces. Linear size. Solid decompositionSurface decomposition

60. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models Surface decomposition good for finding global minimum in disjoint situation. Unique pair of features, both original. Problems for local minima and penetration. The features might be virtual. If a single face is a convex piece itself, penetrating contacts might be missed.

61. UNC Chapel Hill M. C. Lin 6DOF Haptics: Polygonal models Cluster contacts. Find unique contact (distance, normal and point), averaging contacts in a neighborhood. Weighted average, using distance as the weight. This solves the problem of the irregular distribution of stiffness. Pretty smooth varying forces in practice, without need of virtual coupling. Additional features (texture, friction…) can be rendered in a per-contact basis.

62. UNC Chapel Hill M. C. Lin 6DOF Haptics: Unsolved problems Generalization of Virtual Proxy to 6-DOF haptics Haptic Texture Synthesis Applications –Endoscopic sinus surgery –3D Painting with deformable brushes –3D deformable objects

63. UNC Chapel Hill M. C. Lin References R. J. Adams and B. Hannaford. A Two-port Framework for the Design of Unconditionally Stable Haptic Interfaces. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1998. C. Basdogan, C. H. Ho and M. Srinivasan. A Ray-based Haptic Rendering Technique for Displaying Shape and Texture of 3D Objects in Virtual Environments. In the Proc. of ASME Dynamic Systems and Control Division, 1997. G. Burdea. Force and Touch Feedback for Virtual Reality. John Wiley and Sons, 1996. S. A. Ehmann and M. C. Lin. Accurate and Fast Proximity Queries between Polyhedra Using Surface Decomposition. In Proc. of Eurographics, 2001. A. Gregory, M. C. Lin, S. Gottschalk and R. M. Taylor II. H-Collide: A Framework for Fast and Accurate Collision Detection for Haptic Interaction. In Proc. of IEEE Virtual Reality Conference, 1999. A. Gregory, A. Mascarenhas, S. Ehmann, M. C. Lin and D. Manocha. Six Degree-of-Freedom Haptic Display of Polygonal Models. In Proc. of IEEE Visualization, 2000.

64. UNC Chapel Hill M. C. Lin References V. Hayward and B. Armstrong. A New Computational Method of Friction Applied to Haptic Rendering. In Lecture Notes in Control and Information Sciences, Vol. 250, Springer-Verlag, 2000. Y. J. Kim, M. A. Otaduy, M. C. Lin and D. Manocha. Six-Degree-of-Freedom Haptic Display Using Localized Contact Computations. To appear in Proc. of Haptic Symposium, 2002. W. R. Mark, S. C. Randolph, M. Finch, J. M. Van Verth, and R. M. Taylor II. Adding Force Feedback to Graphics Systems: Issues and Solutions. In Proc. of ACM SIGGRAPH, 1996. W. A. McNeely, K. D. Puterbaugh and J. J. Troy. Six Degree-of-Freedom Haptic Rendering Using Voxel Sampling. In Proc. of ACM SIGGRAPH, 1999. D. C. Ruspini, K. Kolarov, and O. Khatib. The Haptic Display of Complex Graphical Environments. In Proc. of ACM SIGGRAPH, 1997. C. Zilles and K. Salisbury. A Constraint-based God Object Method for Haptic Display. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1995.