Haptic Rendering Miguel A. Otaduy COMP 259 Spring 2002

What is haptics? Human-in-the-Loop Strong requirements!! Master-Slave Systems Robot Force Feedback Haptic Device Human Simulation Tactile Feedback In the beginning, haptics was thought for master-slave systems. Now, used in virtual reality. Outputs to the user: tactile feedback (in the skin), force feedback (in the tendons). An extra sensory input to the human. However, the fact that the human closes the control loop adds strong requirements (stability, etc.) to the ‘graphics’ application. Look at the arrows. The data is bi-directional. Virtual Reality Pictures: http://haptic.mech.nwu.edu/intro/gallery/

How do haptics help? Natural 3D interaction with a simulated environment. Not constrained by a 2D projection. Additional sensorial input. Visual is dominant. Exploit extra dimensions, ‘cheat the user through the eyes’… In case of inconsistencies, visual wins. Motivate on why are haptics important.

Why do I like haptics? Computer Science Electrical Engineering Mechanical Engineering Haptic Rendering Control and actuators Mechanical design I like haptic becomes it joins together different engineering disciplines. We’ll focus on the haptic rendering: the computation of the forces that are output through the device. Computation of the forces output by the device

To be covered Haptic rendering for pen-type haptic devices. Point-Object interaction Object-Object interaction 3DOF Haptics (3D force output) 6DOF Haptics (3D force + torque output) In the lecture I’ll only cover haptic rendering for pen-type haptic devices. Other types of devices may have other considerations that I’m going to ignore. There is a major division between 3DOF haptics and 6DOF haptics. The 1st one is referred to outputting a force vector in 3D to the user. The 2nd is about outputting a torque vector too. As you will see later, this dichotomy can also seen as point-object interaction Vs object-object interaction. -Techniques used. -Unsolved problems. DOF = Degree of Freedom

Some History Argonne ’54, first master-slave systems. Master=slave. Salisbury ’80, independent master and slave. Cartesian control of robot arm. Later, interaction with computer simulated slave (VR). GROPE project UNC ’67-’90, molecular docking. Minsky ’90, the Sandpaper. Massie & Salisbury ’94, the Phantom. Early 90’s to ’97, 3DOF haptics. Late 90’s to today, 6DOF haptics.

Control of Haptic Devices Impedance devices Admittance devices User x, v Simulation User F Simulation F Actuators F* x Actuators x* Two types of devices (the terms impedance and admittance come from an analogy with electronic circuits): -Impedance devices. Sense position and velocity of the device. They are used in the simulation to move an object/point/line… This object interacts with the scenario. Forces are computed and returned. Moving the actuators of the motor to produce those forces. The haptic rendering involves collision detection and a contact response model to compute the forces. It’s the most popular type of device. -Admittance devices. Sense forces and move a virtual object governed by those forces. Find position constraints and move the device according to those constraints. Very good for simulating virtual walls. Bulkier and more expensive. Collision detection + contact response model More popular Collision detection + position constraints Bulkier and more expensive Very good for virtual walls

Control of Haptic Devices Impedance devices Admittance devices Explain what COBOTs are for. my little friend, the 6DOF Phantom COBOTs

1KHz The user becomes part of the simulation loop. 1KHz is necessary so that the whole system doesn’t suffer from disturbing oscillations. Think of the analogy with numerical integration of a system with spring, mass and damper, where the frequency of the haptic loop sets the integration step. The Phantom haptic devices run their control loop at 1KHz. Consequence: we are very limited on the amount of computation that we can do.

3DOF Haptics: Intro Output: 3D force  3DOF haptics Limited to applications where point-object interaction is enough. Haptic visualization of data. Painting and sculpting. Some medical applications. Point-object What is 3DOF haptics about? When is it used? Object-object

3DOF Haptics: Basic approach Check if point penetrates an object. Find closest point on the surface. Penalty-based force. x F Here ask a question. Any guess on what problems we can have? Note that every time we are simply computing the closest point on the surface. We don’t care about what was the trajectory or anything like that.

3DOF Haptics: The problems Force discontinuities when crossing boundaries of internal Voronoi cells. Unexpected force discontinuities (both in magnitude and direction) are very disturbing! F2 F1

3DOF Haptics: The problems Pop-through thin objects. After the mid line is crossed, the force helps popping through. motion

3DOF Haptics: Point-to-plane Mark et al., 1996. For distributed applications. The simulator sends the equation of a plane to the haptic loop. The update of the plane is asynchronous. Forces are computed between the haptic point and the plane. Possible stiffness is limited, because too stiff would bring a jerky behavior at low update rates.

3DOF Haptics: God-object Zilles and Salisbury, 1995. Use the position of the haptic interface point (HIP) and a set of local constraint surfaces to compute the position of god-object (GO). Constraint surfaces defined using heuristics. Compute GO as the point that minimizes the distance from HIP to the constraint surfaces. Lagrange multipliers. Just give an outline of the algorithm.

3DOF Haptics: God-object Constraint surfaces: Surfaces impeding motion GO is outside (orientation test) and in the extension of the surface. The HIP is inside the surface. edge/vertex Concavity: 2/3 constraint planes in 3D Explain a little how the constraints are defined in each case. motion motion

3DOF Haptics: God-object Constraint plane equations: Energy function that will account for the distance. Define cost function using Lagrange multipliers. Minimize, solving for x, y, z, l1, l2 and l3. 3 planes at most

3DOF Haptics: God-object Partial derivatives on x, y, z, l1, l2 and l3 yield 6 linear equations: In case of less than 3 planes, the problem has a lower dimension.

3DOF Haptics: Virtual proxy Ruspini et al., 1997. Based on god-object. Virtual proxy is a small sphere, instead of a point. Use configuration-space obstacles (C-obstacles), from robotics. More formal definition of constraint planes. Implementation of additional features, based on relocation of the virtual proxy.

3DOF Haptics: Virtual proxy C-obstacles: for a spherical object, is reduced to computing offset surfaces at a distance equal to the radius of the sphere. Now, check the HIP against the offset surface. This is done to avoid problems with small gaps in the mesh.

3DOF Haptics: Virtual proxy Finding the virtual proxy is based on an iterative search. Basically, find subgoals based on the same distance minimization as for the god-object. At each subgoal, all the planes that go through that point are potential constraints. The minimum set of active constraints is selected. If the subgoal is in free space, set as new subgoal the HIP. The path might intersect the C-obstacles. Add the first plane intersected as a constraint and the intersection point as the current subgoal. The process ends when the virtual proxy becomes stable. Don’t read this slide. Do it along with the picture in the next slide.

3DOF Haptics: Virtual proxy HIP(t) Perform collision detection between the path of the HIP and the C-obstacles HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) Set the subgoal and the constraint plane(s) HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) Find a new subgoal using the active planes and the minimization based on Lagrange multipliers HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) Since the subgoal is in free space, drop the constraints, set the HIP as the new subgoal and perform collision detection between the path and the C-obstacles HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) Recompute subgoal with new constraints HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) The path to the new subgoal intersects another plane, so this is added to the set of constraints HIP(t+Dt)

3DOF Haptics: Virtual proxy HIP(t) Compute active constraints (in 2D there are only 2) and find subgoal For this example, this is the final position of the virtual proxy HIP(t+Dt)

3DOF Haptics: Virtual proxy Quadratic programming approach: The constraint planes define an open convex region (bounded by the plane at infinity). The function to minimize is the distance from the haptic device (HIP) to the new subgoal (VPi+1): The translation from the current location to the new subgoal cannot intersect the constraint planes. Define linear constraints based on the normals of the planes. Quadratic function How to compute the subgoals. Linear constraints

3DOF Haptics: Virtual proxy Special case: simplified approach. Let’s look at the convex region defined by the constraint planes Plane at  Plane at  Face (all surrounded by the plane at ) Vertex Edge

3DOF Haptics: Virtual proxy Normals (n) become points (-n) in a dual space (Gauss map). The plane at infinity is the origin (o). Points (-n) are joint together if the associated planes intersect. The position of the virtual proxy also has a dual: The vertices of the closest feature to P are the duals of the active constraint planes that are used for the minimization with Lagrange multipliers. I don’t completely understand what is behind this representation. I think that every triangle in the Gauss map is the normal cone formed by three planes. Then, we’re trying to locate the direction VP(I+1)-VP(I) in the Gauss map. That direction will be in one of those normal cones. Since the region is convex, the normal cone that includes the mentioned direction will define the active planes. This looks ok, but I don’t know how the plane at infinity is treated, and I don’t see why the mentioned direction is the closest point to P.

3DOF Haptics: Virtual proxy On the left, the planes. On the right, the dual space (the Gauss map).

3DOF Haptics: Virtual proxy Force output: PD (proportional- derivative) control. Produces a force that will try to keep the VP and the HIP at the same position. VP F* F Simulation PD control Actuators HIP User It’s like a spring+damper, but the authors look at it from a control engineering approach

3DOF Haptics: Additional features Force shading by Basdogan et al., 1997. Interpolate per-vertex normals using baricentric coordinates (same as Gouraud shading). Effect: edges and vertices seem ‘rounded’. v3 A1 A2 A3 v2 v1

3DOF Haptics: Additional features Force shading by Ruspini et al., 1997. Modify the position of the VP. A subgoal (HIP’) is computed changing the normal of the plane. This subgoal replaces the HIP, and the final VP is computed. Force-shaded plane VP Imagine a vertex on the right of the plane. With interpolated normals, that would change the normal to the one that is described as force shaded. The eventual effect is that the VP is closer to the vertex, so the typical discontinuity that happens at the vertex won’t take place. Original plane HIP’ HIP

3DOF Haptics: Additional features Other effects by Ruspini et al., 1997. Friction. Textures. Deformation. All of them are achieved performing operations that modify the position of the VP. Imagine a vertex on the right of the plane. With interpolated normals, that would change the normal to the one that is described as force shaded. The eventual effect is that the VP is closer to the vertex, so the typical discontinuity that happens at the vertex won’t take place.

3DOF Haptics: Additional features Friction by Hayward and Armstrong, 2000. Model friction as an elasticity between the actual contact point (x) and a fictitious point, called adhesion point (w). The adhesion point remains static or follows the actual contact depending on the ‘state’ (sliding, static…) w x Note the difference between this type of model and the friction model used by Baraff. This is a ‘cheap’ model for fast and localized computation.

3DOF Haptics: H-Collide What about the collision detection? Spatial uniform decomposition of space. Locate the path traversed by the probe in that grid, using a hash table. Test the path against an OBBTree. Optimized test. Use frame-to-frame coherence, caching the previous contact, and perform incremental computation. When the leaf intersects, compute the surface contact point (SCP). Pass this to GHOST (haptic rendering library used by the Phantoms).

3DOF Haptics: H-Collide Ray Vs. OBBTree test Spatial partitioning with hash table SCP computation SCP

6DOF Haptics: Intro Output: 3D force + 3D torque For applications related to manipulation. Assembly and maintenance oriented design. Removal of parts from complex structures. Typical problem: peg-in-the-hole. What is 3DOF haptics about? When is it used? There is a net torque

6DOF Haptics: Intro Formulate the problem as a rigid body simulation problem and output to the user the forces exerted on the object. Why not? Tooooo expensive. Constantly facing a situation of resting contacts. Too much time performing collision detection to find non-penetrating situations. Solve the linear complementarity problem. Theoretically, it seems like the natural extension of the virtual proxy, though.

Interpenetrations would be unavoidable. 6DOF Haptics: Intro Simulated object (xs, qs) com com com Haptic device (xh, qh) The position of the center of mass (com) and the orientation cannot be governed directly by the haptic device. Interpenetrations would be unavoidable.

Contact points (e.g. local minima of distance function) 6DOF Haptics: Contacts We cannot afford working with ‘exact’ contacts. Allow tolerances. Consider as valid contact whatever is inside a distance tolerance. Outward offset surfaces are better. Distance is cheaper to compute than penetration. Contact volume Contact area Contact points (e.g. local minima of distance function)

6DOF Haptics: Major approaches All the approaches suffer from the fact that it’s an output-dependant problem. Point cloud Vs. voxelized model. It can be linear on the number of points. The number of points grows quadratically with the resolution. Polygonal models. At least linear on the number of convex pairs inside the tolerance. NURBS. Very complex exact distance computation between NURBS surfaces.

6DOF Haptics: Voxel sampling McNeely et al., 1999. Voxelize all the models in the scene. For the haptic object, represent each voxel by a point  Point Shell. Test points against voxels of the scene. Compute force at each point that penetrates an object. Add up all the force and torque, and apply them to the user through virtual coupling.

6DOF Haptics: Voxel sampling All the objects are voxelized. For the haptic object, store points at the center of the voxels, with inward normals determined by the polygonal model. For the objects in the scene, create the voxmap, a 512-tree with 3 levels (the 512-tree is an extension of the octree, with 512 children per node, cube of 8*8*8). The 512-tree optimizes memory. The initial implementation is only for static objects.

6DOF Haptics: Voxel sampling Original object Voxels Pointshell

6DOF Haptics: Voxel sampling All the voxels in the voxmap store info about their position in the object: interior, surface, proximity or free space. interior surface proximity free space

6DOF Haptics: Voxel sampling To start the collision query, all the pointshell has to be transformed to the local coordinates of the voxelized objects. Points that are in the surface or proximity voxels are considered as contacts.

6DOF Haptics: Voxel sampling Forces at each contact are computed according to the tangent-plane force model. Contacts are valid if d<0 (penetration). If penetration to the interior happens, the contact is missed. One might think of computing a distance field in the interior, but the distance depends on the normal N. N Take the plane with normal N that goes through the center of the voxel. Compute d as the distance from the point to that plane. d

6DOF Haptics: Voxel sampling The force is continuous in direction, but discontinuous in magnitude. The virtual coupling (see later) is supposed to filter these discontinuities. N d1 d1d2 The contact might even show up or disappear suddenly d2 motion

6DOF Haptics: Voxel sampling Force per contact: Add all the forces together. Several contacts act as a single contact with higher stiffness. Limit the total stiffness: There can still be problems of irregular distribution of stiffness. If n<10 If n>10

6DOF Haptics: Voxel sampling Irregular distribution of stiffness. Geometry-driven instabilities F1=5*k*d1 F2=k*d2 F2=F1  d2=5*d1 Risk of penetration! 1 contact 5 contacts Note that geometry-driven instabilities can show up in all approaches. Object bounces from one side to the other. More common due to torque on long objects.

6DOF Haptics: Voxel sampling Penetration has to be avoided. Pre-contact braking forces. Strong viscous force applied in the proximity layer. The viscosity is adjusted so that all the kinetic energy is dissipated in a distance equal to the width of the layer.

6DOF Haptics: Voxel sampling Virtual coupling, see Adams and Hannaford, 1998. Spring-damper between the position and orientation of the haptic device and the position and orientation of the object. Filters discontinuities, to enhance stability. We can run into numerical integration problems, though. Remind that we said at the beginning that the position of the object and the device couldn’t be the same if we used rigid body dynamics? Virtual coupling is the solution.

6DOF Haptics: Polygonal models Gregory et al., 2000; Kim et al., 2002. Convex decomposition of polygonal models. Collision detection (modified SWIFT++), returning all pairs of convex pieces within a tolerance. Handle shallow penetrations. Cluster contacts. Combination of penalty-based restoring forces and impulsive response.

6DOF Haptics: Polygonal models SWIFT++, by Ehmann and Lin, 2001. Hierarchy of convex pieces. Voronoi marching between pairs of pieces. Travel down if the distance is less than the tolerance.

6DOF Haptics: Polygonal models Modifications to SWIFT++: Return all pairs of pieces inside the tolerance (group that contains the set of local minima of the distance function). Handle shallow penetrations, computing penetration depth between convex pieces. Check for pairs of features with overlapping normals.

6DOF Haptics: Polygonal models Convex decomposition: Hard problem. The decomposition might not be efficient. Explosion of pairs O(n4). Surface decomposition  Convex patches, completed with virtual faces. Linear size. Solid decomposition Surface decomposition

6DOF Haptics: Polygonal models Surface decomposition good for finding global minimum in disjoint situation. Unique pair of features, both original. Problems for local minima and penetration. The features might be virtual. If a single face is a convex piece itself, penetrating contacts might be missed.

6DOF Haptics: Polygonal models Cluster contacts. Find unique contact (distance, normal and point), averaging contacts in a neighborhood. Weighted average, using distance as the weight. This solves the problem of the irregular distribution of stiffness. Pretty smooth varying forces in practice, without need of virtual coupling. Additional features (texture, friction…) can be rendered in a per-contact basis.

6DOF Haptics: Unsolved problems A distance field approach to voxel-based haptics. Efficient convex decomposition. Looking at contacts between objects in real world, it seems like when two objects touch in many points we don’t need all the detail at particular contacts. Many contact points and velocity between the objects reduces the perception of details. Simplification? How can the virtual proxy be extended to 6DOF haptics? Rigid body simulation is about solving a lot of constraints at a time. But what if some of those constraints are redundant? We don’t care about where the force is applied, we care about the force itself… Maybe if we can find the contacts and then refine them to find the set of active contacts… Maybe then we can solve it a la Baraff…

6DOF Haptics: Unsolved problems How can the virtual proxy be extended to 6DOF haptics? Rigid body simulation is about solving a lot of constraints at a time. But what if some of those constraints are redundant? Maybe if we can find the contacts and then refine them to find the set of active constraints… How can the virtual proxy be extended to 6DOF haptics? Rigid body simulation is about solving a lot of constraints at a time. But what if some of those constraints are redundant? We don’t care about where the force is applied, we care about the force itself… Maybe if we can find the contacts and then refine them to find the set of active contacts… Maybe then we can solve it a la Baraff…

References R. J. Adams and B. Hannaford. A Two-port Framework for the Design of Unconditionally Stable Haptic Interfaces. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1998. C. Basdogan, C. H. Ho and M. Srinivasan. A Ray-based Haptic Rendering Technique for Displaying Shape and Texture of 3D Objects in Virtual Environments. In the Proc. of ASME Dynamic Systems and Control Division, 1997. G. Burdea. Force and Touch Feedback for Virtual Reality. John Wiley and Sons, 1996. S. A. Ehmann and M. C. Lin. Accurate and Fast Proximity Queries between Polyhedra Using Surface Decomposition. In Proc. of Eurographics, 2001. A. Gregory, M. C. Lin, S. Gottschalk and R. M. Taylor II. H-Collide: A Framework for Fast and Accurate Collision Detection for Haptic Interaction. In Proc. of IEEE Virtual Reality Conference, 1999. A. Gregory, A. Mascarenhas, S. Ehmann, M. C. Lin and D. Manocha. Six Degree-of-Freedom Haptic Display of Polygonal Models. In Proc. of IEEE Visualization, 2000.

References V. Hayward and B. Armstrong. A New Computational Method of Friction Applied to Haptic Rendering. In Lecture Notes in Control and Information Sciences, Vol. 250, Springer-Verlag, 2000. Y. J. Kim, M. A. Otaduy, M. C. Lin and D. Manocha. Six-Degree-of-Freedom Haptic Display Using Localized Contact Computations. To appear in Proc. of Haptic Symposium, 2002. W. R. Mark, S. C. Randolph, M. Finch, J. M. Van Verth, and R. M. Taylor II. Adding Force Feedback to Graphics Systems: Issues and Solutions. In Proc. of ACM SIGGRAPH, 1996. W. A. McNeely, K. D. Puterbaugh and J. J. Troy. Six Degree-of-Freedom Haptic Rendering Using Voxel Sampling. In Proc. of ACM SIGGRAPH, 1999. D. C. Ruspini, K. Kolarov, and O. Khatib. The Haptic Display of Complex Graphical Environments. In Proc. of ACM SIGGRAPH, 1997. C. Zilles and K. Salisbury. A Constraint-based God Object Method for Haptic Display. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1995.