1. Columbia GraspIt!: A Versatile Simulator for Robotic Grasping Andrew T. Miller Columbia University

2. Columbia Outline Motivation Simulator Overview Simulator Front-end Grasp Analysis Dynamic Simulation Performing a Grasping Task Future Work

3. Columbia Simulation for Hand Design Mechanical prototypes are costly –Spend time and money Evaluate design in simulation –Examine range of motion, finger placement, kinematics, link geometry –Ensure it can grasp desired objects –Change design easily

4. Columbia Simulation for Grasp Planning Integrated grasp analysis –Grasp quality, weak point, force optimization Visualize grasp from all angles Perform many grasps quickly –Faster than using a real arm and hand Build a library of saved grasps –Recall grasp when object is encountered again

5. Columbia Library of hands and objects Intuitive interface allows many grasps to be tested quickly Visualize grasp wrench space Quality measures evaluate grasp GraspIt: A Tool for Grasping Research

6. Columbia GraspIt Components Hand/Object Construction read object model read link models read kinematics assemble hand User Interface view 3D scene change hand pose auto-grip manually move joints Contact Determination detect collisions adjust contact to object surface find contact area add friction cones Grasp Analysis compute grasp wrench space use metrics on space grasp force optimization Wrench Space Visualization create 3D projections of GWS Front end Back end Object Dynamics compute object motion

7. Columbia Hand Construction The hand kinematics (in D-H notation) specify the transforms between links, and can handle coupled joints. The 3D link geometries are accurately described in CAD model files. The format is flexible and can easily model many of the available complex articulated hands.

8. Columbia Hand Configuration File … Header info and palm description... #-----------------------------------f0--------------------------------------- #number of joints 3 #number of links 3 #Transforms from palm origin to base of the finger #EXAMPLE: #r 90 x (rotation by 90 degrees about x axis) #rr 3.0368729 y (rotation by 3.0368729 radians about y axis) #t 1.92188 134.5 39.0717 (translation in mm) t 25.0 0 -1.0 r 180 y #Joint Descriptions (1 joint per line) #DOFNum theta d a alpha DOFminval DOFmaxval #(joints are ordered from closest to palm outward) #(linear equations are of the form: q#*k+c [no spaces!]) #(coupled joints must come after joints they are coupled to) d0+90 0 50 -90 0 180 d1+5 0 70 0 0 144 d1*0.333333333+40 0 55 0 0 144 #Link Descriptions (1 link per line) #filename material lastJoint #(links are ordered from closest to palm outward) #(lastJoint is the last joint in the chain which can affect this link) link1 plastic 0 link2 plastic 1 link3 plastic 2 #-----------------------------------f1--------------------------------------- etc...

9. Columbia Hand Library BarrettRutgersRobonaut Parallel JawDLR

10. Columbia User Interaction Inventor 3D viewer Rotate hand Translate hand Auto-grasp Manipulate individual DOF’s Simple and intuitive Direct interaction gives sense of hand’s capabilities

11. Columbia Collision Detection PQP system (Larsen et al.) Each body is a polygonal soup Creates bounding volume hierarchies Fast recursive algorithm tests for overlap, or provides minimum distance

12. Columbia Finding Contacts Binary search until link is < 0.1mm away Record all triangle pairs within this distance Compute points of overlap between each pair Only keep contact points on region boundary Mark points with red friction cones Collision detected Search for instant of contact Determine contact region

13. Columbia Grasp Analysis Occurs when a contact is formed or broken Computes space of forces and torques that can be applied by the grasp Quality measures numerically evaluate grasp Provides a means to evaluate grasps –Compare grasps of one hand, one object –Compare grasps of many hands, one object –Compare grasps of many hands, across a task specified object set.

14. Columbia Wrench Spaces In 3-space, a wrench is a 6D vector composed of a force and a torque: The space of wrenches that may need to be applied during a task is the task wrench space. The space of wrenches that can be applied by a grasp is the grasp wrench space. A possible quality measure:

15. Columbia Special Types of Grasps A force-closure grasp completely restrains the object. –Origin is contained within grasp wrench space. A manipulable grasp can impart arbitrary velocities on the object without breaking contact.

16. Columbia Friction Cones Friction at a contact point allows forces in directions other than the contact normal COF,  is determined by the contacting materials Estimate friction cone as convex sum of a force vectors on the boundary assuming a unit normal force,.

17. Columbia Contact Wrenches Each force acts at a position on object. Compute the corresponding wrench with respect to object’s center of gravity:  - torque scale factor, i: contact number j: (1-8) around cone boundary

18. Columbia Objective: find total space of wrenches that can be applied by a grasp of unit magnitude. Grasp vector: Define with norm: –Sum magnitude of contact normal forces is 1. Compute grasp wrench space using qhull: Grasp Wrench Space

19. Columbia Two Measures of Quality Assume task wrench space is unknown. Estimate with wrench space ball - good grasps will resist all wrenches equally well. Previously proposed measures of quality: –Radius, , of the largest wrench space ball that can fit within the unit grasp wrench space. –Volume, v, of unit grasp wrench space.

20. Columbia Grasp Wrench Space Projections To visualize the 6D grasp wrench space project it to 3-space by fixing three coordinates. A useful choice: –See the forces that can be applied without applying a net torque (or vice versa).

21. Columbia Visualizing the Results Numeric quality & Worst case indicator Wrench Space Projection

22. Columbia Comparing Grasps

23. Columbia Comparing Grasps

24. Columbia Comparing Grasps

25. Columbia Comparing Grasps

26. Columbia Grasp Force Optimization Given a specific wrench to resist, find the optimal force to apply at contact points Forces should obey torque limit constraints and avoid friction cone boundaries Previously done by using linearized friction cones and linear programming [Kerr & Roth ’86] –overly conservative

27. Columbia Grasp Force Optimization GraspIt implements LMI technique by Han et al. ’00, Does not require friction cone approximation Feasible Set: Minimize: Determinant Maximization with LMI constraints Using hand Jacobian, it can also compute optimal joint torque values Contact forcesInverse friction cone LMIWeighting factor

28. Columbia Grasp Force Optimization

29. Columbia Simulating Object Dynamics Added realism Can examine grasp formation process Formulate non-penetration and friction cone constraints as a Linear Complementarity Problem [Stewart & Trinkle ’00] Lemke’s algorithm solves LCP to find contact forces that satisfy constraints. force OR acceleration must be 0

30. Columbia Simulating Object Dynamics

31. Columbia Simulating Object Dynamics

32. Columbia Connecting GraspIt to the Real World Object Pose Estimation Frame Grabber Grasp Execution Task Monitoring Robot/Hand Control Presented at ICRA ‘01

33. Columbia Environment Modeling Easy to import environmental objects like workbench. Puma arm with inverse kinematics module also supplied. Prevents user from planning infeasible grasps.

34. Columbia Real-Time Vision System Supply GraspIt with object pose. Needs to track object that is partially occluded by the hand. Since we have an object model, why not use model-based pose estimation and tracking? Single camera located 2m from workspace, calibrated wrt robot coordinate system.

35. Columbia Real-Time Pose Estimation and Tracking Position estimation based on Dementhon and Davis. Tracking based on Araujo and Brown’s extension of Lowe. 3D Projection Matrix Fitting 3D Model POSE Affine Motion Normal Flow Locate Visible Edges Render Model Camera Parameters Manual Initialization

36. Columbia Tracking System Example Wireframe model Selected points are used to determine pose Pose estimated, and model overlaid Normal flow estimation

37. Columbia Grasp Execution Experiments Pose estimatedUser plans graspGrasp performed

38. Columbia Grasp Execution Experiments Pose estimatedUser plans graspGrasp performed

39. Columbia Grasp Execution Experiments Pose estimatedUser plans graspGrasp performed

40. Columbia Grasping and Visual Servoing Once pose has been estimated, grasp may be planned, then executed.

41. Columbia Grasping and Visual Servoing Destination planned in GraspIt. Goal position is projected into image. Image based servoing drives robot to goal.

42. Columbia Planning a Grasping Task in GraspIt

43. Columbia Planning a Grasping Task in GraspIt

44. Columbia Planning a Grasping Task in GraspIt

45. Columbia Contributions Grasping simulator –Flexible method to specify hand construction –Interactive interface –Contact determination system –Inclusion of material properties Integrated grasp analysis theory into simulation system –GWS, quality measures, GFO Visualization techniques Object Dynamics: First implementation of S-T algorithm with large scale and interactive bodies Integration of simulation system into grasping task A useful tool for research in grasping and hand design: –Currently being used by a number of research groups for design and synthesis: NASA, Sandia, Rutgers, U of MD

46. Columbia Future Work Evaluate manipulable grasps Provide method of specifying task wrench space Human hand model Full dynamic simulation Automatic grasp selection –Combine high-level and low-level synthesis techniques –Learning techniques

47. Columbia Grasp Force Optimization Contact forcesFriction cone LMI

48. Columbia