Dimitris Valeris Thijs Ratsma Synthesis of Detailed Hand Manipulations Using Contact Sampling Yuting Ye & C. Karen Liu [2012] Dimitris Valeris Thijs Ratsma

Table of contents Introduction Related work Short overview Technical explanation Results Limitations Future development Questions/discussion

Introduction Problems Subtle realism Tracking body + hands

Introduction Current solutions Manually animate hand Capture hand separately Wearable device

Introduction Proposed solution Synthesize hand manipulation Input Full body motion + wrist movement Object motion Output Set of widely varied, realistic hand motions

Related Work Hand motion Grasping motions using kinematics (Koga et al. [1994]; Huang et al. [1995]; Aydin and Nakajima [1999]) Sampling Desired joint angles (Liu et al. [2010]) Initial joint configuration (Sok et al. [2007])

Related Work Physical Simulation Grasp controllers (Kry and Pai [2006]) Detailed Manipulations Robotics (Ciocarlie and Allen [2008]; Hamer et al. [2011])

Technical Explanation Search for Contact Point Trajectories Basic algorithm: Depth-first search on a tree structure. At each level of the tree there are two main tasks for searching: Generate a small set of nodes as potential expansion of the tree. Test feasibility of a random node from set.

Technical Explanation Notations Definition of state s={q,P,F} Additional definitions: contact point p∈P (pair of p.o and p.h) contact force f∈F (in Cartesian space: fC = f ⊥n+f //) guiding configuration c (max one contact point for each finger) guiding points, p

Technical Explanation Algorithm 2

Technical Explanation Generate New Nodes (Initial case) Identify a frame in two steps: Use the wrist motion to compute a reachable volume for each finger. Intersect volumes with the surface of the object. Call the intersecting areas on the surface of the object, contact patches. Create uniform point samples on the contact patches. Pair each contact patch with a finger tip to form a Guiding point

Technical Explanation Generate New Nodes (General case) Produce a set of new guiding points based on contact dynamics (static, sliding, breaking) by the following steps: Select some seed points from . From and we can create a new guiding point

Technical Explanation Generate New Nodes (General case) Rules to create a new guiding point, for the current level: If is within friction cone, the current contact must remain static. If is on the boundary of friction cone, the current contact can remain static or slide. If = 0, the finger can remain static, or release contact and move to a neighboring location.

Test feasibility (kinematics) Consider a random guiding configuration to test its feasibility. c is kinematically feasible if a penetration-free hand pose can be recovered from c, by solving a sequence of IK problems. (1) is the forward kinematic function that computes the position of in the local coordinates of the object given pose q. outputs the normal direction of the nail on the finger that contains . The third equation prevents large change of hand poses.

Test feasibility (kinematics) Penetrations are considered only on Distal and Intermediate phalanges. Apply collision detection to these phalanges to find the pair of contacts of the deepest penetration. Build a problem by substitute the pair into Equations 1,2. The solution to the problem can reduce penetrations. Iterate until to generate a penetration-free hand pose q.

Test feasibility (dynamics) Test whether the contact points P can produce sufficient contact forces F. If a feasible solution cannot be found, we consider that c fails the dynamic feasibility test. (5)The inertial and the gravitational forces in the generalized coordinates are included in G (6) Constrain the friction direction to be perpendicular to the contact normal. (7) Friction within Coulomb friction cone.

Create Diverse Solutions Sparse Exploration For small window we can keep the same manipulation strategy. To generate more variations, use a random windows size (parameter ε). Avoid similar paths, because sibling nodes often represent similar states. No more than 2,5 feasible paths can be created for each initial sample.

Create Diverse Solutions Informed backtracking Propagate backward the possible failures, to use this information to expand nodes with higher likelihood of success. Check infeasibility in three different situations: Frame fails on kinematic test for a particular finger, Inter-penetration between two fingers, Frame fails on the dynamics test.

Create Diverse Solutions Manipulation strategy preference: Instead of picking a random strategy, we can give high priority to a preferred manipulation strategy for each finger. Contact force preference The possible manipulation strategies are determined by contact forces. By adjusting the Equation 4, we can indirectly favor different strategies.

Reconstruct hand motion After we obtain feasible trajectories of contact points, we apply a spacetime optimization to create smooth hand animations with respect to the contact points. E1 encourages smooth change of joint angles by minimizing acceleration (Eq. 10). E2 favors joints on the same finger having similar bending angles (Eq. 11)

Reconstruct hand motion Fingers in contact Fingers not in contact Attaching hands to wrists of character

Results Performance Kinematic test: 200ms Dynamic test: 5-10ms Variable parameters

Demonstration

ε T Time (sec) Solution Dead end Total frame Success frame Success ratio Solution ratio Milk box (turning) 10 71 834 241 2710 661 24.4% 4.0% Milk box (pick up) 20 101 917 2208 26588 974 3.7% 0.45% Spatula 226 1323 285 6324 3735 59.1% 6.6% Pot (with wine) 120 636 340 3235 1144 35.4% 2.9% Two-hand (small box) 15 141 666 35 273 8833 4901 55.5% 11%

Evaluation Comparison to close-range motion capture data Frequent contact movements and complex contact relations Path variability

Limitations Rigid hand model Independant finger movements No collision with environment Sliding friction constraints Plenty of other possible solutions

Limitations Kinematic test Small objects Limited planning capability

Future development Grasp planning in robotics (Song et al [2016]) Virtual grasp synthesis (Linssen [2018])

Questions?

Discussion Aside from robotics and as input for grasp controllers, what could be other useful applications for this technique? We already proposed some improvements that arose from the limitations, what other improvements to the system could be made? Do you think this technique has such a significant impact on the animation that it can replace existing approaches?