Template-Based Manipulation in Unstructured Environments for Supervised Semi-Autonomous Humanoid Robots Alberto Romay, Stefan Kohlbrecher, David C. Conner, Alexander Stumpf, and Oskar Von Stryk Michelle Levine

Overview Problem-fully autonomous robots and purely teleoperated robots inefficient in unstructured environments Combination of Semi- Autonomy and Semi- Teleoperation through the use of template-based user interface UI allows operator to receive an aggregated worldmodel from onboard sensors and send back perceptual and semantic information from the operators

Weaknesses of Fully Autonomous Robots in Unstructured Environments need extensive databases of possible objects found highly efficient grasping algorithms ability to react to unforeseen circumstances

Weaknesses of Purely Teleoperated Robots Need near real-time feedback without disruptions Require transmission of large amounts of data to the operator

Related Work A lot of successful work on fully autonomous robots in structured environments – object recognition and mission planning becomes more of a challenge in unstructured environments Affordances, JJ Gibson: the possible actions that an object offers to an organism in the environment Object-Action Complexes (OACs), Kruger: define the relationships between objects and actions

Related Work continued Nagatani-work on teleoperated robots – used in hazardous environments – wired network – Example: robot Quince to Fukushima nuclear plant in 2011 Similar approach: – automatic template fitting algorithms rather than human assisted alignment – motion planning in control station rather than onboard the robot – affordance: operator manually rotates valve template rather than specify an action (open, close) with “+/- 360 degrees”

Strengths of this Approach Human strengths: – can work in discrete space – can easily identify objects of interest – quick decision making Robot strengths: – perform calculations and gather physical properties (mass and inertia)

Template-Based Communication Inspired by theory of affordances and OACs Fast communication and more accurate than just direct communication between operator and robot Use for rescue and recovery missions in disaster scenarios

Pipeline with Templates SensePlanWalkGraspUse

Object Templates-overview templates-shown as 3D meshes of the object meshes include – information like mass and center of mass – grasp templates, pre and final grasps and basic grasp types (cylindrical, prismatic, spherical) – possibilities of actions Operator overlaps 3D mesh over sensor data and can then estimate the real object’s pose and iterate through grasp templates before acting

Grasp Template where and how to grasp the object g = (H, E, N, S, P p, P f ) – H = {1, 0}, Lefthand = 1, Righthand = 0 – E = {cylindrical, prismatic, spherical}, type of grasp – N = vector of fingers joint values where fingers make contact with the object – S € R 2, 2D position of robot pelvis relative to template – pose P p € R 3 x SO, position and quaternion orientation of hand for pre-grasp – pose P f € R 3 x SO, position and quaternion orientation of hand for final-grasp

Object Template Definition x = (I, T, M, C, G, U) – I € N, ID number of object of interest – T € N, type of template (tools, debris, hose) – M € R, estimated mass – C € R 3, estimated CoM – G, list of potential grasp templates g – U = {T x, T y, T z, R x, R y, R z } € {0, 1} 6, six dimensional vector (3D translation and rotation), defines if action is possible over a dimension

Object Template Manipulation-World Model Use LIDAR and cameras to gather sensor data IMU on pelvis obtains pose estimate Different coordinate frames to reconstruct pointclouds Visualize all joint states, self filtering from sensor data, and collision avoidance

Planning Use sensor data to generate efficient motion plans Robot creates 3D Octomap representation to avoid collisions 2D grip map slices used for locomotion planning, to plan collision free footstep plan Providing Situational Awareness to Operator Down-sample and crop sensor data (e.g. 3D pointclouds, 2D images, and video) as needed

Pipeline with Templates SensePlanWalkGraspUse

Cartesian and Circular Path Planning Vector U follows Cartesian path between initial and final poses Waypoints created using linear interpolation Use spherical linear interpolations-end effector’s goal pose can be different from start effector’s orientation Circular motion- concatenate multiple short linearly interpolated Cartesian paths (can be designed to maintain end effector’s orientation) Video: om/watch?v=wKFJO- Zkjck

DARPA Challenge, Team ViGIR-Hose Task Video: vLeo

DARPA Challenge-Hose Task

Hose Task continued

Valve Task

Valve Task continued

Conclusions and Future Work Conclusions: – Can efficiently send semantic commands -> can plan missions on the fly – Useful for grasping and manipulating objects – Limitation-small objects require fine manipulation, need to invest time to align template – fastest team to win 2 points in Hose Task and fastest to turn the nozzle out of the only 2 teams that tried (better than 2 nd place team in the Hose Task) Future Work: – automatic template fitting algorithms – automatic grasp planners-create new grasps on the fly – expand from just constrained motion paths to provide forces necessary for those motions